Merge branch 'stable-2.11' into master
Merge branch 'stable-2.10' into stable-2.11
Merge branch 'stable-2.9' into stable-2.10
Make hbal deal with no-LVM storage space properly
Since 2.6, hbal crashes when used on a cluster where noLVM storage is enabled at all. The problem is that italways queries for fields that only sometimes makesense for certain types of storage. This patch will...
Conflicts: NEWS: take both additions configure.ac: ignore suffix bump...
Signed-off-by: Thomas Thrainer <thomasth@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Merge branch 'stable-2.8' into stable-2.9
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Add missing space
Also, refactor the line to keep it under 80 chars.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Add private OS parameters to cluster and instance conf
This updates objects, constructors and mocks for Instance and Clusterobjects in Python and Haskell.
Signed-off-by: Santi Raffa <rsanti@google.com>Reviewed-by: Jose A. Lopes <jabolopes@google.com>
Remove FromString in favor of Error from standard libraries
They have the very same functionality, and using our own FromString onlycauses unnecessary code duplication.
Signed-off-by: Petr Pudlak <pudlak@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>...
Signed-off-by: Petr Pudlak <pudlak@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Fix whitespace
Fix whitespace in several modules.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Merge branch 'stable-2.10' into master
break line longer than 80 chars
hsqueeze: tag nodes before offlining them
hsqueeze is supposed to tag nodes before powering them down, so thatit later can recognize which nodes can be activated later. When showingthe commands to execute, also add the tagging commands.
Signed-off-by: Klaus Aehlig <aehlig@google.com>...
hsqueeze: only consider nodes that are not secondaries
If an instance has a secondary node, it cannot be easilymoved to every node (in the same node group), as otherwiseno node would be distinguished as secondary. As hsqueezeshould only consider nodes were moving the instances away...
Gluster: minimal implementation
Add Gluster to Ganeti by essentially cloning the shared file behavioureverywhere in the code base.
Signed-off-by: Santi Raffa <rsanti@google.com>Signed-off-by: Thomas Thrainer <thomasth@google.com>Reviewed-by: Thomas Thrainer <thomasth@google.com>
Avoid lines longer than 80 chars
...as they're a lint error.
Rename getClient/Server to getLuxiClient/Server
Later they will be split into LUXI-specific and general parts.
Fix evacuation out of drained node
Refactor reading live data in htools
This simplifies different handling of individual items.
Cherry-picked from 8c72f7119f50a11661aacba2a1abffdfdc6f7cfa.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>Reviewed-by: Thomas Thrainer <thomasth@google.com>
Modify --mond to yes|no option
Modify --mond option used by hail, hbal and hinfo from nonargument to yes|no option.
Signed-off-by: Spyros Trigazis <strigazi@gmail.com>Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Rename LuxiSocket to MasterSocket
Rename the constants to name the socket to connect masterd,as the name LuxiSocket hints on luxid, which is differentfrom masterd.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Petr Pudlak <pudlak@google.com>
hsqueeze: fix position of option in gnt-node power
hsqueeze can produce a shell script with the commands to squeezethe cluster; in the script, fix the position of the '-f' optionin the 'gnt-node power' command.
hsqueeze: add option to show or save commands
Add an option to hsqueeze to show, or save in a file, the commandsthat have to be carried out.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Hrvoje Ribicic <riba@google.com>
hsqueeze: when balancing also keep the move sequence
In hsqueeze, when computing the balancing sequence, alsoremember the sequence of moves that lead there.
Add function to get the moves between two configurations
Add a function that, given two adjacent cluster configurations ofa balancing sequence, computes the moves that led from the firstto the second configuration.
In the list of involved nodes, drop "no secondary"
When grouping moves into jobs, a new job set is started, if the newmove involves a node also touched by a previous move. When computingthe list of involved nodes, the new primary and secondary nodes of the...
Move saving of a command list to CLI
Move the function that saves a list of a command in a fileto CLI.hs. In this way, it is reusable by other htools.
hsqueeze: support planing for onlining nodes
If the amount of free resources falls below a given threshold,hsqueeze will suggest putting standby nodes back online untilthe minimum of free resources is reached, or all standby nodesare online.
Add an --minimal-resources option
Add a new option, to be used by hsqueeze, to specifythe amount of free resources that has to be on eachnode, in order not to start onlining standby nodes.It is given as a multiple of the standard allocation,as specified by the instance policy....
Text Backend: correctly read data for offline nodes
With standby nodes, simply ignoring the specification ofan offline node is not sufficient any more.
Change default for target resource to 2.0
The target resources, as used by hsqueeze, are supposed tobe strictly higher than the minimal resources. However,keeping minimal resources of less than a single instanceis not a useful reserve.
Add hsqueeze planning for compression
Add a new htool, hsqueeze, for dynamic power management. Thiscommit only implements the first useful part: plan (but notexecuting) taking nodes offline nodes still keeping within theresource limit.
Provide --target-resources option
Add a new option, to be used by hsqueeze, to specify thetarget free resources on each node. It is given as a multipleof the standard allocation, as specified in the instance policy.
Hs2Py constants: constants related to 'Ganeti.Htools.Types'
Add constants related to the Haskell types defined in'Ganeti.Htools.Types' to the Haskell to Python constant generation.
Merge 'EvacNode' and 'NodeEvacMode'
Merge 'EvacMode' and 'NodeEvacMode' which are associated with theduplicated constants 'IALLOCATOR_NEVAC_*' and 'NODE_EVAC_*'. Thismerge moves the type 'EvacMode' to 'Ganeti.Types' in order to avoid acircular dependency between 'Ganeti.HsConstants' and...
Add test for mond-data mock file
Test if the input file imported with the --mond-data option isparsed properly.
Import MonD data from file
Implement functionality to import MonD data from a mock file toHTools (mainly for testing purposes) with the --mond-dataoption.
Signed-off-by: Spyros Trigazis <strigazi@gmail.com>Signed-off-by: Michele Tartara <mtartara@google.com>...
Fetch data from MonD to HTools
Contact all MonDs from HTools to fetch data from its DataCollectors (only CPUload Data Collector is queried at themoment). This information is available to all HTools with the--mond option and can be ignored if the --ignore-dynu option is...
Make hbal support the option --ignore-dynu
This will allow to do balancing based only on static information.In particular, the effect of balancing towards same number ofinstances per node that the dynamic usage consideration hasif no usage file is provided, can be turned off, if this is...
Make ExtLoader honor the --ignore-dynu option
Make the external data loader correctly honor the --ignore-dynu optionby clearing the dynamic utilisation data from all instances if the optionis given.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Provide utility function to clear utilisation data
Using this utility function, the dynamic utilisation of all instancescan be set to 0. This will have the effect of all utilisation being ignored.
Define a new option --ignore-dynu
This option, if set, will tell htools to ignore any dynamicuse data.
Fix opcodes and parameters
Merge branch 'stable-2.9'
Update Harep, Query server, and tests
Update Harep, Haskell query server, and tests concerning Luxi andopcodes to reflect the changes to Haskell to Python opcodegeneration. This change is necessary because TagObject is replaced byTagKind and some types in opcodes and parameters changed to be...
Add cleanup parameter to instance failover
Most of the code is shared with instance migrate, so we actually only needto add the parameter and pass its value along the the common code.
Also, tests and harep are updated to support the right set of options to...
Merge branch 'stable-2.8' into master
Load CPUs used by the node OS in htools
A new field is added to the Node type, and it's used to initialize the usedCPUs field.
The signature of Node.create has been split among lines to match theparameter list.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>...
hroller: option --full-evacuation
Add an option to hroller, to plan for full evacuation of thenodes to be rebooted, i.e., also plan for replacement secondarynodes for all instances on the node after migrating out instanceswith this node as primary.
Extract a partition functional
Separate the partitionNonRedundant function in hroller into ageneral functional that partitions a list of nodes accordingto some clearing strategy and the specialization of movingnon-redundant instances out. In this way, we don't have to...
Extract functional for greedily clearing nodes
The method clearNodes in hroller greedily clears nodes ofnon-redundant instances by moving them to a different node. This patchseparates the greedy clearing algorithm from the specialization tonon-redundant instances; in this way, we don't have to duplicate code...
Make hroller not consider offline nodes for evacuation
When planing on where to evacuate the non-redundant instancesof the nodes to be rebooted, it doesn't make sense to consideroffline nodes. So add this restriction to hroller.
Update comments in hroller code
hroller schedules moves of instances to have rebooted nodesfree of instances with this node as primary. Update the commentsto reflect that this move planning is for non-redundant instancesonly.
Remove obsolete TODO
Originally, hroller started as a tool for offline maintenance only.There it made sense to warn about instances still running. By now,default planning is to migrate instance off the nodes to be rebooted,with options for other behavior (like pretending that all instances...
Support big-step shrinking in tiered allocation
In tiered allocation, if by shrinking only a single resource a validallocation can be found, shrinking is bound to shrink on this resource.Of course, after shrinking that resource a little bit without finding...
Index instances by their UUID
No longer index instances by their name but by their UUID in the clusterconfig. This change changes large parts of the code, as the followingadjustments were necessary: * Change the index key to UUID in the configuration and the...
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Thomas Thrainer <thomasth@google.com>
In tiered allocation, cut non-promising shrinking tries
The heuristics for tiered allocation has been improved in that itchooses to shrink such a resource next where by shrinking only thisresource a valid allocation can be made, if such a resource exists....
Merge branch 'stable-2.8' into 'master'
Improve hspace shrinking strategy
In tired allocation, hspace shrinks that resource of the instancenext, that causes failure on most nodes. While, this is not a badstrategy in general, it can lead hspace into a dead end if for a largenumber of nodes a particular resource blocks any further allocation of...
Merge branch 'stable-2.7' into stable-2.8
Conflicts: (trival, take union of added files/tests) Makefile.am test/hs/shelltests/htools-hspace.test...
Make shrinkByType aware of individual disks
When shrinking an instances, you can't just get smaller disk footprintwhile leaving the individual disks as they are. Make the shrinkheuristic aware of that fact, and decrease all individual disks aswell. Fixes issue 484....
Index nodes by their UUID
No longer index nodes by their name but by their UUID in the clusterconfig. This change changes large parts of the code, as the followingadjustments were necessary: * Change the index key to UUID in the configuration and the ConfigWriter, including all methods....
Add missing parenthesis to description of --machine-readable
Add type annotation to avoid monomorphism restriction
Even though we need the let-bound variable showMoves onlyat type [(String, String)] -> IO (), it's most general typewould be (PrintfArg a, PrintfArg b) => [(a, b)] -> IO ().This causes the monomorphism restriction apply to that binding,...
add option --print-moves to hroller
If non-redundant instances are present in the cluster, hroller willplan for them to move to other nodes while the group is rebooted.This adds an option to also show this plan.
hspace prints info about spindles
Statistics about spindles are tracked. In human-readable output, spindlesare printed only when used (i.e., exclusive storage is enabled). Formachine-oriented output, they are always there.
Add support for shrinking an instance spindles-wise
This makes tiered allocation in hspace work also with respect to spindles.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Spindles become part of htools resource spec
Spindles are now part of resource spec. Instances get created with spindlesspecified (which are just ignored when exclusive storage is disabled).
htools cluster score takes spindles into account
When exclusive storage is enabled, spindles are used instead of disk spaceto compute the cluster score.
Comments and variable names in computePDsk has been changed to match theactual code.
Update spindles when moving instances in htools
Spindles get updated, and errors raised when not enough free spindlesexist. No new error is raised when exclusive storage is disabled.
Unit tests included.
Unit tests for htools and exclusive storage
The existing tests are run also on nodes with exclusive storage enabled. Thevalues for spindles and exclusive storage are set in a consistent way, forboth nodes and instances.
Load complete instance disk information through LUXI
Information about size and spindles of all the disks of an instance is loadedby the LUXI backend, instead of faking one equivalent big disk. In this wayinstance policy checks are more accurate.
Load node spindles data in htools
The data structure for nodes gets a new field for free spindles, and theexisting field for total spindles gets renamed to avoid identifying it withthe node parameter that had the same name. These fields get filled with...
Check real spindles in ipolicies
When exclusive storage is enabled, the spindles in instance disks are usedto check the instance policies (as outlined in design-partitioned.rst).
Check the full instance specs in htools
Spindles and disk count are checked too. Existing functions have beenrefactored, so common parts are not duplicated.
Add spindles to instance disks in htools
A new data type is introduced for disks to store both size and spindles.When available, spindles are filled with input data. Except for loading andstoring, spindles are ignored.
Load exclusive_storage in htools
The node parameter is loaded into the data structures. No behavior is yetmodified.
Add unit test for text backend + fix bug
Test serialization and deserialization of instances. Fix check of secondarynode.
Restrict instance moves in hroller to the same node group
When scheduling rolling reboots, hroller looks for nodes to evacuatethe non-redundant instances to. This is done by greedily movinginstances to other nodes that can take them, policy wise and capacity...
Add tests for network-aware allocation
hail-alloc-invalid-network defines a cluster with two nodegroups and anallocation request which does not fit on any of the groups. Group 1 hasinvalid disk-templates while Group 2 is not connected to the rightnetworks....
Honor network connections in hail
Before trying to allocate nodes in node groups, node groups are nowfiltered based on the networks they are connected to an the networkswhich are required by the new instance.
Signed-off-by: Thomas Thrainer <thomasth@google.com>...
Parse NIC data from allocation request in hail
Add a NIC type and extend the Instance type by a list of NIC's. Parsethe NIC's in allocation requests and store them for now. Later patcheswill make use of this field in order to ensure that the requestedinstance is only placed in node groups wich are connected to those...
Support group networks in Text backend
The Text backend now parses network UUID (comma separated) andserializes them in the same form.The test data is adapted to the new format.
Signed-off-by: Thomas Thrainer <thomasth@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Parse node group networks
Extend the Group by the network ids it is connected to. Adaptthe IAlloc backend such that the networks are parsed correctly.This also required the adaption of test data.
hroller: option to ignore non-redundant instances
Add an option to hroller restoring the old behavior on not takingany non-redundant instances into account when forming rebootgroups.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Make hroller also plan for non-redundant instances
Non-redundant instances need to be moved to a different nodebefore maintenance of the node. Even though they can be moved toany node, there must be enough capacity to host the instances of thereboot group to be evacuated....
hroller: option to skip nodes with non-redundant instances
So far, hroller ignores the fact, that non-redundant instances exist.One option to deal is non-redundant instances is to not schedule thosenodes for reboot. This is supported by adding the option --skip-non-redundant....
Remove trailing whitespace
Support online-maintenance in hroller
Make hroller take into account the nodes (redundant) instanceswill be migrated to. This be behavior can be overridden by the--offline-maintenance option which will make hroller plan underthe assumption that all instances will be shutdown before starting...
Support construction of the graph of all reboot constraints
For online rolling reboots, there are two kind of restrictions. First,we cannot reboot the primary and secondary nodes of an instancetogether. Secondly, two nodes cannot be rebooted simultaneously, if...
Add option --one-step-only to hroller
Add a new option to hroller to only output information about the firstreboot group. Together with the option --node-tags this allows for thefollowing work flow. First tag all nodes; then repeatedly compute thefirst node group, handle these nodes and remove the tags. In between...
Sort reboot groups by size
Make hroller output the node groups not containing the master nodesorted by size, largest group first. The master node still remainsthe last node of the last reboot group. In this way, most progressis made when switching back to normal cluster operations after the...