Rename getClient/Server to getLuxiClient/Server
Later they will be split into LUXI-specific and general parts.
Signed-off-by: Petr Pudlak <pudlak@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Modify --mond to yes|no option
Modify --mond option used by hail, hbal and hinfo from nonargument to yes|no option.
Signed-off-by: Spyros Trigazis <strigazi@gmail.com>Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Rename LuxiSocket to MasterSocket
Rename the constants to name the socket to connect masterd,as the name LuxiSocket hints on luxid, which is differentfrom masterd.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Petr Pudlak <pudlak@google.com>
hsqueeze: fix position of option in gnt-node power
hsqueeze can produce a shell script with the commands to squeezethe cluster; in the script, fix the position of the '-f' optionin the 'gnt-node power' command.
Signed-off-by: Klaus Aehlig <aehlig@google.com>...
hsqueeze: add option to show or save commands
Add an option to hsqueeze to show, or save in a file, the commandsthat have to be carried out.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Hrvoje Ribicic <riba@google.com>
hsqueeze: when balancing also keep the move sequence
In hsqueeze, when computing the balancing sequence, alsoremember the sequence of moves that lead there.
Add function to get the moves between two configurations
Add a function that, given two adjacent cluster configurations ofa balancing sequence, computes the moves that led from the firstto the second configuration.
In the list of involved nodes, drop "no secondary"
When grouping moves into jobs, a new job set is started, if the newmove involves a node also touched by a previous move. When computingthe list of involved nodes, the new primary and secondary nodes of the...
Move saving of a command list to CLI
Move the function that saves a list of a command in a fileto CLI.hs. In this way, it is reusable by other htools.
hsqueeze: support planing for onlining nodes
If the amount of free resources falls below a given threshold,hsqueeze will suggest putting standby nodes back online untilthe minimum of free resources is reached, or all standby nodesare online.
Add an --minimal-resources option
Add a new option, to be used by hsqueeze, to specifythe amount of free resources that has to be on eachnode, in order not to start onlining standby nodes.It is given as a multiple of the standard allocation,as specified by the instance policy....
Text Backend: correctly read data for offline nodes
With standby nodes, simply ignoring the specification ofan offline node is not sufficient any more.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Change default for target resource to 2.0
The target resources, as used by hsqueeze, are supposed tobe strictly higher than the minimal resources. However,keeping minimal resources of less than a single instanceis not a useful reserve.
Add hsqueeze planning for compression
Add a new htool, hsqueeze, for dynamic power management. Thiscommit only implements the first useful part: plan (but notexecuting) taking nodes offline nodes still keeping within theresource limit.
Provide --target-resources option
Add a new option, to be used by hsqueeze, to specify thetarget free resources on each node. It is given as a multipleof the standard allocation, as specified in the instance policy.
Hs2Py constants: constants related to 'Ganeti.Htools.Types'
Add constants related to the Haskell types defined in'Ganeti.Htools.Types' to the Haskell to Python constant generation.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>Reviewed-by: Thomas Thrainer <thomasth@google.com>
Merge 'EvacNode' and 'NodeEvacMode'
Merge 'EvacMode' and 'NodeEvacMode' which are associated with theduplicated constants 'IALLOCATOR_NEVAC_*' and 'NODE_EVAC_*'. Thismerge moves the type 'EvacMode' to 'Ganeti.Types' in order to avoid acircular dependency between 'Ganeti.HsConstants' and...
Add test for mond-data mock file
Test if the input file imported with the --mond-data option isparsed properly.
Import MonD data from file
Implement functionality to import MonD data from a mock file toHTools (mainly for testing purposes) with the --mond-dataoption.
Signed-off-by: Spyros Trigazis <strigazi@gmail.com>Signed-off-by: Michele Tartara <mtartara@google.com>...
Fetch data from MonD to HTools
Contact all MonDs from HTools to fetch data from its DataCollectors (only CPUload Data Collector is queried at themoment). This information is available to all HTools with the--mond option and can be ignored if the --ignore-dynu option is...
Make hbal support the option --ignore-dynu
This will allow to do balancing based only on static information.In particular, the effect of balancing towards same number ofinstances per node that the dynamic usage consideration hasif no usage file is provided, can be turned off, if this is...
Make ExtLoader honor the --ignore-dynu option
Make the external data loader correctly honor the --ignore-dynu optionby clearing the dynamic utilisation data from all instances if the optionis given.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Provide utility function to clear utilisation data
Using this utility function, the dynamic utilisation of all instancescan be set to 0. This will have the effect of all utilisation being ignored.
Define a new option --ignore-dynu
This option, if set, will tell htools to ignore any dynamicuse data.
Fix opcodes and parameters
Merge branch 'stable-2.9'
Merge branch 'stable-2.8' into stable-2.9
Update Harep, Query server, and tests
Update Harep, Haskell query server, and tests concerning Luxi andopcodes to reflect the changes to Haskell to Python opcodegeneration. This change is necessary because TagObject is replaced byTagKind and some types in opcodes and parameters changed to be...
Add cleanup parameter to instance failover
Most of the code is shared with instance migrate, so we actually only needto add the parameter and pass its value along the the common code.
Also, tests and harep are updated to support the right set of options to...
Merge branch 'stable-2.8' into master
Load CPUs used by the node OS in htools
A new field is added to the Node type, and it's used to initialize the usedCPUs field.
The signature of Node.create has been split among lines to match theparameter list.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>...
hroller: option --full-evacuation
Add an option to hroller, to plan for full evacuation of thenodes to be rebooted, i.e., also plan for replacement secondarynodes for all instances on the node after migrating out instanceswith this node as primary.
Extract a partition functional
Separate the partitionNonRedundant function in hroller into ageneral functional that partitions a list of nodes accordingto some clearing strategy and the specialization of movingnon-redundant instances out. In this way, we don't have to...
Extract functional for greedily clearing nodes
The method clearNodes in hroller greedily clears nodes ofnon-redundant instances by moving them to a different node. This patchseparates the greedy clearing algorithm from the specialization tonon-redundant instances; in this way, we don't have to duplicate code...
Make hroller not consider offline nodes for evacuation
When planing on where to evacuate the non-redundant instancesof the nodes to be rebooted, it doesn't make sense to consideroffline nodes. So add this restriction to hroller.
Update comments in hroller code
hroller schedules moves of instances to have rebooted nodesfree of instances with this node as primary. Update the commentsto reflect that this move planning is for non-redundant instancesonly.
Remove obsolete TODO
Originally, hroller started as a tool for offline maintenance only.There it made sense to warn about instances still running. By now,default planning is to migrate instance off the nodes to be rebooted,with options for other behavior (like pretending that all instances...
Support big-step shrinking in tiered allocation
In tiered allocation, if by shrinking only a single resource a validallocation can be found, shrinking is bound to shrink on this resource.Of course, after shrinking that resource a little bit without finding...
Index instances by their UUID
No longer index instances by their name but by their UUID in the clusterconfig. This change changes large parts of the code, as the followingadjustments were necessary: * Change the index key to UUID in the configuration and the...
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Thomas Thrainer <thomasth@google.com>
In tiered allocation, cut non-promising shrinking tries
The heuristics for tiered allocation has been improved in that itchooses to shrink such a resource next where by shrinking only thisresource a valid allocation can be made, if such a resource exists....
Merge branch 'stable-2.8' into 'master'
Improve hspace shrinking strategy
In tired allocation, hspace shrinks that resource of the instancenext, that causes failure on most nodes. While, this is not a badstrategy in general, it can lead hspace into a dead end if for a largenumber of nodes a particular resource blocks any further allocation of...
Merge branch 'stable-2.7' into stable-2.8
Conflicts: (trival, take union of added files/tests) Makefile.am test/hs/shelltests/htools-hspace.test...
Make shrinkByType aware of individual disks
When shrinking an instances, you can't just get smaller disk footprintwhile leaving the individual disks as they are. Make the shrinkheuristic aware of that fact, and decrease all individual disks aswell. Fixes issue 484....
Index nodes by their UUID
No longer index nodes by their name but by their UUID in the clusterconfig. This change changes large parts of the code, as the followingadjustments were necessary: * Change the index key to UUID in the configuration and the ConfigWriter, including all methods....
Add missing parenthesis to description of --machine-readable
Add type annotation to avoid monomorphism restriction
Even though we need the let-bound variable showMoves onlyat type [(String, String)] -> IO (), it's most general typewould be (PrintfArg a, PrintfArg b) => [(a, b)] -> IO ().This causes the monomorphism restriction apply to that binding,...
add option --print-moves to hroller
If non-redundant instances are present in the cluster, hroller willplan for them to move to other nodes while the group is rebooted.This adds an option to also show this plan.
hspace prints info about spindles
Statistics about spindles are tracked. In human-readable output, spindlesare printed only when used (i.e., exclusive storage is enabled). Formachine-oriented output, they are always there.
Add support for shrinking an instance spindles-wise
This makes tiered allocation in hspace work also with respect to spindles.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Spindles become part of htools resource spec
Spindles are now part of resource spec. Instances get created with spindlesspecified (which are just ignored when exclusive storage is disabled).
htools cluster score takes spindles into account
When exclusive storage is enabled, spindles are used instead of disk spaceto compute the cluster score.
Comments and variable names in computePDsk has been changed to match theactual code.
Update spindles when moving instances in htools
Spindles get updated, and errors raised when not enough free spindlesexist. No new error is raised when exclusive storage is disabled.
Unit tests included.
Unit tests for htools and exclusive storage
The existing tests are run also on nodes with exclusive storage enabled. Thevalues for spindles and exclusive storage are set in a consistent way, forboth nodes and instances.
Load complete instance disk information through LUXI
Information about size and spindles of all the disks of an instance is loadedby the LUXI backend, instead of faking one equivalent big disk. In this wayinstance policy checks are more accurate.
Load node spindles data in htools
The data structure for nodes gets a new field for free spindles, and theexisting field for total spindles gets renamed to avoid identifying it withthe node parameter that had the same name. These fields get filled with...
Refactor reading live data in htools
This simplifies different handling of individual items.
Check real spindles in ipolicies
When exclusive storage is enabled, the spindles in instance disks are usedto check the instance policies (as outlined in design-partitioned.rst).
Check the full instance specs in htools
Spindles and disk count are checked too. Existing functions have beenrefactored, so common parts are not duplicated.
Add spindles to instance disks in htools
A new data type is introduced for disks to store both size and spindles.When available, spindles are filled with input data. Except for loading andstoring, spindles are ignored.
Load exclusive_storage in htools
The node parameter is loaded into the data structures. No behavior is yetmodified.
Add unit test for text backend + fix bug
Test serialization and deserialization of instances. Fix check of secondarynode.
Restrict instance moves in hroller to the same node group
When scheduling rolling reboots, hroller looks for nodes to evacuatethe non-redundant instances to. This is done by greedily movinginstances to other nodes that can take them, policy wise and capacity...
Add tests for network-aware allocation
hail-alloc-invalid-network defines a cluster with two nodegroups and anallocation request which does not fit on any of the groups. Group 1 hasinvalid disk-templates while Group 2 is not connected to the rightnetworks....
Honor network connections in hail
Before trying to allocate nodes in node groups, node groups are nowfiltered based on the networks they are connected to an the networkswhich are required by the new instance.
Signed-off-by: Thomas Thrainer <thomasth@google.com>...
Parse NIC data from allocation request in hail
Add a NIC type and extend the Instance type by a list of NIC's. Parsethe NIC's in allocation requests and store them for now. Later patcheswill make use of this field in order to ensure that the requestedinstance is only placed in node groups wich are connected to those...
Support group networks in Text backend
The Text backend now parses network UUID (comma separated) andserializes them in the same form.The test data is adapted to the new format.
Signed-off-by: Thomas Thrainer <thomasth@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Parse node group networks
Extend the Group by the network ids it is connected to. Adaptthe IAlloc backend such that the networks are parsed correctly.This also required the adaption of test data.
hroller: option to ignore non-redundant instances
Add an option to hroller restoring the old behavior on not takingany non-redundant instances into account when forming rebootgroups.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Make hroller also plan for non-redundant instances
Non-redundant instances need to be moved to a different nodebefore maintenance of the node. Even though they can be moved toany node, there must be enough capacity to host the instances of thereboot group to be evacuated....
hroller: option to skip nodes with non-redundant instances
So far, hroller ignores the fact, that non-redundant instances exist.One option to deal is non-redundant instances is to not schedule thosenodes for reboot. This is supported by adding the option --skip-non-redundant....
Remove trailing whitespace
Support online-maintenance in hroller
Make hroller take into account the nodes (redundant) instanceswill be migrated to. This be behavior can be overridden by the--offline-maintenance option which will make hroller plan underthe assumption that all instances will be shutdown before starting...
Support construction of the graph of all reboot constraints
For online rolling reboots, there are two kind of restrictions. First,we cannot reboot the primary and secondary nodes of an instancetogether. Secondly, two nodes cannot be rebooted simultaneously, if...
Add option --one-step-only to hroller
Add a new option to hroller to only output information about the firstreboot group. Together with the option --node-tags this allows for thefollowing work flow. First tag all nodes; then repeatedly compute thefirst node group, handle these nodes and remove the tags. In between...
Sort reboot groups by size
Make hroller output the node groups not containing the master nodesorted by size, largest group first. The master node still remainsthe last node of the last reboot group. In this way, most progressis made when switching back to normal cluster operations after the...
Fix lint errors (redundant bracket)
Add option to hroller to select nodes based on tags
Add option --node-tags to tell hroller to consider only nodeswith these tags. A use case would be a tag tracking on whichnodes the maintenance has not yet been carried out, e.g., ifrolling reboots are interleaved with other cluster operations....
Make Rapi backed set node tags correctly
Since the htools representation of a node now allows addingthe node tags, populate this field correctly in the Rapibackend.
Make LUXI backed set node tags correctly
Since the htools representation of a node now allows addingthe node tags, populate this field correctly in the LUXIbackend.
Extend the text format to contain node tags
In order to allow htools to make use of node tags, add them to thetext format. This is done by adding a new column at the end of thenode lines. If this column is missing, the default value (whichis the empty list) is left unchanged, thus yielding the current...
Extend the Node in the htools to allow adding node tags
Since hroller (and probably other tools in the future) will supportnode selection based on node tags, extend the node data structure toallow adding this information.
Make hroller filter the nodes before coloring the graph
Hroller used to first compute a coloring of the node graph and thenfilter out the nodes that it had to work on. While the only filteringwas according to node groups this did not make a difference, as there...
Make mkNodeGraph ignore edges to non-present nodes
Change the behavior of mkNodeGraph to tacitly ignore all instanceswhere one of the nodes is not in the list of nodes. In this way, wecan construct sub-graphs by filtering the nodes and ignoring anypossibly added isolated nodes for the missing indexes....
hspace: Handle multiple ipolicy specs
With tiered allocation, hspace uses all the max specs in turn as theinitial instance spec.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Add multiple min/max specs in instance policy
Now instance policies can contain more than one min/max specs. This is themain element of the "Constrained instance sizes" section in the"Partitioned Ganeti" design doc.
This is a big patch, but changing the type of a configuration item requires...
Extend the simulation backend to also simulate a master node
In a simulated cluster as created by the simulation backend tothe htools, make the first node of the first node group the masternode. In this way, hools (like hroller) that require a master node...
Extend Text format by marking the master node
Sometimes, e.g., for hroller, it is necessary to know which nodeis the master node. Therefore this information has to be includedin the text format as well. Since we never use an offline node asmaster node, we can put this information can be put in the "is...
Make hroller insist on finding precisely one master node
As people rely on the master node being the last node of the lastgroup, make hroller fail, if no master node could be found in thecluster. This happens, e.g., if a backend format is used that does not...
In Rapi, set master correctly
The cluster data contains the information about the masternode. Use this information to set the isMaster bit correctly.
In Luxi, set the master correctly
Utility function to set the master node in a node list
The information about which node is the master node is a cluster-widesetting, in most formats provided independently of the node information.Most backends therefore have to set the isMaster bit indepently in the...
Make Hroller present master node last
If in the list of nodes to be scheduled for maintaince,one is marked as being the master node, schedule itas the last node in the last group.
Extend the node description by isMaster
Extend the description of the node by the propertyof being the master node; also provide and appropriatesetter function. This property will be used, e.g., byhroller to schedule reboot of the master last.
Fix warnings hlint 1.8.43 complained about
These lines are ok according to previous versions of hlint but triggeran error with version 1.8.43.
Signed-off-by: Thomas Thrainer <thomasth@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Merge branch 'devel-2.7'
Make the disks parameter available to the constructor
In that way, tools building on Instance will benefit from the correctedverification semantics of the instance policy on disk space.
Verify individual disks in Instance
Instance policy on disks is specified on a per-disk basis. So extendthe instance description by the sizes of the individual disks and modifythe instance policy verification to correctly check individual disks.