Refactor NodeInfo RPC regarding storage reporting
The NodeInfo RPC call is refactored to handle now more thanjust storage reporting for volume groups.
Since NodeInfo now returns storage space information notnecessarily for volume groups, but also for other storage...
Add storage type to NodeInfo result
So far, the storage information returned from RPC callNodeInfo contained only information about volume groups.In order to extend the storage reporting to other storagetype, we include another field "type" in the result of...
Fix lookup of xen toolstack in Haskell
There was a bug in the haskell implementation of nodequery which made the lookup of the xen toolstack xm/xlfail.
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Thomas Thrainer <thomasth@google.com>
Index nodes by their UUID
No longer index nodes by their name but by their UUID in the clusterconfig. This change changes large parts of the code, as the followingadjustments were necessary: * Change the index key to UUID in the configuration and the ConfigWriter, including all methods....
Add hvparams to RPC call 'node_info'
This patch adds the hvparams parameter to the RPC call'node_info'. It also adjusts the related code in noded.pyand Query/Node.hs
Add type annotation to avoid monomorphism restriction
Even though we need the let-bound variable showMoves onlyat type [(String, String)] -> IO (), it's most general typewould be (PrintfArg a, PrintfArg b) => [(a, b)] -> IO ().This causes the monomorphism restriction apply to that binding,...
add option --print-moves to hroller
If non-redundant instances are present in the cluster, hroller willplan for them to move to other nodes while the group is rebooted.This adds an option to also show this plan.
Signed-off-by: Klaus Aehlig <aehlig@google.com>...
hspace prints info about spindles
Statistics about spindles are tracked. In human-readable output, spindlesare printed only when used (i.e., exclusive storage is enabled). Formachine-oriented output, they are always there.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>...
Add support for shrinking an instance spindles-wise
This makes tiered allocation in hspace work also with respect to spindles.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Spindles become part of htools resource spec
Spindles are now part of resource spec. Instances get created with spindlesspecified (which are just ignored when exclusive storage is disabled).
htools cluster score takes spindles into account
When exclusive storage is enabled, spindles are used instead of disk spaceto compute the cluster score.
Comments and variable names in computePDsk has been changed to match theactual code.
Update spindles when moving instances in htools
Spindles get updated, and errors raised when not enough free spindlesexist. No new error is raised when exclusive storage is disabled.
Unit tests included.
Unit tests for htools and exclusive storage
The existing tests are run also on nodes with exclusive storage enabled. Thevalues for spindles and exclusive storage are set in a consistent way, forboth nodes and instances.
Load complete instance disk information through LUXI
Information about size and spindles of all the disks of an instance is loadedby the LUXI backend, instead of faking one equivalent big disk. In this wayinstance policy checks are more accurate.
Load node spindles data in htools
The data structure for nodes gets a new field for free spindles, and theexisting field for total spindles gets renamed to avoid identifying it withthe node parameter that had the same name. These fields get filled with...
Refactor reading live data in htools
This simplifies different handling of individual items.
Export node spindles
Node spindles (queried live) are exported through the LUXI, RAPI, andiallocator interfaces.
Fix shadowing of library function
The "reads" field shadows a library function from Prelude. This commitfixes the problem.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Let the monitoring daemon provide the diskstats collector
Add the collector to the list of those provided my the monitoring daemon.
Add diskstats data collector
Add a new data collector responsible for gathering disk performancestatistics.
Add a CLI parameter for input files
For many data collectors it is useful (especially for testing) to have topossibility to specify an input file.
This commit adds a generic option for doing that.
Add /proc/diskstats parser
Add a parser for interpreting the content of the /proc/diskstats file,providing information about the state of the disks of the system.
Check real spindles in ipolicies
When exclusive storage is enabled, the spindles in instance disks are usedto check the instance policies (as outlined in design-partitioned.rst).
Check the full instance specs in htools
Spindles and disk count are checked too. Existing functions have beenrefactored, so common parts are not duplicated.
Add spindles to instance disks in htools
A new data type is introduced for disks to store both size and spindles.When available, spindles are filled with input data. Except for loading andstoring, spindles are ignored.
Load exclusive_storage in htools
The node parameter is loaded into the data structures. No behavior is yetmodified.
New function to load JSON arrays of optional values
This will be needed to load spindles in some htools backends. Unit testsprovided.
Add unit test for text backend + fix bug
Test serialization and deserialization of instances. Fix check of secondarynode.
Restrict instance moves in hroller to the same node group
When scheduling rolling reboots, hroller looks for nodes to evacuatethe non-redundant instances to. This is done by greedily movinginstances to other nodes that can take them, policy wise and capacity...
Merge branch 'stable-2.8' into master
Add tests for network-aware allocation
hail-alloc-invalid-network defines a cluster with two nodegroups and anallocation request which does not fit on any of the groups. Group 1 hasinvalid disk-templates while Group 2 is not connected to the rightnetworks....
Honor network connections in hail
Before trying to allocate nodes in node groups, node groups are nowfiltered based on the networks they are connected to an the networkswhich are required by the new instance.
Signed-off-by: Thomas Thrainer <thomasth@google.com>...
Parse NIC data from allocation request in hail
Add a NIC type and extend the Instance type by a list of NIC's. Parsethe NIC's in allocation requests and store them for now. Later patcheswill make use of this field in order to ensure that the requestedinstance is only placed in node groups wich are connected to those...
Support group networks in Text backend
The Text backend now parses network UUID (comma separated) andserializes them in the same form.The test data is adapted to the new format.
Signed-off-by: Thomas Thrainer <thomasth@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Parse node group networks
Extend the Group by the network ids it is connected to. Adaptthe IAlloc backend such that the networks are parsed correctly.This also required the adaption of test data.
Add disks_active to configuration
This flag tracks if the disks of an instace are supposed to be active.That's the case when an instance is running or when its disks gotactivated explicitly (and in a couple of other cases).It will be used by watcher to re-activate disks after a node reboot....
Add spindles field to disk object
The field is filled with the value provided on the command line.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Thomas Thrainer <thomasth@google.com>
hroller: option to ignore non-redundant instances
Add an option to hroller restoring the old behavior on not takingany non-redundant instances into account when forming rebootgroups.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Make hroller also plan for non-redundant instances
Non-redundant instances need to be moved to a different nodebefore maintenance of the node. Even though they can be moved toany node, there must be enough capacity to host the instances of thereboot group to be evacuated....
hroller: option to skip nodes with non-redundant instances
So far, hroller ignores the fact, that non-redundant instances exist.One option to deal is non-redundant instances is to not schedule thosenodes for reboot. This is supported by adding the option --skip-non-redundant....
Remove trailing whitespace
RPC 'node_info': <storage_type,key> instead of vg_names
This replaces the field 'vg_names' in the RPC call of 'node info' by'storage_units'. A storage unit is a tuple <storage_type,key>and a generalization of a vg_name. The list of vg names is replaced by...
Make HS ConfD client IPv6 compatible
The Haskell ConfD client was assuming internet addresses to be IPv4. Thispatch modifies the client so that it is able to automatically detect theprotocol it should use by analyzing the address it is told to connect to....
Factor out resolveAddr function
This function can be useful to many parts of the code to convert the stringrepresentation of an IP (v4 or v6) address into the proper data type.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Use dcName in mon-collector
Instead of manually specify the name of the data collectors in mon-collector,just use the dcName field each of them exports.
Factor out the mergeStatuses function
It will be used by multiple data collectors, not only the DRBD collector.
Add global status field to the instance status collector
The global status is computed from the statuses of the single instances.
The output json format is adapted to include this piece of information, asprescribed by the design document.
Signed-off-by: Michele Tartara <mtartara@google.com>...
Export the Instance Status collector report
It will need to be accessed by the monitoring daemon.
Add inst-status-xen to the monitoring daemon
Enable the monitoring daemon to invoke the Xen instance status data collector.Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Add HS functions for getting the instance reason path
The getInstReasonFilename is built to resemble the python correspondingfunction.
Add module containing function for getting info from Xen
The Xen instance status data collector will require to get some informationfrom the hypervisor. This commit introduces a module providing such functions.
Add the core of the instance status collector
Add the Xen instance status data collector with only its core features.The next commits will add more reporting functionalities.
The access to the collector is made possible through the mon-collectortool....
Export the actual instance state
Compute the actual state of the instance and export it.
Determine status of one instance
Added function for determining whether the status of an instance is ok, and torepresent this information in the corresponding field in the report.
Include the reason trail in the instance collector output
Fetch the reason trail from file, failing gracefully if it is not found, andinclude it in the output of the instance status data collector.
Export Instance Status collector information
Name, version, format version, category and kind of the Instance Status datacollector are now exported.
Factor out function for building report
Instead of building the report as part of the "Main" function, have itbuilt by its own dedicated function, so that it will be able to export itdirectly to the monitoring daemon when needed.
Support online-maintenance in hroller
Make hroller take into account the nodes (redundant) instanceswill be migrated to. This be behavior can be overridden by the--offline-maintenance option which will make hroller plan underthe assumption that all instances will be shutdown before starting...
Support construction of the graph of all reboot constraints
For online rolling reboots, there are two kind of restrictions. First,we cannot reboot the primary and secondary nodes of an instancetogether. Secondly, two nodes cannot be rebooted simultaneously, if...
Add option --one-step-only to hroller
Add a new option to hroller to only output information about the firstreboot group. Together with the option --node-tags this allows for thefollowing work flow. First tag all nodes; then repeatedly compute thefirst node group, handle these nodes and remove the tags. In between...
Sort reboot groups by size
Make hroller output the node groups not containing the master nodesorted by size, largest group first. The master node still remainsthe last node of the last reboot group. In this way, most progressis made when switching back to normal cluster operations after the...
Fix lint errors (redundant bracket)
Add option to hroller to select nodes based on tags
Add option --node-tags to tell hroller to consider only nodeswith these tags. A use case would be a tag tracking on whichnodes the maintenance has not yet been carried out, e.g., ifrolling reboots are interleaved with other cluster operations....
Make Rapi backed set node tags correctly
Since the htools representation of a node now allows addingthe node tags, populate this field correctly in the Rapibackend.
Make LUXI backed set node tags correctly
Since the htools representation of a node now allows addingthe node tags, populate this field correctly in the LUXIbackend.
Extend the text format to contain node tags
In order to allow htools to make use of node tags, add them to thetext format. This is done by adding a new column at the end of thenode lines. If this column is missing, the default value (whichis the empty list) is left unchanged, thus yielding the current...
Extend the Node in the htools to allow adding node tags
Since hroller (and probably other tools in the future) will supportnode selection based on node tags, extend the node data structure toallow adding this information.
Make hroller filter the nodes before coloring the graph
Hroller used to first compute a coloring of the node graph and thenfilter out the nodes that it had to work on. While the only filteringwas according to node groups this did not make a difference, as there...
Make mkNodeGraph ignore edges to non-present nodes
Change the behavior of mkNodeGraph to tacitly ignore all instanceswhere one of the nodes is not in the list of nodes. In this way, wecan construct sub-graphs by filtering the nodes and ignoring anypossibly added isolated nodes for the missing indexes....
Merge stable-2.7 into stable-2.8
Conflicts: INSTALL NEWS configure.ac src/Ganeti/Query/Group.hs test/hs/Test/Ganeti/Objects.hs test/hs/Test/Ganeti/Query/Query.hs
Add reason trail to haskell opcode format
The haskell type definition of opcodes should remain aligned with the pythonone.
Add the reason trail type definition in Haskell
It will be added to the haskell definition of opcodes, to keep it aligned tothe python one, and it will be used for fetching the reason trail by theinstance status data collector.
Remove outdated TODO comment
Add proper error management to monitoring daemon
Produce a personalized 404 error when the requested resource is not available.
Produce a report for all the data collectors
Implement the API function of the monitoring daemon that provides the reportof all the data collectors.
Generate report for a single collector
Allow to ask the monitoring daemon for the report of one specific datacollector.
Export the full report from DRBD data collector
Export the full report instead of just the data from the DRBD data collector.
Update JSON serialization for "category"
Change the JSON serialization for the "category" field of data collectors,in accordance to the latest version of the design document.
Implement the "list collectors" API in monitoring
Implement the handler for outputting the list of collectors(name, category, kind) in JSON format.
Fix bug in group queries related to node/instance fields
Since we use the primitive string type for group UUIDs, the groupfields have a bug where we pass the group name as filter for nodetests, whereas the nodes themselves use the group UUID. This results...
hspace: Handle multiple ipolicy specs
With tiered allocation, hspace uses all the max specs in turn as theinitial instance spec.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Add multiple min/max specs in instance policy
Now instance policies can contain more than one min/max specs. This is themain element of the "Constrained instance sizes" section in the"Partitioned Ganeti" design doc.
This is a big patch, but changing the type of a configuration item requires...
Log at info level the success of a luxi query
So far, any successful query was only logged at debug level, thus leavingno traces in the logs, unless debugging was enabled. Therefore, we adda notes about successful completion at info level. Also, repeat a failed...
Fix typo in documentation string
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Limit specs in instance policies are always complete
Specs used to specify limits in instance policies are always complete, evenat group level (only the whole limit set can be overridden). This is inpreparation for introducing multiple limits.
Extend the simulation backend to also simulate a master node
In a simulated cluster as created by the simulation backend tothe htools, make the first node of the first node group the masternode. In this way, hools (like hroller) that require a master node...
Extend Text format by marking the master node
Sometimes, e.g., for hroller, it is necessary to know which nodeis the master node. Therefore this information has to be includedin the text format as well. Since we never use an offline node asmaster node, we can put this information can be put in the "is...
Remove old "reason" implementation
Remove the useless parts of the old, partial, implementation of the support fortracking the reason of instances state change, before implementing the newreason trail support, as per the design document.
Make hroller insist on finding precisely one master node
As people rely on the master node being the last node of the lastgroup, make hroller fail, if no master node could be found in thecluster. This happens, e.g., if a backend format is used that does not...
In Rapi, set master correctly
The cluster data contains the information about the masternode. Use this information to set the isMaster bit correctly.
In Luxi, set the master correctly
Utility function to set the master node in a node list
The information about which node is the master node is a cluster-widesetting, in most formats provided independently of the node information.Most backends therefore have to set the isMaster bit indepently in the...
Make Hroller present master node last
If in the list of nodes to be scheduled for maintaince,one is marked as being the master node, schedule itas the last node in the last group.
Extend the node description by isMaster
Extend the description of the node by the propertyof being the master node; also provide and appropriatesetter function. This property will be used, e.g., byhroller to schedule reboot of the master last.
Fix warnings hlint 1.8.43 complained about
These lines are ok according to previous versions of hlint but triggeran error with version 1.8.43.
Signed-off-by: Thomas Thrainer <thomasth@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Add nic/disk name and UUID fields in Haskell code
Add name and uuid fields to Disk and nic Haskell objects. Also,since they now have an UUID, make them instances of UuidObject.
Signed-off-by: Christor Stavrakakis <cstavr@grnet.gr>Reviewed-by: Helga Velroyen <helgav@google.com>
Add name to INicParams and IDiskParams
This commits adds name to NIC and Disk modification definition.
Signed-off-by: Chris Stavrakakis <cstavr@grnet.gr>Reviewed-by: Helga Velroyen <helgav@google.com>
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Merge branch 'devel-2.7'
Removes the 'sharedfile' storage type
Since storage type are supposed to represent the underlying technologyof disk templates, the storage type 'sharedfile' is superfluous, becausetechnically both disk templates 'file' and 'sharedfile' use the file system....