Adjust all callers of RPC 'call_node_info'
This patch contains all adjustments necessary to code thatuses the 'call_node_info' call. The adjustment is necessarybecause the call's signature changed from receiving a listof hypervisor names to a list of tuples...
Add hvparams to RPC call 'node_info'
This patch adds the hvparams parameter to the RPC call'node_info'. It also adjusts the related code in noded.pyand Query/Node.hs
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Thomas Thrainer <thomasth@google.com>
backend.py: use hvparams in GetNodeInfo
This patch extends the GetNodeInfo function of the backendmodule by a hvparams parameter. For now, it contains afallback solution for when the parameter is None. Thisfallback will be removed in a later patch of this series....
hypervisors: add hvparams to GetNodeInfo
This patch extends the GetNodeInfo function of thehypervisors by a hvparams parameter. The parameteris currently only used by the xen hypervisor to determinewhich xen toolstack ('xm' or 'xl') to use.
Signed-off-by: Helga Velroyen <helgav@google.com>...
hv_xen.py: renaming residual 'xm' occurrences
Some methods still refered to 'xm', which is inaccurate nowthat xen can also be run with 'xl'. No functional changesotherwise.
Watcher: Use hvparams when listing instances
This patch makes the watcher use the hvparams from ssconfto retrieve the list of instances from the hypervisors.
Include hvparams in ssconf files
The watcher needs access to the cluster's hypervisorparameters. This patch extends the ssconf infrastructureby adding a hvparams file for each hypervisor. The formatof the hvparams file is a simple key=value format.
Use hvparams in instance migration
This patch makes use of hvparams on instance migration todetermine the list of instance before the migration. This way, inxen, the xl/xm choice is respected.
Verify: node info and instance list
This patch enhances node verification by:- Factoring out the verification of the node info and the instance list to increase testability.- Make both of them use the hvparams parameter.
Verify: xen toolstack, hypervisor and hvparams
This patch extends the node verification by:- Adding a check for the xen toolstack when the hypervisor is verified.- Factoring out the hypervisor verification in a subfunction to increase testability.- Factoring out the hvparams verification in a...
rpc / noded: add hvparams to instance_list call
The 'instance_list' call needs to be extended to accepthvparams. This is one of the few RPCs where it isnecessary, because there is no instance given whosehvparams could be reused.
backend.py: use instance's hvparams whenever possible
The backend module's function 'ListInstances' is calledwithin the module several times. In many cases, it ispossible to reuse the instance's hvparams for the'ListInstances' call and thus avoiding having to extend...
backend.py: GetInstanceListForHypervisor
In same cases, the call 'GetInstanceList' is only done forone hypervisor. In those cases, it is more convenient tonot provide the full collection of hypervisor parametersfor all hypervisors, but only the ones for this hypervisor....
backend.py: hvparams in 'GetInstanceList'
Propagating the use of hvparams further up the callinghierarchy, this patch makes the backend module use thehvparams in 'GetInstanceList'. Unit tests provided.
hv_xen.py: _RunXen and GetInstanceList use hvparams
Propagating the use of hvparams further up in the callinghierarchy, the functions '_RunXen' and 'GetInstanceList'use the given hvparams. Unit tests provided. Calling function'StopInstance' adjusted in this patch as well....
hv_xen.py: rename RunXmList to RunInstanceList
Since one can retrieve xen's instance list with both, 'xm'or 'xl', the function name 'RunXmList' is no longerappropriate. Renaming it to 'RunInstanceList'. Nofunctional changes otherwise.
hv_xen.py: rename GetXmList to GetInstanceList
Since one can retrieve xen's instance list via both 'xm'or 'xl', it is no longer appropriate to call the function'GetXmList'. This patch renames it to 'GetInstanceList'.Other than that, there are no functional changes....
hv_xen._GetCommand: retrieve xen command from hvparams
This patch adds a (so far optional) hvparams parameterto the '_GetCommand' function. This dictionary is usedto retrieve the xen command (xm or xl). It is optionalfor now to make the refactoring possible without breaking...
Unit tests for hv_xen._GetCommand
This patch adds unit tests for the current state of the'GetCommand' method. The tests are rather trivial, buthaving tests before starting to refactor seems to bea good idea.
Add xen command (xm, xl) to hv parameters
This patch adds the xen command (xm, xl) as hypervisorparameter to the constants and adds validation tothe xen hypervisors.
Fixing docstring of hv_xen_unittest.py
The introductory comment of hv_xen_unittest.py referred tothe wrong hypervisor.
Make SubmitOpCode expect options without print_jobid
As opposed to SubmitOrSend, the SubmitOpCode function iscalled also by commands not supporting submit options. Henceonly inspect these options, if they are actually present.
Signed-off-by: Klaus Aehlig <aehlig@google.com>...
Add an option to print job id of submitted jobs on stdout
When submitting a job, the job id, prefixed by 'JobID: ', is output tostderr. While this is easy to understand for humans, for scripting, itis more convenient to have (additionally) the id on stdout (and only...
Add new command gnt_job wait
This command tacitly waits for a job to finish. In thatway, job dependency can also be handled in shell scripts.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Add type annotation to avoid monomorphism restriction
Even though we need the let-bound variable showMoves onlyat type [(String, String)] -> IO (), it's most general typewould be (PrintfArg a, PrintfArg b) => [(a, b)] -> IO ().This causes the monomorphism restriction apply to that binding,...
add option --print-moves to hroller
If non-redundant instances are present in the cluster, hroller willplan for them to move to other nodes while the group is rebooted.This adds an option to also show this plan.
New tests for hspace and exclusive storage
hspace is run on clusters with exclusive storage enabled.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
hspace prints info about spindles
Statistics about spindles are tracked. In human-readable output, spindlesare printed only when used (i.e., exclusive storage is enabled). Formachine-oriented output, they are always there.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>...
Add support for shrinking an instance spindles-wise
This makes tiered allocation in hspace work also with respect to spindles.
Spindles become part of htools resource spec
Spindles are now part of resource spec. Instances get created with spindlesspecified (which are just ignored when exclusive storage is disabled).
htools cluster score takes spindles into account
When exclusive storage is enabled, spindles are used instead of disk spaceto compute the cluster score.
Comments and variable names in computePDsk has been changed to match theactual code.
Update spindles when moving instances in htools
Spindles get updated, and errors raised when not enough free spindlesexist. No new error is raised when exclusive storage is disabled.
Unit tests included.
Unit tests for htools and exclusive storage
The existing tests are run also on nodes with exclusive storage enabled. Thevalues for spindles and exclusive storage are set in a consistent way, forboth nodes and instances.
Load complete instance disk information through LUXI
Information about size and spindles of all the disks of an instance is loadedby the LUXI backend, instead of faking one equivalent big disk. In this wayinstance policy checks are more accurate.
Load node spindles data in htools
The data structure for nodes gets a new field for free spindles, and theexisting field for total spindles gets renamed to avoid identifying it withthe node parameter that had the same name. These fields get filled with...
Refactor reading live data in htools
This simplifies different handling of individual items.
Export node spindles
Node spindles (queried live) are exported through the LUXI, RAPI, andiallocator interfaces.
node_info RPC reports info about free spindles
node_info RPC can be used to get information about free/total spindles.
Update htools text-backend documentation
Both instances and nodes have new fields.
Use RpcResult.Warn where appropriate the cmdlib
To avoid repetitive code, make use of the functionality factoredout to the Warn method of RpcResult.
Add a Warn function to RpcResult
At various places, LU code just wants to warn abouta failed rpc, but continue nevertheless. So factor outthat common checking, message formatting, and notifying.
Fix shadowing of library function
The "reads" field shadows a library function from Prelude. This commitfixes the problem.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Add shelltests for diskstats
Check that the behavior of running the diskstats data collector is asexpected.
Let the monitoring daemon provide the diskstats collector
Add the collector to the list of those provided my the monitoring daemon.
Add diskstats data collector
Add a new data collector responsible for gathering disk performancestatistics.
Add constant with the path of the diskstats
This will be used by the diskstats storage collector.
Add a CLI parameter for input files
For many data collectors it is useful (especially for testing) to have topossibility to specify an input file.
This commit adds a generic option for doing that.
Add unit tests for the Diskstats parser
Add one test parsing an actual /proc/diskstats file and one QuickChecktest.
Add function for generating arbitrary non-negative numbers
In many tests it is useful to have the possibility to easily generatenon-negative integer numbers.
Factor out and rename testFile function
The testFile function used in the DRBD parser test file can actually beuseful in any case where a parser has to be tested.
This patch moves it to the TestCommon file, and renames it to a more meaningfultestParser....
Fix erroneous docstring
Fix erroneously ordered files in Makefile.am
Add /proc/diskstats parser
Add a parser for interpreting the content of the /proc/diskstats file,providing information about the state of the disks of the system.
Add design for diskstats data collector
This patch adds the design for the lowest level storage collector,gathering data about disk usage statistics.
Modify storage collector design
Specify better how the storage collectors will actually be implemented.
Unit tests for spindles and exclusive storage in htools
Everything should work the same when exclusive storage is active, asspindles are not yet handled as a resource. A new test has been added forthe interactions between spindles and policies.
Check real spindles in ipolicies
When exclusive storage is enabled, the spindles in instance disks are usedto check the instance policies (as outlined in design-partitioned.rst).
Check the full instance specs in htools
Spindles and disk count are checked too. Existing functions have beenrefactored, so common parts are not duplicated.
Add spindles to instance disks in htools
A new data type is introduced for disks to store both size and spindles.When available, spindles are filled with input data. Except for loading andstoring, spindles are ignored.
Export instance disk spindles
Instance disk spindles can be queried via LUXI and RAPI, and are exportedthrough the allocator interface. This is a prerequisite for htools tohandle spindles.
The length of a RAPI query including all the instance fields now has become...
Load exclusive_storage in htools
The node parameter is loaded into the data structures. No behavior is yetmodified.
New function to load JSON arrays of optional values
This will be needed to load spindles in some htools backends. Unit testsprovided.
Add unit test for text backend + fix bug
Test serialization and deserialization of instances. Fix check of secondarynode.
Test with non-redundant instances and several node-groups
The example cluster contains two node groups. The first contains three nodes,with two non-redundant instances each. As each node has only capacity for threeinstances, the nodes have to be rebooted individually. Additionally, there is...
Restrict instance moves in hroller to the same node group
When scheduling rolling reboots, hroller looks for nodes to evacuatethe non-redundant instances to. This is done by greedily movinginstances to other nodes that can take them, policy wise and capacity...
Merge branch 'stable-2.8' into master
Move an item into the correct section of NEWS
The removal of '--no-lvm-storage' happened on the master/2.9 branch.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Merge branch 'stable-2.7' into stable-2.8
Conflicts: doc/iallocator.rst...
kvm: Use -uuid to set system UUID
Use kvm '-uuid' option to set guest's system UUID to instance's UUID.
Signed-off-by: Christos Stavrakakis <cstavr@grnet.gr>Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Document users-setup tool
The users-setup tool was added but had no documentation up to now. This commitfixes this issue.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Support spindles in move-instance
The script was failing when exclusive storage was active (bug introduced in7c848a6a).
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Adjusting permissions after confd start
This is a workaround for issue 477. Confd resets thepermissions of the query socket in a wrong way. Thispatch fixes them after the start of confd.
Ensure the queue socket has the right permissions
RAPI needs to be able to perform queries as well.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Update IAllocator interface documentation
Update the missing documentation for the tags fields of node groups.
Signed-off-by: Thomas Thrainer <thomasth@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>(partially cherry picked from commit 2daca99b5be21ea4667637721ee2c65a14444520)...
Add NEWS entry for hail honoring networks
The entry also mentions the required change in the IAllocator protocol.
Signed-off-by: Thomas Thrainer <thomasth@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Add tests for network-aware allocation
hail-alloc-invalid-network defines a cluster with two nodegroups and anallocation request which does not fit on any of the groups. Group 1 hasinvalid disk-templates while Group 2 is not connected to the rightnetworks....
Honor network connections in hail
Before trying to allocate nodes in node groups, node groups are nowfiltered based on the networks they are connected to an the networkswhich are required by the new instance.
Signed-off-by: Thomas Thrainer <thomasth@google.com>...
Parse NIC data from allocation request in hail
Add a NIC type and extend the Instance type by a list of NIC's. Parsethe NIC's in allocation requests and store them for now. Later patcheswill make use of this field in order to ensure that the requestedinstance is only placed in node groups wich are connected to those...
Support group networks in Text backend
The Text backend now parses network UUID (comma separated) andserializes them in the same form.The test data is adapted to the new format.
Parse node group networks
Extend the Group by the network ids it is connected to. Adaptthe IAlloc backend such that the networks are parsed correctly.This also required the adaption of test data.
Document the newly added network field of node groups, and also updatethe missing documentation for the other fields of node groups.
Export connected networks to IAllocator
IAllocators should not put instances on nodes in nodegroups which arenot connected to the network the instance should be connected to.Therefore, export the networks a node group is connected to to theIAllocator, so it can account for this restriction....
Add generic daemon options support binding to interfaces
In some situations, it is desirable to bind one of our servers only toa particular interface, like eth0, instead of an IP address. Therefore,add an option -i to specify an interface; its address will then be bound...
Update ganeti-rapi man page
ganeti-rapi supports a couple of options not yet mentioned in theman page; mention them.
Complete ganeti-noded synopsis line
The synopsis section of the ganeti-noded man page, certain optionswere only mentioned in the description. Add then to the synopsisline as well.
Re-activate previously active disks in watcher
The watcher process previously (incorrectly) examined the admin_statefield to figure out if the disks of an instance should be activatedafter a node reboot. This is changed to use the disks_active field of...
Make the disks_active flag queryable
gnt-instance list now also supports the disks_active field.
Signed-off-by: Thomas Thrainer <thomasth@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Consider disks_active in UpgradeConfig
The disks_active flag is updated when masterd reads the configuration.Also, cfgupgrade now removes the disks_active flag during downgrades.
Add disks_active to configuration
This flag tracks if the disks of an instace are supposed to be active.That's the case when an instance is running or when its disks gotactivated explicitly (and in a couple of other cases).It will be used by watcher to re-activate disks after a node reboot....
Report missing spindles in the configuration
cfgupgrade and cluster-verify run a check for missing spindles in diskconfiguration when exclusive storage is enabled. The check in cfgupgrademay give false positives due to the complexity of getting the correct value...
Adapt striping to the requested number of spindles
Otherwise LVM may use a smaller number of PVs (spindles) to accommodate forthe default stripes.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Spindles are mandatory with exclusive storage
If they are not specified in the command line an error is reported. Also,disk creation would fail without them.
QA has been updated.
gnt-cluster repair-disk-sizes repairs spindles too
When exclusive storage is active, any wrong or missing spindles informationin disks gets updated too.
New RPC to get size and spindles of disks
This RPC replaces the existing one that only returned disk size.
New BlockDev methods to get spindles
Two new methods are created to get the number of spindles, one alone, andone together with size. For devices that don't support spindles, None isreturned.
Attaching a logical volume builds a list of PVs
When an LV gets attached, the list of the PVs used by the LV is built. Thiswill be used to count spindles for exclusive_storage, but it could also beuseful to optimize disk growing and snapshotting.
Refactor code for attaching a logical volume
The parsing of "lvs" output is moved into private methods. The code isslightly more readable and testable. The split in two methods is usefulfor the following patches. Unit tests for the new functions areprovided....
Fix release of instances in QA
Also, fix some uses of keyword arguments for arguments with a defaultvalue, as required by the style guide, and a typo.
Fix missing "raise" in QA
The bug was introduced in 345d395d.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Thomas Thrainer <thomasth@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Merge branch 'stable-2.8'