Add /proc/diskstats parser
Add a parser for interpreting the content of the /proc/diskstats file,providing information about the state of the disks of the system.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Add design for diskstats data collector
This patch adds the design for the lowest level storage collector,gathering data about disk usage statistics.
Modify storage collector design
Specify better how the storage collectors will actually be implemented.
Unit tests for spindles and exclusive storage in htools
Everything should work the same when exclusive storage is active, asspindles are not yet handled as a resource. A new test has been added forthe interactions between spindles and policies.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>...
Check real spindles in ipolicies
When exclusive storage is enabled, the spindles in instance disks are usedto check the instance policies (as outlined in design-partitioned.rst).
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Check the full instance specs in htools
Spindles and disk count are checked too. Existing functions have beenrefactored, so common parts are not duplicated.
Add spindles to instance disks in htools
A new data type is introduced for disks to store both size and spindles.When available, spindles are filled with input data. Except for loading andstoring, spindles are ignored.
Export instance disk spindles
Instance disk spindles can be queried via LUXI and RAPI, and are exportedthrough the allocator interface. This is a prerequisite for htools tohandle spindles.
The length of a RAPI query including all the instance fields now has become...
Load exclusive_storage in htools
The node parameter is loaded into the data structures. No behavior is yetmodified.
New function to load JSON arrays of optional values
This will be needed to load spindles in some htools backends. Unit testsprovided.
Add unit test for text backend + fix bug
Test serialization and deserialization of instances. Fix check of secondarynode.
Test with non-redundant instances and several node-groups
The example cluster contains two node groups. The first contains three nodes,with two non-redundant instances each. As each node has only capacity for threeinstances, the nodes have to be rebooted individually. Additionally, there is...
Restrict instance moves in hroller to the same node group
When scheduling rolling reboots, hroller looks for nodes to evacuatethe non-redundant instances to. This is done by greedily movinginstances to other nodes that can take them, policy wise and capacity...
Merge branch 'stable-2.8' into master
Move an item into the correct section of NEWS
The removal of '--no-lvm-storage' happened on the master/2.9 branch.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Merge branch 'stable-2.7' into stable-2.8
Conflicts: doc/iallocator.rst...
kvm: Use -uuid to set system UUID
Use kvm '-uuid' option to set guest's system UUID to instance's UUID.
Signed-off-by: Christos Stavrakakis <cstavr@grnet.gr>Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Document users-setup tool
The users-setup tool was added but had no documentation up to now. This commitfixes this issue.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Support spindles in move-instance
The script was failing when exclusive storage was active (bug introduced in7c848a6a).
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Adjusting permissions after confd start
This is a workaround for issue 477. Confd resets thepermissions of the query socket in a wrong way. Thispatch fixes them after the start of confd.
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Thomas Thrainer <thomasth@google.com>
Ensure the queue socket has the right permissions
RAPI needs to be able to perform queries as well.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Update IAllocator interface documentation
Update the missing documentation for the tags fields of node groups.
Signed-off-by: Thomas Thrainer <thomasth@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>(partially cherry picked from commit 2daca99b5be21ea4667637721ee2c65a14444520)...
Add NEWS entry for hail honoring networks
The entry also mentions the required change in the IAllocator protocol.
Signed-off-by: Thomas Thrainer <thomasth@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Add tests for network-aware allocation
hail-alloc-invalid-network defines a cluster with two nodegroups and anallocation request which does not fit on any of the groups. Group 1 hasinvalid disk-templates while Group 2 is not connected to the rightnetworks....
Honor network connections in hail
Before trying to allocate nodes in node groups, node groups are nowfiltered based on the networks they are connected to an the networkswhich are required by the new instance.
Signed-off-by: Thomas Thrainer <thomasth@google.com>...
Parse NIC data from allocation request in hail
Add a NIC type and extend the Instance type by a list of NIC's. Parsethe NIC's in allocation requests and store them for now. Later patcheswill make use of this field in order to ensure that the requestedinstance is only placed in node groups wich are connected to those...
Support group networks in Text backend
The Text backend now parses network UUID (comma separated) andserializes them in the same form.The test data is adapted to the new format.
Parse node group networks
Extend the Group by the network ids it is connected to. Adaptthe IAlloc backend such that the networks are parsed correctly.This also required the adaption of test data.
Document the newly added network field of node groups, and also updatethe missing documentation for the other fields of node groups.
Export connected networks to IAllocator
IAllocators should not put instances on nodes in nodegroups which arenot connected to the network the instance should be connected to.Therefore, export the networks a node group is connected to to theIAllocator, so it can account for this restriction....
Add generic daemon options support binding to interfaces
In some situations, it is desirable to bind one of our servers only toa particular interface, like eth0, instead of an IP address. Therefore,add an option -i to specify an interface; its address will then be bound...
Update ganeti-rapi man page
ganeti-rapi supports a couple of options not yet mentioned in theman page; mention them.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Complete ganeti-noded synopsis line
The synopsis section of the ganeti-noded man page, certain optionswere only mentioned in the description. Add then to the synopsisline as well.
Re-activate previously active disks in watcher
The watcher process previously (incorrectly) examined the admin_statefield to figure out if the disks of an instance should be activatedafter a node reboot. This is changed to use the disks_active field of...
Make the disks_active flag queryable
gnt-instance list now also supports the disks_active field.
Signed-off-by: Thomas Thrainer <thomasth@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Consider disks_active in UpgradeConfig
The disks_active flag is updated when masterd reads the configuration.Also, cfgupgrade now removes the disks_active flag during downgrades.
Add disks_active to configuration
This flag tracks if the disks of an instace are supposed to be active.That's the case when an instance is running or when its disks gotactivated explicitly (and in a couple of other cases).It will be used by watcher to re-activate disks after a node reboot....
Report missing spindles in the configuration
cfgupgrade and cluster-verify run a check for missing spindles in diskconfiguration when exclusive storage is enabled. The check in cfgupgrademay give false positives due to the complexity of getting the correct value...
Adapt striping to the requested number of spindles
Otherwise LVM may use a smaller number of PVs (spindles) to accommodate forthe default stripes.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Spindles are mandatory with exclusive storage
If they are not specified in the command line an error is reported. Also,disk creation would fail without them.
QA has been updated.
gnt-cluster repair-disk-sizes repairs spindles too
When exclusive storage is active, any wrong or missing spindles informationin disks gets updated too.
New RPC to get size and spindles of disks
This RPC replaces the existing one that only returned disk size.
New BlockDev methods to get spindles
Two new methods are created to get the number of spindles, one alone, andone together with size. For devices that don't support spindles, None isreturned.
Attaching a logical volume builds a list of PVs
When an LV gets attached, the list of the PVs used by the LV is built. Thiswill be used to count spindles for exclusive_storage, but it could also beuseful to optimize disk growing and snapshotting.
Refactor code for attaching a logical volume
The parsing of "lvs" output is moved into private methods. The code isslightly more readable and testable. The split in two methods is usefulfor the following patches. Unit tests for the new functions areprovided....
Fix release of instances in QA
Also, fix some uses of keyword arguments for arguments with a defaultvalue, as required by the style guide, and a typo.
Fix missing "raise" in QA
The bug was introduced in 345d395d.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Thomas Thrainer <thomasth@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Merge branch 'stable-2.8'
Bump up version for 2.7.0~rc2 release
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
QA for spindles in creating disks
"gnt-instance add" and "gnt-instance recreate-disks" are tested giving thenumber of spindles, when supported. Also, QA for "gnt-instancerecreate-disks" now covers the case where disks are resized.
Create overall design document for 2.8
Also, cleanup the list of draft designs.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
cfgupgrade: Remove spindles on downgrade
Spindles didn't exist in 2.8 and should be removed to downgrade.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Thomas Thrainer <thomasth@google.com>
gnt-instance info prints information about spindles
For each disk, the number of requested spindles (if present) is shown.
Spindles are used in creating new disks
The requested number of spindles is used to allocate PVs when creating newLVs.
Document disk device Create() functions
Document all the arguments and return values of bdev.Create() andbdev.BlockDev.Create().
Add spindles field to disk object
The field is filled with the value provided on the command line.
Check that spindles are not given when they are not allowed
Masterd checks that specifications for new disks don't include spindleswhen exclusive storage is disabled.
Add "spindles" option to instance disk creation commands
The option is parsed but ignored, for the moment.
Fix style inconsistencies in gnt-instance man page
Escaping and initial capitals were not uniform.
cfgupgrade: Downgrade is a NO-OP
The configuration is still the same as in 2.8 (the reference stable versionfor this branch), so downgrade shouldn't do anything.
Unit tests are also updated, with a new 2.8 configuration file. Theconfiguration file used for the upgrade+downgrade test was tailored to the...
Add NEWS entry for SO_PEERCRED fix
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Thomas Thrainer <thomasth@google.com>
Design doc for internal shutdown detection
Ganeti is currently not able to detect a legit shutdown request performed by auser from inside a Xen domain.
This patch provides a design document to implement a mechanism able to cope withsuch events.
Signed-off-by: Michele Tartara <mtartara@google.com>...
Document recent hroller changes in the NEWS file
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Document hroller options recently added
hroller now also supports the options --skip-non-redundant and--ignore-non-redundant, and this should be documented in theman page as well.
While there, also use the same order in the options sectionas in the synopsis, and in the synopsis group the algorithms...
Workaround missing SO_PEERCRED
This bug happens in a few new distributions, so we workaround it bydefining the constant ourselves, if it's missing.
Add debugging clause to _ExpandCheckDisks error
This has been reported by users, so we should have the extra debuggingavailable.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Reduce pylint maximum file length to 4500
The longest Python files we still have are around 4200 lines long. Inorder to prevent future growth, limit the maximum file length (checkedby pylint) to 4500 lines.
Extend hroller tests by options for non-redundant instances
The cluster now consists of 3 nodes, with drbd instances betweennodes 1 and 2, and 2 and 3. Additionally, nodes 1 and 3 each containa non-redundant instance, but node 2 cannot hold two additional...
Mention hail network incompatibility in manpages
We can't fix this in the 2.7 version, so it should be documented.
Remove obsolete Debian-related documentation
This part of the documentation refers to Grub instead of Grub2, butGrub2 has been the standard boot loader since Squeeze. As this part only(wrongly) repeats the preceeding documentation, it's removed completely....
Update NEWS for 2.7.0 rc2
Move "local" entries at the bottom, and leave global 2.7 entries at thetop, as for the other releases.
Test for hroller taking non-redundant instances into account
The example cluster consists of 6 nodes, each hosting 2 instances andhaving capacity for 3. So, while the drbd-induced graph consists ofonly insulated nodes, no more than two nodes can be rebooted at the...
hroller: option to ignore non-redundant instances
Add an option to hroller restoring the old behavior on not takingany non-redundant instances into account when forming rebootgroups.
Make hroller also plan for non-redundant instances
Non-redundant instances need to be moved to a different nodebefore maintenance of the node. Even though they can be moved toany node, there must be enough capacity to host the instances of thereboot group to be evacuated....
hroller: option to skip nodes with non-redundant instances
So far, hroller ignores the fact, that non-redundant instances exist.One option to deal is non-redundant instances is to not schedule thosenodes for reboot. This is supported by adding the option --skip-non-redundant....
Remove trailing whitespace
Improve installation documentation
Based on user feedback the installation documentation is clarified andextended.
Signed-off-by: Thomas Thrainer <thomasth@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>(cherry picked from commit 3913eaa72b8a48706ca4caa59dd022fa1a05805a)...
Signed-off-by: Thomas Thrainer <thomasth@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
RPC 'node_info': <storage_type,key> instead of vg_names
This replaces the field 'vg_names' in the RPC call of 'node info' by'storage_units'. A storage unit is a tuple <storage_type,key>and a generalization of a vg_name. The list of vg names is replaced by...
Add Harep man page
Add the man page for the Ganeti auto-repair tool.
Stash Xen config file after a failed startup
This is a fix for a minor bug.Currently, a failed Xen VM start results in a stale config file leftbehind on the filesystem.This change introduces a new log directory, where the Xen VM configfile is moved after a failed startup....
Fix owner of the OS log dir
The OS log dir is used by noded, not masterd.
Signed-off-by: Balazs Lecz <leczb@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Update NEWS with disk creation fixes
Also document a couple more fixes.
Sort cmdlib-related entries in Makefile.am
Files in the cmdlib directory are sorted alphabetically inMakefile.am.
Signed-off-by: Thomas Thrainer <thomasth@google.com>Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
cmdlib: Cleanup public/private functions
All functions/classes which are used outside of their defining module(with tests as an exception) no longer have a leading underscore.
cmdlib: Extract instance query related functionality
Split instance.py further by extracting instance querying relatedlogical units and functions to instance_query.py.
cmdlib: Extract instance operation functionality
Split instance.py further by extracting instance operations(start/stop/reboot/etc.) related logical units and functions toinstance_operation.py.
The extracted operations have in common that they affect the operating...
cmdlib: Extract migration related functionality
Split instance.py further by extracting migration related logical unitsand functions to instance_migration.py.
cmdlib: Extract storage related functionality
Split instance.py further by extracting storage related logical unitsand functions to instance_storage.py.
Reformat and define exports in cmdlib/__init__.py
cmdlib/__init__.py now simply defines the interface of the cmdlib moduleby importing all classes which should be visible to clients.
Also don't ignore C0302 (Too many lines in module) any more.
Extract miscellaneous logical units from cmdlib
All remaining classes in init.py are extracted to misc.py.
Extract os related logical units from cmdlib
All LUOs* classes are extracted to operating_system.py.
Extract query related logical units from cmdlib
All LUQuery* classes are extracted to query.py.
Extract backup related logical units from cmdlib
All LUBackup* classes are extracted to backup.py.
Extract instance related logical units from cmdlib
All LUInstance* classes are extracted to instance.py. Common functionsare moved to common.py if used by non-instance logical units as well.Additionally, helper functions which are only used by LUBackup* and...
Extract node related logical units from cmdlib
All LUNode* classes are extracted to node.py. Common functions are movedto common.py if used by non-node logical units as well.
Extract group related logial units from cmdlib
All LUGroup* classes are moved to group.py. Common functions areextracted to common.py.