Factor out check for storage params
This patch factors out the sanity checks for storageparameters in its own function. Unit tests included.
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Tighten sanity checks for '_GetLvmVgSpaceInfo'
This patch tightens the sanity checks for the input of'_GetLvmVgSpaceInfo' and provides unit tests for it.
More sanity checks for spindle space reporting
This patch adds a wrapper around the space reportingfor spindles which includes more sanity checks forthe input. Unit tests provided.
Unit tests for _GetVgSpindlesInfo
This patch provides unit tests for _GetVgSpindlesInfo.In order to mock the used function for the 'bdev' moduleI made the low-level storage function an optional parameterof _GetVgSpindlesInfo. I also found out that mocked...
Fix module functions overridden by mocks
This patch fixes remaining occurrences of module functionsthat were overridden by mock functions. They need to beset back to their original functions after the test caseends.
Signed-off-by: Helga Velroyen <helgav@google.com>...
Fix single-execution of Haskell tests and shelltests
Add Makefile.am target hs-test-% for single-execution of Haskelltests.
Add Makefile.am target hs-shell-% for single-execution of Haskellshelltests.
Update documentation on running individual tests....
Merge branch 'stable-2.8' into master
Include "instance" information in LV data collector
This commit enables the logical volume data collector to get information aboutthe instances and to link it to the information about logical volumes.
The list of parameters accepted by the collector is expanded to allow proper...
Add "includeLogicalId" function for Disks
This function checks whether a disk contains a given Lvm logical ID,directly or through its children.
Unit tests are added as well.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Add "instance" field to LVInfo
Extend the LVInfo data structure with the field for storing the name ofthe instance it is paired with.
Update the tests accordingly.
Also remove prop_IterateAlloc_sane from test list
In f4d1bb7 that test was removed, but forgotten to remove itfrom the list of tests to be executed. Fix that.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Fix documentation for prop_Alloc_sane
As discussed in the last commit, placing a new instance on the clustercan lead to a cluster that can be improved by moving previously addedinstances. For an empty cluster, however, there are no previousinstances. So add this to the test description to make obvious why...
Remove IterateAllocSane test
The test is testing for a property that just isn't true. Iteratedallocation greedily place one instance at a time taking the locallymost balanced solution. Then it is tested whether the resulting globalallocation can be improved....
Improve handling of storage info
This patch improves the processing of storage informationas result of the 'node_info' call in the following way:- It removes the override for LVM storage. This was needed in MakeLegacyNode info to be compatible to the...
Turn 'exclusive_storage' into storage parameter (hs)
This is the haskell implementation of my patch "Extend RPC call'node_info' by storage parameters". It turns the 'exclusivestorage' flag into a storage parameter of the LVM storage types.Besides that, this patch moves some types into the Types.hs....
Add general storage parameters to node info call
As described in the design doc about storage types,we plan to generalize the RPC call "node info" wrt tostorage types. This patch extends the call by acceptinga list of storage units, that is not only identified...
backend unit tests: remove unused imports and variables
This patch removes some unused imports and an unusedvariable of the backend unit test. No functional changesotherwise.
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Thomas Thrainer <thomasth@google.com>
Export CPUs used by the node OS
They are exported through the LUXI, RAPI, and IAllocator interfaces.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Load CPUs used by the node OS in htools
A new field is added to the Node type, and it's used to initialize the usedCPUs field.
The signature of Node.create has been split among lines to match theparameter list.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>...
Add htools shell test for VCPU handling
This tests that VCPUs used by both instances and node OS are correctlyaccounted in allocation.
Hypervisors report CPUs used by the node OS
The XenHypervisor's "dom0_cpus" parameter (introduced in 1d60fec6 but neverused) is renamed to "cpu_dom0" for consistency with other parameternames. The analogous information is returned for the other hypervisors....
Add tests for the LV data collector
This commit adds a few shelltests for the logical volume data collector.
Also, it performs some cleanup of the comments already present in the file:given that the number of the tests should be changed manually, and this is...
Add tests for LVs parser
Add unit tests and Quickcheck tests for the LV parser.
Also, fix the alphabetic order of some imports in htest.hs.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Add function testing parsers with QuickCheck
Parsing a string automatically generated by QuickCheck can be useful in manycases. A function for doing this is added to the common test functions.
Add function for generating arbitrary UUID
Add a function for generating arbitrary UUIDs to be used for QuickCheck tests.
Add test for the --full-evacuation option
In this example cluster, there are four pairs of nodes, each hostingtwo instances. Each node has disk capacity for an additional instance.Therefore, for normal online rolling reboots, there are two node groups,...
iallocator: storage processing accepts space_info structure
This is a small refactoring to make the processing ofthe storage space information not accept a node inforesult, but the more specific space info structure instead.
iallocator: extract storage info directly from node info
So far, the iallocator made a node info RPC call, convertedthe result into a legacy node info format and then extractedthe storage information from that. This patch removes themiddle step of this procedure, skipping the legacy node info...
iallocator: computation of storage data
This patch extracts the processing of the storage-relateddata from the RPC call 'node_info' in a separate functionto increase testability. In addition, this patch containsunit tests for this function.
rpc.py: override default storage type in node info
This patch changes the semantics of dealing with the resultof the RPC call "node_info" when converting them into aLegacyNodeInfo. A legacy node info, can only hold storageinformation about one storage type, which so far was always...
Index instances by their UUID
No longer index instances by their name but by their UUID in the clusterconfig. This change changes large parts of the code, as the followingadjustments were necessary: * Change the index key to UUID in the configuration and the...
Fix testGetHvDefaults testcase
The introduction of the migration_mode parameter to the fake hypervisorbroke the testGetHvDefaults. This patch fixes this by taking the defaultparameters of the fake hypervisor into account.
Signed-off-by: Thomas Thrainer <thomasth@google.com>...
Version bump to 2.9.0~alpha1
Bump the version number to 2.9.0~alpha1.
This also requires updating version numbers in cfgupgrade tool.
Furthermore, a data file for test representing a 2.8 configuration had theversion number set to 2.7 in order to prevent tests failing while the current...
Merge branch 'stable-2.8' into 'master'
Rename directory 'Block' to 'Storage'
This patch renames the 'Block' directory to 'Storage' inthe Haskell code base. The same rename was done in thepython code base earlier this quarter. We generalize thename, because we needed a place for general storage...
Add test for shrink heuristics over different resources
In this example, memory prevents adding more than one instance onthe first node. Yet, on the other nodes, 2 instances can be placedeach, if disks are shrunk appropriately. This, however, requiresto courageous decision to shrink disks next, even though more nodes...
Add tests for check-news
Unit tests verifying the behaviour of the check-news script are added.
This required adding a new make target, "autotools-check" directly called from"commit-check", because the usual tests are executed in the build directory,whereas these, being relative to the autotools, cannot be executed there because...
Refactor NodeInfo RPC regarding storage reporting
The NodeInfo RPC call is refactored to handle now more thanjust storage reporting for volume groups.
Since NodeInfo now returns storage space information notnecessarily for volume groups, but also for other storage...
Utility functions for storage types
Handling various storage types for the free space reportingrequires some utility functions. They will be invoked inlater patches. Unit tests are included.
Unit tests for ApplyStorageSpaceFunction
So far, the function 'ApplyStorageSpaceFunction' was notcovered by unit tests. Before the integration of spacereporting for file storage, we add those missing tests.
Renamining space reporting for file storage
Before being integrated, the function for space reportingfor file storage is renamed to a more expressive name.
Merge branch 'stable-2.7' into stable-2.8
Conflicts: (trival, take union of added files/tests) Makefile.am test/hs/shelltests/htools-hspace.test...
Add a test for hspace to respect instance policy
In this example, the instance policy for disks prevents allocationof more than one instance per node. The test verifies that tieredallocation still respects this and doesn't attempt to place policyviolating instances on the cluster....
Make shrinkByType aware of individual disks
When shrinking an instances, you can't just get smaller disk footprintwhile leaving the individual disks as they are. Make the shrinkheuristic aware of that fact, and decrease all individual disks aswell. Fixes issue 484....
Convert node indices to UUIDs in configuration
cfgupgrade replaces the occurences of node names (where they are used asindex) with the node UUID.It also supports the inverse replacement, so downgrades are supportedtoo.
Tests for cfgupgrade are adapted as well....
Index nodes by their UUID
No longer index nodes by their name but by their UUID in the clusterconfig. This change changes large parts of the code, as the followingadjustments were necessary: * Change the index key to UUID in the configuration and the ConfigWriter, including all methods....
Remove XEN_CMD from constants, adjust to PowercycleNode
This patch extend the RPC to powercycle a node by ahvparams parameter. This parameter is used by thexen hypervisor to determine which xen command to use.The respective LU is adjusted as well.
Since this was the last operation depending on the...
hv_xen.py: make hvparams mandatory, remove fallbacks
This patch removes the fallback to the xen command from theauto config and adds an exception instead. Also, thehvparams parameter of the functions GetCommand, _RunXen and_GetInstanceList are made mandatory....
Add hvparams to RPC call 'node_info'
This patch adds the hvparams parameter to the RPC call'node_info'. It also adjusts the related code in noded.pyand Query/Node.hs
backend.py: use hvparams in GetNodeInfo
This patch extends the GetNodeInfo function of the backendmodule by a hvparams parameter. For now, it contains afallback solution for when the parameter is None. Thisfallback will be removed in a later patch of this series....
hypervisors: add hvparams to GetNodeInfo
This patch extends the GetNodeInfo function of thehypervisors by a hvparams parameter. The parameteris currently only used by the xen hypervisor to determinewhich xen toolstack ('xm' or 'xl') to use.
hv_xen.py: renaming residual 'xm' occurrences
Some methods still refered to 'xm', which is inaccurate nowthat xen can also be run with 'xl'. No functional changesotherwise.
Include hvparams in ssconf files
The watcher needs access to the cluster's hypervisorparameters. This patch extends the ssconf infrastructureby adding a hvparams file for each hypervisor. The formatof the hvparams file is a simple key=value format.
Use hvparams in instance migration
This patch makes use of hvparams on instance migration todetermine the list of instance before the migration. This way, inxen, the xl/xm choice is respected.
Verify: xen toolstack, hypervisor and hvparams
This patch extends the node verification by:- Adding a check for the xen toolstack when the hypervisor is verified.- Factoring out the hypervisor verification in a subfunction to increase testability.- Factoring out the hvparams verification in a...
backend.py: hvparams in 'GetInstanceList'
Propagating the use of hvparams further up the callinghierarchy, this patch makes the backend module use thehvparams in 'GetInstanceList'. Unit tests provided.
hv_xen.py: _RunXen and GetInstanceList use hvparams
Propagating the use of hvparams further up in the callinghierarchy, the functions '_RunXen' and 'GetInstanceList'use the given hvparams. Unit tests provided. Calling function'StopInstance' adjusted in this patch as well....
hv_xen.py: rename GetXmList to GetInstanceList
Since one can retrieve xen's instance list via both 'xm'or 'xl', it is no longer appropriate to call the function'GetXmList'. This patch renames it to 'GetInstanceList'.Other than that, there are no functional changes....
hv_xen._GetCommand: retrieve xen command from hvparams
This patch adds a (so far optional) hvparams parameterto the '_GetCommand' function. This dictionary is usedto retrieve the xen command (xm or xl). It is optionalfor now to make the refactoring possible without breaking...
Unit tests for hv_xen._GetCommand
This patch adds unit tests for the current state of the'GetCommand' method. The tests are rather trivial, buthaving tests before starting to refactor seems to bea good idea.
Fixing docstring of hv_xen_unittest.py
The introductory comment of hv_xen_unittest.py referred tothe wrong hypervisor.
New tests for hspace and exclusive storage
hspace is run on clusters with exclusive storage enabled.
Spindles become part of htools resource spec
Spindles are now part of resource spec. Instances get created with spindlesspecified (which are just ignored when exclusive storage is disabled).
htools cluster score takes spindles into account
When exclusive storage is enabled, spindles are used instead of disk spaceto compute the cluster score.
Comments and variable names in computePDsk has been changed to match theactual code.
Update spindles when moving instances in htools
Spindles get updated, and errors raised when not enough free spindlesexist. No new error is raised when exclusive storage is disabled.
Unit tests included.
Unit tests for htools and exclusive storage
The existing tests are run also on nodes with exclusive storage enabled. Thevalues for spindles and exclusive storage are set in a consistent way, forboth nodes and instances.
Load node spindles data in htools
The data structure for nodes gets a new field for free spindles, and theexisting field for total spindles gets renamed to avoid identifying it withthe node parameter that had the same name. These fields get filled with...
Export node spindles
Node spindles (queried live) are exported through the LUXI, RAPI, andiallocator interfaces.
Add a force option to the ClusterSetParams Opcode
If set, the op code will, in particular, try to set the master IPon the new netdev, even if shutting down the master IP on the oldnetdev failed.
Fix shadowing of library function
The "reads" field shadows a library function from Prelude. This commitfixes the problem.
Add shelltests for diskstats
Check that the behavior of running the diskstats data collector is asexpected.
Add unit tests for the Diskstats parser
Add one test parsing an actual /proc/diskstats file and one QuickChecktest.
Add function for generating arbitrary non-negative numbers
In many tests it is useful to have the possibility to easily generatenon-negative integer numbers.
Factor out and rename testFile function
The testFile function used in the DRBD parser test file can actually beuseful in any case where a parser has to be tested.
This patch moves it to the TestCommon file, and renames it to a more meaningfultestParser....
Fix erroneous docstring
Unit tests for spindles and exclusive storage in htools
Everything should work the same when exclusive storage is active, asspindles are not yet handled as a resource. A new test has been added forthe interactions between spindles and policies.
Check real spindles in ipolicies
When exclusive storage is enabled, the spindles in instance disks are usedto check the instance policies (as outlined in design-partitioned.rst).
Add spindles to instance disks in htools
A new data type is introduced for disks to store both size and spindles.When available, spindles are filled with input data. Except for loading andstoring, spindles are ignored.
Load exclusive_storage in htools
The node parameter is loaded into the data structures. No behavior is yetmodified.
New function to load JSON arrays of optional values
This will be needed to load spindles in some htools backends. Unit testsprovided.
Add unit test for text backend + fix bug
Test serialization and deserialization of instances. Fix check of secondarynode.
Test with non-redundant instances and several node-groups
The example cluster contains two node groups. The first contains three nodes,with two non-redundant instances each. As each node has only capacity for threeinstances, the nodes have to be rebooted individually. Additionally, there is...
Add tests for network-aware allocation
hail-alloc-invalid-network defines a cluster with two nodegroups and anallocation request which does not fit on any of the groups. Group 1 hasinvalid disk-templates while Group 2 is not connected to the rightnetworks....
Parse NIC data from allocation request in hail
Add a NIC type and extend the Instance type by a list of NIC's. Parsethe NIC's in allocation requests and store them for now. Later patcheswill make use of this field in order to ensure that the requestedinstance is only placed in node groups wich are connected to those...
Support group networks in Text backend
The Text backend now parses network UUID (comma separated) andserializes them in the same form.The test data is adapted to the new format.
Signed-off-by: Thomas Thrainer <thomasth@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Parse node group networks
Extend the Group by the network ids it is connected to. Adaptthe IAlloc backend such that the networks are parsed correctly.This also required the adaption of test data.
Make the disks_active flag queryable
gnt-instance list now also supports the disks_active field.
Signed-off-by: Thomas Thrainer <thomasth@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Add disks_active to configuration
This flag tracks if the disks of an instace are supposed to be active.That's the case when an instance is running or when its disks gotactivated explicitly (and in a couple of other cases).It will be used by watcher to re-activate disks after a node reboot....
Attaching a logical volume builds a list of PVs
When an LV gets attached, the list of the PVs used by the LV is built. Thiswill be used to count spindles for exclusive_storage, but it could also beuseful to optimize disk growing and snapshotting.
Refactor code for attaching a logical volume
The parsing of "lvs" output is moved into private methods. The code isslightly more readable and testable. The split in two methods is usefulfor the following patches. Unit tests for the new functions areprovided....
Merge branch 'stable-2.8'
Add spindles field to disk object
The field is filled with the value provided on the command line.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Thomas Thrainer <thomasth@google.com>
cfgupgrade: Downgrade is a NO-OP
The configuration is still the same as in 2.8 (the reference stable versionfor this branch), so downgrade shouldn't do anything.
Unit tests are also updated, with a new 2.8 configuration file. Theconfiguration file used for the upgrade+downgrade test was tailored to the...
Extend hroller tests by options for non-redundant instances
The cluster now consists of 3 nodes, with drbd instances betweennodes 1 and 2, and 2 and 3. Additionally, nodes 1 and 3 each containa non-redundant instance, but node 2 cannot hold two additional...
Test for hroller taking non-redundant instances into account
The example cluster consists of 6 nodes, each hosting 2 instances andhaving capacity for 3. So, while the drbd-induced graph consists ofonly insulated nodes, no more than two nodes can be rebooted at the...
Stash Xen config file after a failed startup
This is a fix for a minor bug.Currently, a failed Xen VM start results in a stale config file leftbehind on the filesystem.This change introduces a new log directory, where the Xen VM configfile is moved after a failed startup....
cmdlib: Cleanup public/private functions
All functions/classes which are used outside of their defining module(with tests as an exception) no longer have a leading underscore.
Signed-off-by: Thomas Thrainer <thomasth@google.com>Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
cmdlib: Extract storage related functionality
Split instance.py further by extracting storage related logical unitsand functions to instance_storage.py.