Split the list of program personalities into a module
… a mini-module, actually. This is only done for unit-testing, as wedon't want to duplicate the personalities list in the tests.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Parse options in htools.hs, not in each program
This patch moves the call to getArgs/parseOpts into htools.hs, asopposed to the current situation where htools.hs only calls the mainfile of each program, which in turn calls these two functions.
By doing it only once we remove some duplicate code and we also export...
Add some very basic options test
Just tests that passing a string argument returns it correctly. We'renot interested in the flags themselves that much just that thefunction itself parses some options at least correctly.
Signed-off-by: Iustin Pop <iustin@google.com>...
Split CLI.parseOpts into IO/non-IO parts
This will allow unittesting the CLI options much more easily. Thepatch also fixes an inconsistency: usage help was displayed with anextra new line in the error case (but not in --help).
Add test for CLI.parseYesNo
Also fix a tiny typo in the docstring.
Fix bug in CLI.parseISpecString and add test
'zip' finishes on the shortest list, so parseISpecString accepted anystring of 3 values or more, which is not good. Fix this and add testfor both successful and failed parsing.
Add a unittest for the 'Simu' backend
This only test successful parsing, but for this case it testsreasonably well the resulting cluster properties, I think (coverageis >70% for expressions).
Further optimise instance test data generation
Following on the lessons from this week (commit 7806125e), this patchreplaces generation of instances `suchThat` they are smaller than onnode with direct generation of instances with upper bounds onmemory/disk/cpu....
Add a Makefile target for profiling targets
This simplifies the building of profiling-enabled htools binaries;there are two targets, one used for the initial two-stage build, andone that can be used for incremental builds.
The patch adds a new internal variable for the htools/test binary,...
gnt-instance: Adapt replace-disks to the new opcode
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
TLReplaceDisk: Add ipolicy checks
Fix very slow unit-test data generation in some cases
Learning, learning. It turns out that for objects with even just fivefields, generating them and filtering can be costly, especially whenthe filtering is nested (as it was in this case, arbitrary1 →...
Fix one bad unit-test data generation case
It could happen that the node is generated with 0 disk available, inwhich case the operation will fail due to disk issues, not CPU. Fixthis by using the recently-introduced genOnlineNode helper.
Further JSON unit-tests for various Types.hs types
Add a unit-test helper function
… instead of always using "printTestCase … False".
Extend node evacuation unittests
Currently they only check for change secondary, via iallocator'srelocate method. We add two separate unittests that checks all thenode evacuate modes (ChangeAll, ChangePrimary) and also the groupchange request. All of these are done only for DRBD instances....
Fix a few issues in unittest data generation
This patch fixes two ugly but non-critical issues:
- the getFQDN wrongly generated only the first component as limited-length name, the rest could be any length, and that makes the generated name bad for display purposes...
Remove some dead code
While wondering why Cluster.tryReloc is not unit-tested, I realisedthat this is no longer uses. Apparently, last summer's new IAllocatormodes and related changes have silently deprecated this function.
Fix another data generation issue in tests
Another "good" seed value manages to generate nodes with identicalnames, which of course makes the sets be of smaller value than thelists. Avoid this issue by generating unique node names.
Further fixes to instance policy validation
As a followup from "Remove extraneous check in policy creation", thereare more places where we build an ipolicy, and then manually check forits validity. This is very bad style, as it duplicates theverification code across many places....
Add a help function for generating test nodes
This abstracts the concept of "give an sane, online node" in a singleplace, rather than spread around the tests. We also change thesignature for isNodeBig for easier usage.
Generate clusters with unique node names in tests
Nothing breaks otherwise in the cluster routines, but serialisationto/from text gets confused otherwise.
Add a complex allocation/serialisation/load test
This is unfortunately a complex test that checks if:
- starting from a empty cluster- allocating a few instances on it- serialising it via the Text backend- loading it back into internal data structures...
Implement saving/restoring of policies in Text backend
This adds a new optional (so we're backwards-compat) section to thecluster data so that we can reliably save/restore the policy data.
Note that different from the other sections, we save all policies...
Further unittests for ipolicy/ispec text serialisation
Fix a couple of bugs in handling offline instances
The addition of the offline state for instances has introduced a fewbugs related to their handling. The current code adds a helperfunction for making the check whether an instances uses reservedmemory on the secondary (and uses it instead of just autoBalance),...
Add a helper function for 'not offline' instances
Unittests for the instance policy checks on allocation
Show explicit message when hspace fails to alloc
Currently, in the human-readable mode, hspace doesn't show anymessages when no instances can be allocated in tiered-alloc mode. Thisis confusing, so let's add an explicit message.
Obey instance policy in allocation routines
This currently checks the minimum/maximum sizes and the disktemplate. Unittests will come later.
Set group policy on nodes
In order to be able to use the group policy during allocation, we havea few choices:
- pass the group policy as a separate argument to all involved functions, or- add the policy as an attribute to nodes, or- add the group (not only the group index) as an attribute to nodes...
A trivial optimisation in Loader.mergeData
During loading, we can eliminate some intermediate datastructures. This only matters for big clusters, and even there it'sless than 1% less memory allocated, but still, it's a no-brainer.
Trivial performance optimisation
This is not in the hot-path, but still, no sense in doing O(n) insteadof O(1).
Add ipolicy attribute to group objects
This is straightforward; as before, so far the text and simu backendshave no way to use a non-default ipolicy.
Furthermore, we do some unit-test preparation for future changes(these seem a no-op for now, but they are needed for later patches)....
Enable Eq instances for TH-built objects
This is needed for the next patch.
Switch hspace to read the disk template from the cluster
The default disk template is the first enabled template.
Read the disk templates part of the ipolicy
The default value is badly defined (hardcoded defaults)…
Move DiskTemplate definition around
This is needed since we'll need the DiskTemplate definition in theIPolicy one.
Add new disk_templates parameter to instance policy
This is a bit more complex patch, as it requires changing theassumption that all keys in the policy dict points to values that arethemselves dicts. Right now we introduce an assumption that anynon-dicts are lists, we'll see in the future if this holds or whether...
Remove extraneous check in policy creation
The values are already checked in CreateIPolicy, no need to manuallycheck them again.
Move the instance specs options to cli.py
Currently these are defined twice, instead of a single time incli.py. Also adds the new disk_templates option to the common block,even though it's not yet used.
Add a new disk-template ipolicy option
Add a new CLI option type 'list'
This simply splits the value in the option parser, instead of needingto do it in the client code.
Fix handling of errors from InstancePolicy.Check...
This code raises a configuration error, but we need to transform itinto a prereq error (or possibly exec error), depending on when wecall this function.
Switch hspace defaults to the cluster policy
This changes from the current hardcoded defaults to the clusterpolicy. The command line options now override the defaults from thecluster, and the tiered spec mode is always enabled.
Also fixes a tiny typo in the man page (together with the man page...
Add a helper function converting ispecs to rspecs
The rspec type is simpler than the ispec one; most likely it should bedeprecated later.
Abstract creation of instance from a spec
Load cluster ipolicy via Rapi
This requires changing from querying the /tags resource to the /inforesource.
Update memory/maxmem reading in Rapi backend
Recent changes to the instance beparams have replaced memory withmaxmem in Rapi bulk queries. Until this is either reverted (forbackwards compat) or we decide to go ahead with only maxmem, we changethe backend to read both; this only affects the "instance down" code...
Load cluster ipolicy via Luxi
Also show it in hbal's verbose output (helpful for debugging).
Extend ClusterData with the cluster instance policy
This attribute is always initialised to the default, and is not (yet)read/saved in the various backends.
Add default ipolicy declarations
Add support for RE patterns to convert constants
This is a trivial conversion.
Add support for lists/frozensets in convert-constants
Unfortunately, we only support lists of simple types, and not evenlists of tuples. If we actually needed those, it would be possible toimplement them, with a bit more complexity in the converter.
Add support for tuples in convert-constants
More improvements to convert-constants
This prepares for tuple and other conversions.
Improve convert-constants to handle dictionaries
The two main drawbacks for convert-constants are the fact that itcan't handle sets/frozensets (mainly due to the fact that I don't knowhow useful this would be to the Haskell code) and that it cannotexport dictionaries....
Add object definitions for the ispec and ipolicy
Stop exporting JSON functionality from Utils.hs
This completes the Utils/JSON split started in commit f047f90f. Theimport graph should be cleaner now.
More reshuffling of code
Following the split Types/BasicTypes, we can remove the lastJSON-related stuff from Utils.hs, and do some more cleanup.
htools: split parts of HTools/Types.hs into BasicTypes.hs
The 'Result' type is common and:
- might be used outside of HTools-specific code too- is better split as we need these basic types for building the more complex ones in Types.hs
Fix acquisition of node lock in LUInstanceGrowDisk
Ensure node level locks are recalculated properlyin LUInstanceGrowDisk.
Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Fix cluster ispecs on upgrade
Cluster-level parameters must not be empty. Plus two more minorchanges.
Better max ispec defaults
Instead of being equal to the minimum ones, these are reusing currentconstants.
Rename optISpec to optStdSpec
More consistency with the tiered allocation mode.
Change how hspace reads cmdline args for std specs
For the standard specs, hspace currently takes separate options forthe memory, disk and VCPUs. For the tiered specs, which wereintroduced later, it takes a single option with all three valuescombined....
Abstract tiered specs command line parsing
This will be used in the future for the standard specs too, so let'sabstract it away.
cmdlib._ComputeMinMaxSpec: Add unittest for this function
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
cmdlib: _VerifyInstancePolicy has been replaced and is dead code
This removes this method
cmdlib: Rename _CheckMinMaxSpecs to _ComputeMinMaxSpec
This is to reflect the fact that _Check functions do raise Op*Errorwhich is not the case for _CheckMinMaxSpec.
utils.ResetTempfileModule: Improve performance
This function is called for after every fork (e.g. for handling an RPCrequest). With the changes in this patch generating the next randomfilename is about 30% faster.
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
LUInstanceCreate: Release node resource lock
… after running iallocator. Otherwise an assertion later in the codewould fail (for a good reason).
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
_CheckTargetNodeIPolicy: instance.primary_node is not objects.Node
cmdlib: node.group is not a nodegroup object
LUInstanceCreate: self.disks is a dict not a disk object
qa-sample.json: Fix a typo
Fix wrong variable name
Commit bc5d0215 added support for disk params, but due to copy-pasteit tries to enforce the hvparams into disk params values, leading to:
$ gnt-cluster modify -H kvm:initrd_path=/boot/initrd-2.6-kvmUParameter Error: Unknown parameter 'initrd_path'...
InstanceStartup: use overridden beparams in prereq
Without doing this memory checks happen on instance default parametersrather than any overridden ones.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
gen-coverage: Change filter
It is not necessary to run a Python script to generate the filter, whichdidn't work very well anyway on systems with modules in/usr/share/pyshared.
rpc: More unittests
- Fix: Don't check st_atime for temporary file for upload tests- Add tests for five encoders
Document "gnt-node add" behavior in case of failure
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
gnt-node: Add instance policy to migrate
cmdlib: Adapt LUClusterVerifyGroup for instance policy
gnt-backup: Adding force option to import
This makes it possible to bypass the instance policy
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
gnt-instance: Adding instance policy to add
gnt-instance: Adding verification of instance policy to modify
gnt-instance: Adding instance policy to move
gnt-instance: Adding instance policy to migrate
gnt-instance: Adding instance policy to failover
cmdlib: Adding helper for instance policy
rpc.ConfigRunner: Fix uploading file
Commit 601dfcb made some changes to the default encoders. This made“ConfigRunner.call_upload_file” fail due to a missing encoder. Thispatch applies the necessary changes to ConfigRunner and updates theunittest for “call_upload_file”....
Update GrowDisk docstring
It forgot about one argument.
Merge branch 'devel-2.5'
Conflicts: NEWS: trivial
Move helper class from watcher to utils.io
“FileStatHelper” can be used together with “ReadFile” to a file's statuswhile it's opened. This avoids certain race conditions.
rpc._PrepareFileUpload: Use FileStatHelper
Use fstat(2) on the file handle instead of doing a separate stat(2).
rpc: Change signature of RpcRunner
Instead of receiving the whole masterd context, it only gets theconfiguration and the lock monitor callback. This simplifiesunittesting.
Additionaly, two new arguments for testing are added.
Add unittest for uploading file via RPC
After some preparing patches this unittest can finally be added.
rpc._RpcClientBase: Add check for number of arguments
Just an additional check, useful in unittests.
Add unittests for RPC client
This patch adds a number of unittests for the RPC client base class.Some small changes were necessary in “rpc.py” to allow for bettertesting.
runtime: Style fix in docstring