Add a help function for generating test nodes
This abstracts the concept of "give an sane, online node" in a singleplace, rather than spread around the tests. We also change thesignature for isNodeBig for easier usage.
Signed-off-by: Iustin Pop <iustin@google.com>...
Generate clusters with unique node names in tests
Nothing breaks otherwise in the cluster routines, but serialisationto/from text gets confused otherwise.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Add a complex allocation/serialisation/load test
This is unfortunately a complex test that checks if:
- starting from a empty cluster- allocating a few instances on it- serialising it via the Text backend- loading it back into internal data structures...
Fix a couple of bugs in handling offline instances
The addition of the offline state for instances has introduced a fewbugs related to their handling. The current code adds a helperfunction for making the check whether an instances uses reservedmemory on the secondary (and uses it instead of just autoBalance),...
Add a helper function for 'not offline' instances
Unittests for the instance policy checks on allocation
Show explicit message when hspace fails to alloc
Currently, in the human-readable mode, hspace doesn't show anymessages when no instances can be allocated in tiered-alloc mode. Thisis confusing, so let's add an explicit message.
Obey instance policy in allocation routines
This currently checks the minimum/maximum sizes and the disktemplate. Unittests will come later.
Set group policy on nodes
In order to be able to use the group policy during allocation, we havea few choices:
- pass the group policy as a separate argument to all involved functions, or- add the policy as an attribute to nodes, or- add the group (not only the group index) as an attribute to nodes...
A trivial optimisation in Loader.mergeData
During loading, we can eliminate some intermediate datastructures. This only matters for big clusters, and even there it'sless than 1% less memory allocated, but still, it's a no-brainer.
Trivial performance optimisation
This is not in the hot-path, but still, no sense in doing O(n) insteadof O(1).
Add ipolicy attribute to group objects
This is straightforward; as before, so far the text and simu backendshave no way to use a non-default ipolicy.
Furthermore, we do some unit-test preparation for future changes(these seem a no-op for now, but they are needed for later patches)....
Enable Eq instances for TH-built objects
This is needed for the next patch.
Switch hspace to read the disk template from the cluster
The default disk template is the first enabled template.
Read the disk templates part of the ipolicy
The default value is badly defined (hardcoded defaults)…
Move DiskTemplate definition around
This is needed since we'll need the DiskTemplate definition in theIPolicy one.
Add new disk_templates parameter to instance policy
This is a bit more complex patch, as it requires changing theassumption that all keys in the policy dict points to values that arethemselves dicts. Right now we introduce an assumption that anynon-dicts are lists, we'll see in the future if this holds or whether...
Remove extraneous check in policy creation
The values are already checked in CreateIPolicy, no need to manuallycheck them again.
Move the instance specs options to cli.py
Currently these are defined twice, instead of a single time incli.py. Also adds the new disk_templates option to the common block,even though it's not yet used.
Add a new disk-template ipolicy option
Add a new CLI option type 'list'
This simply splits the value in the option parser, instead of needingto do it in the client code.
Fix handling of errors from InstancePolicy.Check...
This code raises a configuration error, but we need to transform itinto a prereq error (or possibly exec error), depending on when wecall this function.
Switch hspace defaults to the cluster policy
This changes from the current hardcoded defaults to the clusterpolicy. The command line options now override the defaults from thecluster, and the tiered spec mode is always enabled.
Also fixes a tiny typo in the man page (together with the man page...
Add a helper function converting ispecs to rspecs
The rspec type is simpler than the ispec one; most likely it should bedeprecated later.
Abstract creation of instance from a spec
Load cluster ipolicy via Rapi
This requires changing from querying the /tags resource to the /inforesource.
Update memory/maxmem reading in Rapi backend
Recent changes to the instance beparams have replaced memory withmaxmem in Rapi bulk queries. Until this is either reverted (forbackwards compat) or we decide to go ahead with only maxmem, we changethe backend to read both; this only affects the "instance down" code...
Load cluster ipolicy via Luxi
Also show it in hbal's verbose output (helpful for debugging).
Extend ClusterData with the cluster instance policy
This attribute is always initialised to the default, and is not (yet)read/saved in the various backends.
Add default ipolicy declarations
Add support for RE patterns to convert constants
This is a trivial conversion.
Add support for lists/frozensets in convert-constants
Unfortunately, we only support lists of simple types, and not evenlists of tuples. If we actually needed those, it would be possible toimplement them, with a bit more complexity in the converter.
Add support for tuples in convert-constants
More improvements to convert-constants
This prepares for tuple and other conversions.
Improve convert-constants to handle dictionaries
The two main drawbacks for convert-constants are the fact that itcan't handle sets/frozensets (mainly due to the fact that I don't knowhow useful this would be to the Haskell code) and that it cannotexport dictionaries....
Add object definitions for the ispec and ipolicy
Stop exporting JSON functionality from Utils.hs
This completes the Utils/JSON split started in commit f047f90f. Theimport graph should be cleaner now.
More reshuffling of code
Following the split Types/BasicTypes, we can remove the lastJSON-related stuff from Utils.hs, and do some more cleanup.
htools: split parts of HTools/Types.hs into BasicTypes.hs
The 'Result' type is common and:
- might be used outside of HTools-specific code too- is better split as we need these basic types for building the more complex ones in Types.hs
Fix acquisition of node lock in LUInstanceGrowDisk
Ensure node level locks are recalculated properlyin LUInstanceGrowDisk.
Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Fix cluster ispecs on upgrade
Cluster-level parameters must not be empty. Plus two more minorchanges.
Better max ispec defaults
Instead of being equal to the minimum ones, these are reusing currentconstants.
Rename optISpec to optStdSpec
More consistency with the tiered allocation mode.
Change how hspace reads cmdline args for std specs
For the standard specs, hspace currently takes separate options forthe memory, disk and VCPUs. For the tiered specs, which wereintroduced later, it takes a single option with all three valuescombined....
Abstract tiered specs command line parsing
This will be used in the future for the standard specs too, so let'sabstract it away.
cmdlib._ComputeMinMaxSpec: Add unittest for this function
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
cmdlib: _VerifyInstancePolicy has been replaced and is dead code
This removes this method
cmdlib: Rename _CheckMinMaxSpecs to _ComputeMinMaxSpec
This is to reflect the fact that _Check functions do raise Op*Errorwhich is not the case for _CheckMinMaxSpec.
utils.ResetTempfileModule: Improve performance
This function is called for after every fork (e.g. for handling an RPCrequest). With the changes in this patch generating the next randomfilename is about 30% faster.
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
LUInstanceCreate: Release node resource lock
… after running iallocator. Otherwise an assertion later in the codewould fail (for a good reason).
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
_CheckTargetNodeIPolicy: instance.primary_node is not objects.Node
cmdlib: node.group is not a nodegroup object
LUInstanceCreate: self.disks is a dict not a disk object
qa-sample.json: Fix a typo
Fix wrong variable name
Commit bc5d0215 added support for disk params, but due to copy-pasteit tries to enforce the hvparams into disk params values, leading to:
$ gnt-cluster modify -H kvm:initrd_path=/boot/initrd-2.6-kvmUParameter Error: Unknown parameter 'initrd_path'...
InstanceStartup: use overridden beparams in prereq
Without doing this memory checks happen on instance default parametersrather than any overridden ones.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
gen-coverage: Change filter
It is not necessary to run a Python script to generate the filter, whichdidn't work very well anyway on systems with modules in/usr/share/pyshared.
rpc: More unittests
- Fix: Don't check st_atime for temporary file for upload tests- Add tests for five encoders
Document "gnt-node add" behavior in case of failure
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
gnt-node: Add instance policy to migrate
cmdlib: Adapt LUClusterVerifyGroup for instance policy
gnt-backup: Adding force option to import
This makes it possible to bypass the instance policy
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
gnt-instance: Adding instance policy to add
gnt-instance: Adding verification of instance policy to modify
gnt-instance: Adding instance policy to move
gnt-instance: Adding instance policy to migrate
gnt-instance: Adding instance policy to failover
cmdlib: Adding helper for instance policy
rpc.ConfigRunner: Fix uploading file
Commit 601dfcb made some changes to the default encoders. This made“ConfigRunner.call_upload_file” fail due to a missing encoder. Thispatch applies the necessary changes to ConfigRunner and updates theunittest for “call_upload_file”....
Update GrowDisk docstring
It forgot about one argument.
Merge branch 'devel-2.5'
Conflicts: NEWS: trivial
Move helper class from watcher to utils.io
“FileStatHelper” can be used together with “ReadFile” to a file's statuswhile it's opened. This avoids certain race conditions.
rpc._PrepareFileUpload: Use FileStatHelper
Use fstat(2) on the file handle instead of doing a separate stat(2).
rpc: Change signature of RpcRunner
Instead of receiving the whole masterd context, it only gets theconfiguration and the lock monitor callback. This simplifiesunittesting.
Additionaly, two new arguments for testing are added.
Add unittest for uploading file via RPC
After some preparing patches this unittest can finally be added.
rpc._RpcClientBase: Add check for number of arguments
Just an additional check, useful in unittests.
Add unittests for RPC client
This patch adds a number of unittests for the RPC client base class.Some small changes were necessary in “rpc.py” to allow for bettertesting.
runtime: Style fix in docstring
rpc: Style fix
“assert” is a statement, not a function.
Add unittest for RPC compression
Merge branch 'stable-2.5' into devel-2.5
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Andrea Spadaccini <spadaccio@google.com>
check-news/NEWS: impose more standards
- Always two empty lines before a version.- Standardize how version names should be called.
KVM: support version reported by 1.0
This of course was working for all the rcs, but broke with 1.0 itself.
In addition: - split between running kvm --version and parsing its output - unittest parsing for various known --help outputs - updated NEWS file...
qa: fix disk parameters tests
Fix an error in the disk parameters tests (the arguments to gnt-clusterwere not passed correctly) and move them to separate functions.
Signed-off-by: Andrea Spadaccini <spadaccio@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
rpc._NodeConfigResolver: Support resolving offline nodes
This is needed to powercycle a node marked offline.
rpc_defs: Enable “node_powercycle” for offline nodes
With this change it's possible to powercycle nodes marked as offline. Ofcourse the node might not be running the node daemon anymore.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
man/gnt-node: Improve powercycle documentation
Reword a bit, capitalize “SSH”.
rpc._RpcProcessor: Make read timeout a non-keyword argument
It is always used and can be set to “None” if intended.
rpc: Add support for resolver options callback
This adds support for a callback returning name resolver options. Thisis required for powercycling offline nodes. While it would be possibleto implement this using a separate RPC client class, doing so would...
rpc: Pass resolver options to actual resolver
QA: Make the ispec values configurable
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Update the resource model DD with current state
On my work for instance policies we decided not to overload --forcemore, instead we go with a new flag --ignore-ipolicy. This is just asmall patch to reflect this decision.
Signed-off-by: René Nussbaumer <rn@google.com>...
Fix failure in distcheck and coverage tests
autotools/testrunner was missing in distcheck. Now some changes from7004106119c5c56c93085ccfb1250b3561b5b6ad have been reverted, and testrunneris not built anymore. coverage now uses fakeroot when needed.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>...
Makefile: Fix parallel distcheck
Running “make -jN distcheck” for any N > 1 would reliably break whenbuilding the RPC client wrappers. This was due to wrong dependencies inMakefile.am, fixed by this patch.
doc/admin: Clarify archived jobs
Also mention that archived jobs can be viewed using “gnt-job info”.
test/lockperf: Use a list of integers, not booleans
Add lock performance utility
I had an idea for improving locking performance. To see if it worked Iwrote this tool. Unfortunately the idea didn't quite work (brokeunittests left and right), but the tool is still handy for evaluatingfuture changes to the “SharedLock” class....
jqueue: Factorize checking job processor's result
This allows for more unittesting.