Refactor IAllocator code
The IAllocator class was handling all the requests on its own, passingin parameters on top level which works, but is hard to maintain and notflexible.
With the upcoming change to the IAllocator for MultiAllocate we can'tuse the toplevel parameters anymore. Therefore, we refactor the code...
Make the slots functionality more modular
As we will introduce another set of containers using the slots trickwe abstract away as much as possible to separate bases classes. Thechild classes then adapt them for their needs. This leads to less code...
Move _CalculateGroupIPolicy and _ComputeDiskSize
This makes it possible to better untangle the IAllocator code, whichwould otherwise lead to a recursive import structure as we need thosehelpers there as well.
Signed-off-by: René Nussbaumer <rn@google.com>...
Move shelltests from test/ to htest/shelltests/
This is another rather trivial patch, moving all the (htools)shelltests to their own directory.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Move htools test files from test/data/htools to htest/data
Following up on the program moves, we now move the test data files.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Agata Murawska <agatamurawska@google.com>
Implement support for query only clients in Rapi
This implements the same query=True|False functionality as inGetClient for cli.py, however since the RAPI code is much moreunit-tested (and the unit-test clients are mocked, for the most part,without support for addresses) we have to do many adaptations in the...
Move necessary constants from rpc & rpc_defs to constants
As timeout constants will now be used also on Haskell side, they havebeen moved from rpc_defs and rpc to constants. The same is true forconnection timeout for connecting to nodes.
Signed-off-by: Agata Murawska <agatamurawska@google.com>...
Merge branch 'devel-2.6'
Add test for checking that all gnt-* subcommands run OK
This is a bit of a shell munging trickery, but works for now. Makingit more generic can be done later.
Switch job IDs to numeric
This has been a long-standing cleanup item, which we've alwaysrefrained from doing due to the high estimated effort needed.
In reality, it turned out that after some infrastructure improvements(the previous patches), the actual job queue-related changes are quite...
Make jqueue unittests more generic
This patch removes/abstracts some hardcoded values in the jqueueunittests. Currently we have a per-resource type name field, but thename field values are hardcoded at the call sites, instead of beingabstracted into separate variables....
Merge branch 'stable-2.6'
Add a shell test for hbal and split instances
This is not perfect, as we only test that hbal completes successfullyand that it show a score improvement, but it's better than nothing.
Add newline at the end of shelltest files
Signed-off-by: Agata Murawska <agatamurawska@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Add forgotten unittest changes for instance_os_add
The previous patch which fixed disk parameters didn't adapt theunittests so it lead to failing QA.
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Agata Murawska <agatamurawska@google.com>
Style fixes in shelltests
Shelltestrunner tests for hcheck
Simple tests for hcheck using shelltestrunner. Among other, we checkthat we can run hcheck on multi-group cluster.
Update the hooks documentation
Also provide some extended unittests to catch those cases.
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>Reviewed-by: Agata Murawska <agatamurawska@google.com>
Verify user supplied dicts against defaults
This verifies the user (especially in nested dicts) does notprovide a key which is not seen in the defaults dict for that dict.
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
jstore: Add unittests
Add unittests for FormatJobID, GetArchiveDirectory, ParseJobId.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
query: Comparison of timestamps, job IDs
This patch adds the necessary conversions for comparing split timestamps(such as used by jobs) and job IDs (by defining custom converters).Unittests are included.
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
Fix cfgupgrade unittests
Sorry, I broke the cfgupgrade unittests via 904910c4, since thatcommit added the requirement for the "instances" dict in theconfiguration.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
query2: Add <, >, <=, >= comparison operators
These can be used, for example, to get jobs submitted after a certaintimestamp.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Adjust cfgupgrade for new minor version
Also does some abstracting of the versions.
Beautify disk ipolicy violations in cluster-verify
Currently, we only get:
instance3: ['disk-size value 512 is not in range [1024, 1048576]'
which doesn't explain which disk we are talking about. This patchextends the verification functions to take an additional parameter...
ganeti.query_unittest: Adding testcase for diskparams
Make x509 unittest testClockSkew a bit less flaky
Since the tested function actually uses time.time(), it cannot be makefully stable, but 1 second is very dangerous; let's just test SKEW * 2and higher since that should be good (if the delta between _GenCert...
query: Expose diskparamters through query
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Merge branch 'devel-2.5'
Merge commit 'v2.5.1' into devel-2.5
Workaround changed LVM behaviour
The vgreduce command has changed behaviour from when we initiallywrote the code (2.02.02 versus 2.02.66, 4 years delta):
- if there are LVs which will be impacted, it requires --force- otherwise refuses to proceed, but it still returns exit code 0...
cmdlib: Remove all diskparams calculations not required anymore
Fix exception re-raising in Python Luxi clients
Commit e687ec01 (present in 2.5 since the 2.5 beta 3) did consistencyfixes across the code-base. Unfortunately this was done without enoughchecks on the actual meaning of one of the fixes, which means error...
Allow clock skews in certificate verification
Currently we allow for up to NODE_MAX_CLOCK_SKEW time differencebetween nodes in some operations, but not everywhere: SSL certificateverification (import/export, both intra and inter-cluster) has a zerolimit (downwards), and a week upwards. This can cause even...
Correct capitalisation of two Luxi calls
Two Luxi calls have inconsistent an name/value mapping (in the Pythoncode):
- REQ_AUTOARCHIVE_JOBS versus AutoArchiveJobs (versus AutoarchiveJobs)- REQ_QUEUE_SET_DRAIN_FLAG versus SetDrainFlag (no Queue)
While these are only a consistency issue, let's fix them so that the...
QA: Enable use of OR conditions in test checks
Until now “TestRunIf” and “TestEnabled” could only handle AND. With thispatch a new class named “Either” is added to “qa_config” and allows OR.The name “Either” was chosen instead of “Or” as the latter is very close...
Fix rapi.testutils unittest
Since we use a testutils.InputTestClient(), then the actual errorexpected is VerificationError, and not GanetiApiError (which is usedat real run-time).
Add more RAPI test utilities
This patch adds a mock RAPI client to test input values to methods. Allmethods either raise an exception if there was a problem or return None.Third-party code can use this to test their input values to the RAPIclient.
rapi.testutils.FakeCurl: Add header support
With this patch headers are constructed from the PycURL optionsand passed to the mock implementation.
Move _FakeCurl from tests/ganeti.rapi.client to ganeti.rapi.testutils
This is preparation for the mock system, where we need the same cURLmock.
Signed-off-by: René Nussbaumer <rn@google.com>Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Convert listing exports to query2
This solves one case where locks are acquired during LUXI queries.Pretty late into the transition I noticed that OpBackupQuery had a“use_locking” parameter for a long time, but didn't use it. Sincemost of the other changes were already and this allows exports to...
Copy debug level, priority and set comment for LU-generated opcodes
Before this patch, a node evacuation submitted with high priority wouldonly compute the solution at that priority, but the actual evacuationran at normal priority.
Fix query unittests after converting jobs to query2
I missed these among some shelltest-related failures.
jqueue: Convert GetInfo to query2
This rather inefficient implementation (fields are evaluated on everycall to GetInfo) is not good for WaitForJobChanges and doesn't supportfilters, but that will be rectified in later patches.
qlang.MakeFilter: Enable use of different name field
Jobs don't have a “name” field, so we must be able to controlthe field used for simple filters.
Merge cli.FormatTimestamp and utils.FormatTime
… to some degree at least. Unittests are included.
constants: Don't hardcode priorities for LOCK_ATTEMPTS_TIMEOUT
Also include unittest for LOCK_ATTEMPTS_TIMEOUT.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Add whitelist for opcodes using BGL
RPC: Add a new client type for DNS only
This patch moves the “call_version” to a new RPC client definition andthen adds a new runner using the DNS resolver for getting the hostaddress.
The standard “BootstrapRunner”, where the call was before, tries to...
Add trivial tests for gnt-* cli
While testing some other stuff, I realised that the gnt-* commandscould be broken (as in, the script fails with syntax errors), but makecheck doesn't detect it. Since we have shelltest, we can now addtrivial tests for this case....
cmdlib: Stop forking in LUClusterQuery
While debugging another issue we realized that LUClusterQuery forks.This turned out to be the “platform.architecture” function from thePython library. It uses the “file” command to determine the architectureof the Python binary....
Convert manual shell tests to shelltestrunner
This is more of a RFC… Basically most of the shell-based tests areconverted from exec+grep to shelltestrunner.
Things are not all fine and nice though:
- we have dependencies between tests, as some generate some data files...
locking: Handle spurious notifications on lock acquire
This was already a TODO since the implementation of lock priorities inSeptember 2010. Under certain conditions a waiting acquire can benotified at a time when it can't actually get the lock. In this case it...
locking: Fix lock deletion with timeout
While working on another SharedLock fix I realized timeouts on lockdeletion don't work very well if the timeout actually expires. Thispatch fixes the issue and adds a new unittest.
Move _TimeoutExpired to utils
Signed-off-by: Andrea Spadaccini <spadaccio@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
(cherry picked from commit f8326fcaac87958241d78526e5868d23d78ac286)
Add tests for custom text files
Add RAPI shell-level unittest
Add new hail (shell) tests
Using some hand-crafted files (manually modified from a real clusterrunning master branch), we test that hail behaves as we expect:doesn't load obvious wrong data, can do relocations/allocations, etc.
Signed-off-by: Iustin Pop <iustin@google.com>...
Rename spindle_usage to spindle_use on Ganeti side
ipolicy: Keep track of spindle usage
Fix lint error in commit 035b33e2
Commit 035b33e2 forgot one blank space, and current pylint in DebianSid doesn't run on our code… so I didn't realise this until aftercommit, sorry.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Fix tempfile reset code & test on newer Python
Python 2.7.3 (rc status) and 3.2.3/3.3 (rc, respectively alpha status)have fixed http://bugs.python.org/issue12856 which we worked aroundourselves.
This means two things:
- we don't need to manually reset the module...
Merge remote-tracking branch 'origin/devel-2.5'
Conflicts: Makefile.am (trivial, test data files added on both branches) lib/opcodes.py (trivial, master renamed filter→qfilter, 2.5 fixed the type of the parameter)...
Add result checks for OpTags*
opcodes: Annotate the OP_RESULT of query operations
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Fix asyncnotifier unit tests
On my workstation, if I run the following two tests in this exactorder:
- TestSingleFileEventHandler.testError- TestSingleFileEventHandler.testReplace
the second test will fail, because there is no "unregister" ofprevious tests (and the poll() done by asyncore will fire on the...
Add ndp/* fields to group/node query
Also change the meaning of group ndparams to mean the actual (notcustom) nd params, and add custom_ndparams for the current meaning ofndparams.
Add new spindle_count node parameter
Currently this is not handled by Ganeti, just recorded.
opcodes: Add result checks for OpBackup*
Improve opcode result check test to have a whitelist
Only whitelisted opcodes may not have a result check.
config: Add check for disk's “iv_name”
This check verifies “iv_name” of all instance disks. If one is wrong(which shouldn't happen in the first place), cluster verification willwarn:
“ERROR: cluster: Instance 'inst.example.com' has wrongly named disks:...
cli: Handle negative numbers when parsing key-value assignments
To remove the last disk, or to add a NIC to the end, one can use theindex 1. This wouldn't work as intended as “” is a special prefix.
utils.text: Add function to truncate string
The function adds an ellipse if the string was actually truncated. Alsostart using it in mcpu for result checks (where the message is alsoslightly changed to use a colon).
LUInstanceSetParams: Update disk's “iv_name”
When modifications are made, disks may not have the same index anymore.Updating all disks fixes this.
ApplyContainerMods: Fix issues with indices
When adding an item the index given to the callback function would beincorrect under certain conditions. This patch also adds assertions andmore tests.
gnt-instance modify: Support new-style NIC/disk modifications
This patch adds support for adding/removing NICs/disks at arbitraryindices on the command line. To add a disk at a specified index, use“--disk 3:size=16G”. To remove the second disk, use “--disk 2:remove”....
Add unittest for cmdlib._GenerateDiskTemplate
This is in preparation to de-duplicating significant chunks of code incmdlib._GenerateDiskTemplate.
rapi.testutils: Add exported functions to verify opcode input/result
These can be used by third-party code to verify mock code. Further workon mocks is forthcoming, so this is only a start.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>...
ApplyContainerMods: Return changes from callbacks
… instead of passing the list of changes as a parameter.
cmdlib: Add generic algorithm for modifying NICs/disks
While preparing this patch series I identified at least three differentimplementations of the algorithm for adding/removing/changingNICs/disks. These two functions and corresponding unittests provide a...
Fix unittest breakage after commit a2aadb34b
Thou shalt run “make check” before sending patches.
rapi.client: Removing constants not needed anymore
They were not referenced through the whole code and were marked internalonly.
Add stricter checks for OpInstanceSetParams.{nics,disks}
Stricter check for OS modifications passed to OpClusterSetParams
Don't just check the first element of each item, but also make sure theOS name is a string and not empty.
Implement rbd disk template
Introduce the rbd disk template, which handles provisioning andmanagement of instance disks as block devices mapped to rbd volumeson a RADOS cluster.
The rbd template is of type DTS_EXT_MIRROR, meaning rbd-basedinstances can failover or migrate to any node, without...
Add UnescapeAndSplit unittest for multi-escapes
This would have caught the bug in the first place. Argh,hand-generated test cases!
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
cmdlib._ComputeMinMaxSpec: Add unittest for this function
cmdlib: Rename _CheckMinMaxSpecs to _ComputeMinMaxSpec
This is to reflect the fact that _Check functions do raise Op*Errorwhich is not the case for _CheckMinMaxSpec.
_CheckTargetNodeIPolicy: instance.primary_node is not objects.Node
Merge branch 'devel-2.4' into stable-2.5
rpc: More unittests
- Fix: Don't check st_atime for temporary file for upload tests- Add tests for five encoders
cmdlib: Adding helper for instance policy
rpc.ConfigRunner: Fix uploading file
Commit 601dfcb made some changes to the default encoders. This made“ConfigRunner.call_upload_file” fail due to a missing encoder. Thispatch applies the necessary changes to ConfigRunner and updates theunittest for “call_upload_file”....
Conflicts: NEWS: trivial
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Add unittest for uploading file via RPC
After some preparing patches this unittest can finally be added.
Add unittests for RPC client
This patch adds a number of unittests for the RPC client base class.Some small changes were necessary in “rpc.py” to allow for bettertesting.
Add unittest for RPC compression