Unit tests for bdev and utils.lvm
Also amended a comment of a tested method.The tests are for methods introduced in commit 63c73073 (LVM disk creationuses dedicated PVs).
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Support instance-minor pairing in the DRBD collector
This commits enables the DRBD data collector to use the Confd client togather information about the pairing between DRBD minors and instances.
For testing purposes, the DRBD data collector now requires either zero...
Support integrating instance information in the DRBD parser
This commit modifies the DRBD parser and its data structures to includeinformation about the instance a DRBD minor belongs to.
Test files have been updated as well, to support the new field in the data...
Add "use_locking" parameter to network query opcode
This was extracted from a patch by Dimitris Aragiorgis with the subject“Add locking to _NetworkQuery”. The rest of the patch did no longerapply and will require more work.
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
Move src/Ganeti/HTools/Program.hs to Program/Main.hs
This removes one more tab conflict; this is the last module in ourcode where we have both x.hs and x/.
Furthermore, we collapse all actual code into the new Main.hs module,leaving the htools.hs basically empty (will allow better testing in...
Remove duplicate test file functions
This patch unifies the functions that generated the treereorganisation discussion in the first place :)
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Rename test/hs/test to htest
The current base name of 'test' means that this binary will generateprofiling, coverage, etc. files with the name 'test.*', result intab-completion conflicts with the test directory. Let's slightlychange its base name for typing easiness....
Move htest/ files under the test/ tree
htest/data becomes test/data/htools (basically reverting commit8feabc89), and htest/* becomes test/hs/*.
Most changes beside the rename are trivial s/…/…, with the exceptionof autotools/run-in-tempdir, which needed some more changes now that...
Move python test files to test/py
This is the first step of the test files reorganisation: moving test/*(except test/data) to new directory test/py/.
Add the gnt-storage client
Add a new client called 'gnt-storage'.The client interacts with the ExtStorage interface, similarly tothe way gnt-os interacts with the OS interface.
For now, only two commands are supported: 'info' and 'diagnose'.
'diagnose' calculates the node status of each provider on each node,...
gnt-* list-fields: Show friendly field type
For people writing query filters (documented in ganeti(7)) knowing a field'stype can be useful.
$ gnt-instance list-fields name be/memoryName Type Title Descriptionname Text Instance Instance name...
Add utility to format dictionary as key=value strings
This will be used in QA to format network interface parameters.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
cmdlib: Additional opportunistic locking check
If opportunistic locking is requested in the opcode, but no iallocatoris used (not specified or no default), an error will be raised.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Replace frozenset with compat.UniqueFrozenset
This is not a trivial s/frozenset/compat.UniqueFrozenset/, but ratheronly replaces “frozenset” where appropriate. Most of the places are“static” information that doesn't change after the module has beenloaded....
Add exclusive_storage node parameter
Unit tests updated and expanded with an inheritance check.
The flag has no effect yet.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Fix error during cluster initialization due to hv_kvm
Commit 141d148 was a bit too enthusiastic. The three parameters added tothe list of parameters to be checked default to a value not evaluatingto false, leading to a failure on cluster initialization....
Add test for SPICE parameter list, add missing ones
“_SPICE_ADDITIONAL_PARAMS” is supposed to be the full list ofSPICE-related KVM hypervisor parameters with the exception of“HV_KVM_SPICE_BIND”. The new test checks if all parameters starting with“HV_KVM_SPICE_*” are included. Three previously missing parameters are...
Add optional formatting for OP_DSC_FIELD
For some opcodes, the output is not "stable", and depends on the exactinput values; this makes it harder to check consistency againstHaskell code.
To compensate for this, we add a way to override the formatting of the...
Add RPC for setting watcher pause
The watcher pause file should be set/unset on all nodes at once, notonly the master node. For that a new RPC is needed.
Rename test for backend._CommonRestrictedCmdCheck
“TestWriteFile” was not renamed when adding the file based on“ganeti.utils.io_unittest-runasroot.py”.
jqueue: Improve inotify error reporting
This addresses issue 218. When the number of inotify watches isexhausted, for example by being set too low from the beginning or byother programs, waiting for a job to change would just report a lost job(e.g. “Error checking job status: Job with id 7817 lost”)....
Improve test for tools.ensure_dirs
- Add more checks, some of them are deliberately redundant- Descriptive error messages- Add comment describing order to “tools.ensure_dirs”- Avoid copying a list in an assertion in “tools.ensure_dirs”
Remove checks wrt IDISK_PARAMS from OpCode level
Change the "--disks" option validation, to just check the formatof the dict and do not check whether the keys are included in theIDISK_PARAMS constant at OpCode level. This allows the passing ofarbitrary parameters at the CLI, which will then be logically...
Add utility function to create frozenset with unique values
When used instead of a plain call to “frozenset”, this would haveavoided the issue fixed in commit e2dd6ec. The new function is locatedin the “compat” module as it will be used at module load time in most...
Export error codes from RAPI client module
Until now the error codes were not available from the RAPI clientmodule. A newly added unit test ensures all error codes are contained in“ECODE_ALL”, as well as ensuring consistency between the RAPI client and...
mcpu: Verify node allocation lock mode
Add verification code to mcpu to check an LU's locks. Two whitelists areprovided to exclude LUs from the two tests.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Fix node-daemon-setup test with older pyOpenSSL
Older versions use “-----BEGIN RSA PRIVATE KEY-----” instead of“-----BEGIN PRIVATE KEY-----”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Add tool to configure node daemon
The design for this is in “doc/design-node-add.rst”. The tool receives aJSON data structure on stdin and configures the node's daemon afterverifying the received values.
jqueue: Don't modify input opcode when changing priority
Commit 4679547 implemented the ability to change job's priority after itwas submitted. The code contained a bug whereby it would modify theinput data for an opcode, something the job queue shouldn't do (logical...
iallocator: Add node whitelist
In the future instance creations might have a lock on all nodes as wasthe case until the implementation of opportunistic locking. Nodes forwhich the lock is not held will be shown to the iallocator plugin as ifthey were marked offline....
utils.text: Function to verify MAC address prefix
The network management code needs to verify a MAC address prefix.Instead of (ab)using NormalizeAndValidateMac, clean code should be used.Unit tests for NormalizeAndValidateMac are updated and new ones for...
Factorize code for checking node daemon certificate
This code is going to be used by a new utility for setting up the nodedaemon. Unit tests are updated/added.
Additionally, the certificate and key stored in “server.pem” areverified, too.
locking: Implement opportunistic locking in LockSet
This patch adds a new parameter to “LockSet.acquire” named“opportunistic”. When enabled the lockset will try to acquire as manylocks as possible, but it won't wait for them (with the exception of thelockset-internal lock in case the whole set is acquired). This is...
Add ssconf function to read all files
Configuring a node daemon on a newly added node will need all ssconfvalues.
ssconf: Add dry-run support for writing files
A new utility for configuring the node daemon will support a dry-runmode. This patch adds the necessary functionality to“ssconf.SimpleStore” and provides comprehensive tests for“SimpleStore.WriteFiles”. To enable the latter, a testing-only parameter...
ssconf: Add function to verify keys
The new utility for configuring the node daemon will have to checkwhether it received valid ssconf names.
Add test for mutable default values in opcode parameters
This is not comprehensive, since in Python one can't determine what isand what is not mutable; but I've added a few base cases (list, dict,set).
The patch also improves (makes more uniform) the error messages in the...
Introduce ht.TMaybeValueNone and ht.TValueNone
TValueNone checks if a value is "none" and TMaybeValueNone is a wrapperof TOr(TValueNone, x). This is used by OpNetworkSetParam in order toreset a network value (e.g. mac_prefix, gateway, etc.)
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>...
opcodes: Replace manual loop with map
Also remove a superfluous empty line in test file.
Fix type descriptions in RAPI documentation
This patch adds descriptors to the “_CheckCIDR*” functions in opcodesand improves the descriptions generated by “ht.TInstanceOf”, therebyindirectly fixing bad type descriptions in the RAPI documentation.
Before this patch:...
Move cluster verification out of prepare-node-join
A new tool for configuring the node daemon will also have to verify thecluster name, so it's better to have this function in a central place.In the process of moving it to ssconf it is also changed to use...
ssconf: Verify file size when reading, add some tests
Until now ssconf would limit the amount read from files to 128 KiB andsilently ignored files larger than that. With this patch a check isadded by using fstat(2) on the file descriptor while it's being read....
Factorize code to load and verify JSON
A new tool to configure the node daemon will also have to load andverify JSON data.
Factorize logging setup in tools
Most tools had their own “SetupLogging” function, but they were allessentially the same. This patch adds a generic version to “utils.log”and provides unit tests.
locking: Method to check if LockSet is fully acquired
A new method is added to check whether the LockSet-internal lock isheld. This is the case after LockSet.acquire was called withlocking.ALL_SET.
Unit tests are updated, including one where the list of names must be...
Add new lock level for node allocations
The new lock is similar to the BGL in the sense that it has its ownlevel and there is only one. It is called “node allocation lock”.Logical units will use it to synchronize with instance creations, whichin turn will start using opportunistic locks on nodes....
Introduce a TMaybe combinator
We have many cases in the code where we write TOr(TNone, a), so let'sintroduce a combinator that simplifies this case.
Beside replacing the above with TMaybe(a), I did a few other parameterfixes:
- noop change TOr(TNone, TDict) to TMaybeDict...
Replace dict() with {}
The network patches and an existing test added function-call baseddict construction as opposed to literal sintax.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Rename leftovers from remote to restricted commands
As per Iustin Pop's suggestion in <20121115131730.GX824@google.com> on<ganeti-devel@googlegroups.com>.
backend: Rename RunRemoteCommand to RunRestrictedCmd
Add tests for repr in locking classes
“locking.PipeCondition” and “locking.SharedLock” define “__repr__”,which until now was not tested at all.
test/*.py: s/'/"/
Now that 2.6 is essentially finished and 2.7 going to be branchedsoon-ish, I thought it would be a good moment to replace some singlequotes in test/*.py. Merge pains should be limited.
In one place in test/ganeti.locking_unittest.py, spaces are added for...
Add unit test for default parameter default values
Fails if the default value of an opcode parameter doesn't verify.
locking: Add test for downgrade without names
Until now there was no test for calling “LockSet.downgrade” withoutspecifying any names.
Fixes to pass unittests (make check)
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>Reviewed-by: Iustin Pop <iustin@google.com>
Add DRBD parser unit tests
This adds tests that existing test files can be parsed by the Haskellparser as well, plus one new test file.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Cleanup ht's use of positive/strictpositive
Currently, ht.py uses a bad terminology for positive/non-negativenumbers. Per http://en.wikipedia.org/wiki/Positive_number, this is thecorrect terminology:
- A number is positive if it is greater than zero.- A number is negative if it is less than zero....
Add opcode for running commands remotely
The opcode doesn't pay attention to the build-time flag to enable ordisable restricted commands. In a cluster different nodes could havedifferent settings.
Node locks are acquired in shared mode by default, but the use of an...
backend: Implement remote commands
As per design document (doc/design-remote-commands.rst), a number ofrather strict tests is applied to any incoming request, a delay isinserted upon errors and returned error messages are very generic(unless it's the actual command that failed). There are unit tests for...
Add unit test for RAPI handler access definitions
- Ensure query-related resources have the same access permissions (specifically “/2/query/*” and “/2/*/console”)- Check access permission consistency (write implies read)
rapi: Add new user option for querying
This was requested in issue 301. Before this patch, requests to“/2/query/*” and “/2/instances/*/console” would require authenticationwith a user with write access. Since that is not strictly necessary, anew user option named “read” is added....
Don't check for remote command directory as file storage
This test does not work properly if localstatedir is not “/etc”.
pathutils: Add directory for remote commands
Also add tests to ensure it's never allowed as a file storage path. Aconstant for the lock file is also added.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Add previously missing node daemon GID to getent mock
The UID is there, the GID wasn't.
Add test utility to count calls to function
In some cases it's nice to verify a function has been called exactly Ntimes. This is going to be used in tests for remote commands.
Add new test for RAPI
Unlike existing tests, this actually tests RAPI at the interface withthe HTTP server. This way authentification can also be tested. A testfor “/2/query/…” is included as it's a bit special.
Expose changing job priority via LUXI
A new LUXI request is added, in both Python and Haskell.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
workerpool: Add method to change task's priority
Using the task ID a pending task's priority can be changed. This will beused to change the priority of jobs in the workerpool.
workerpool: Preserve task number when deferring
When a task is deferred it should receive the same task ID upon beingreturned to the pool.
jqueue: Allow changing of job priority
This is due to a feature request. Sometimes one wants to change thepriority of a job after it has been submitted, e.g. after submitting animportant job only to later notice many other pending jobs which will beprocessed first. Priority changes only take effect at the next lock...
workerpool: Change data structure for priority change
To prepare for the addition of a new function allowing changing apending task's priority, the internal data structure is slightlychanged. The (optional) task ID is stored as part of the task entry. A...
RunCmd: Expose "postfork" callback
The “_postfork_fn” parameter was only used for tests until now. Toimplement a good locking scheme, remote commands must also make use ofthis callback to release a lock when the command was successfullystarted (but did not yet finish)....
jqueue/mcpu: Determine priority using callback
Instead of being given the priority for acquiring locks by means of aparameter, mcpu will now call back. This is in preparation forimplementing a command to change a job's priority on the fly and allowsto change it while locks are being acquired (taking effect on the next...
Merge branch 'devel-2.6'
Merge branch 'stable-2.6' into devel-2.6
rapi.testutils: Return headers from mock utility
A newly added test for RAPI will also verify the returned headers. Atest in ganeti.rapi.client_unittest.py is split into smaller stand-alonetests.
Add missing tests for commit f0d2286
Commit f0d2286 changed the logic ofgnt_instance._ConvertNicDiskModifications to also allow a parameternamed “modify”. Unfortunately the corresponding unittest was notupdated. An “if”/“else” condition is also merged....
Add utility to check if file is executable
This replaces direct calls to “os.access” and“os.path.exists”/“os.path.isfile”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
utils.io: Improve handling of double and single slashes
Up until now “IsBelowDir("/", …)” would never return True. The reasonwas that an additional slash was added to the root path resulting in“//", which is “implementation-defined” in posix and treated specially...
jqueue: Return jobs to queue when shutting down
When a job is still waiting for locks and the queue is shutting down,they should be returned and not actually start processing. Until nowjobs which transitioned from “queued” to “waiting” were alreadyconsidered to be running as far as the shutdown code was concerned....
Remove duplicate workerpool test
Commit 52c47e4e (July 2010) added the exact test twice, probably due toa copy & paste error.
cfgupgrade: Write file for file storage paths
When file storage is used this file is now mandatory.
Make Paramiko an optional dependency for listrunner
With the move away from “setup-ssh”, Paramiko is no longer necessary toconfigure SSH on nodes.
ssh: Add function to get all of user's SSH files
This new function returns the file paths for all of a user's SSH-relatedfiles (RSA, DSA and authorized_keys).
RunCmd: Support standard input file descriptor
This patch changes “utils.RunCmd” to accept a file-like object or anumeric file descriptor which will be used as the command's standardinput. One use-case will be to pass all necessary data to“prepare-node-join”....
prepare-node-join: Swap private and public keys
Other places, such as “ssh.GetUserFiles”, use a structure where theprivate key comes before the private key. Until now prepare-node-joindid the opposite, that is the public key came first. To avoid confusion...
prepare-node-join: Use public key directly for auth…_keys
A public key already includes the necessary prefix (“ssh-rsa” or“ssh-dss”), so there is no need to add it again.
ssh.GetUserFiles: Parameter to disable directory check
Without this parameter, either an error would be raised or “.ssh” wouldhave to be created. Now it is possible to retrieve the paths withoutrequiring the “.ssh” directory to exist.
Remove unused cache implementation
Note that this commit has no Makefile.am changes, as the files werenot actually used. So it's better to actually remove them.
bdev: Add verification for file storage paths
An earlier version of this patch series verified all paths in cmdlib inthe master daemon. With this change all that verification code is movedto bdev to run inside the node daemon. The checks are much stricter...
Add initial implementation of prepare-node-join
This is a new tool as per the design document “design-ssh-setup”. Itreceives a JSON data structure on its standard input and configures theSSH daemon and root's SSH keys accordingly. Unit tests are included....
ssh.GetUserFiles: RSA support, unit tests
This patch changes “ssh.GetUserFiles” to support two different kinds ofSSH keys, RSA and DSA. Before it would always use DSA. Newly writtenunit tests are included.
Compare significant fields only for simple SSH keys
For simple SSH keys, that is those without options such as“command="…"”, only the first two parts need to be compared. The thirdfield is a free-form comment.
This patch changes the comparison used in...
test/*.py: Replace '' with ""
There might be more, but at least replace all these low-hanging fruits.
Group.hs: add 'allTags'; adjust loaders and test data for it
This commit adds a Group.allTags field to store the tags of node groups,and teaches each loader backend in HTools to populate it (additionally, theIAllocator class in lib/cmdlib.py now includes tags for groups too). Test...
htools-excl.test: add test case for exclusion tags in hbal
In preparation for future modifications in the exclusion tags field, add atest that verifies that exclusion tags are being honored: in a test clusterwith two instances of the same exclusion group in each node, hbal should...
gnt-job cancel: Confirmation and selection of jobs
New parameters, “--pending”, “--queued” and “--waiting”, are added toselect all jobs in the respective state. If one of those options is usedand “--force” is not given, the user is asked to confirm the operation....
Replace custom algorithm in constants unittest
There is no need for the “_IsUniqueSequence” function anymore, it caneasily be replaced by utils.FindDuplicates. Also, pass the message as akeyword parameter and use the more commonly used assert* functions....