Rename leftovers from remote to restricted commands
As per Iustin Pop's suggestion in <20121115131730.GX824@google.com> on<ganeti-devel@googlegroups.com>.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
backend: Rename RunRemoteCommand to RunRestrictedCmd
Add tests for repr in locking classes
“locking.PipeCondition” and “locking.SharedLock” define “__repr__”,which until now was not tested at all.
test/*.py: s/'/"/
Now that 2.6 is essentially finished and 2.7 going to be branchedsoon-ish, I thought it would be a good moment to replace some singlequotes in test/*.py. Merge pains should be limited.
In one place in test/ganeti.locking_unittest.py, spaces are added for...
Add unit test for default parameter default values
Fails if the default value of an opcode parameter doesn't verify.
locking: Add test for downgrade without names
Until now there was no test for calling “LockSet.downgrade” withoutspecifying any names.
Fixes to pass unittests (make check)
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>Reviewed-by: Iustin Pop <iustin@google.com>
Add DRBD parser unit tests
This adds tests that existing test files can be parsed by the Haskellparser as well, plus one new test file.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Cleanup ht's use of positive/strictpositive
Currently, ht.py uses a bad terminology for positive/non-negativenumbers. Per http://en.wikipedia.org/wiki/Positive_number, this is thecorrect terminology:
- A number is positive if it is greater than zero.- A number is negative if it is less than zero....
Add opcode for running commands remotely
The opcode doesn't pay attention to the build-time flag to enable ordisable restricted commands. In a cluster different nodes could havedifferent settings.
Node locks are acquired in shared mode by default, but the use of an...
backend: Implement remote commands
As per design document (doc/design-remote-commands.rst), a number ofrather strict tests is applied to any incoming request, a delay isinserted upon errors and returned error messages are very generic(unless it's the actual command that failed). There are unit tests for...
Add unit test for RAPI handler access definitions
- Ensure query-related resources have the same access permissions (specifically “/2/query/*” and “/2/*/console”)- Check access permission consistency (write implies read)
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
rapi: Add new user option for querying
This was requested in issue 301. Before this patch, requests to“/2/query/*” and “/2/instances/*/console” would require authenticationwith a user with write access. Since that is not strictly necessary, anew user option named “read” is added....
Don't check for remote command directory as file storage
This test does not work properly if localstatedir is not “/etc”.
pathutils: Add directory for remote commands
Also add tests to ensure it's never allowed as a file storage path. Aconstant for the lock file is also added.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Add previously missing node daemon GID to getent mock
The UID is there, the GID wasn't.
Add test utility to count calls to function
In some cases it's nice to verify a function has been called exactly Ntimes. This is going to be used in tests for remote commands.
Add new test for RAPI
Unlike existing tests, this actually tests RAPI at the interface withthe HTTP server. This way authentification can also be tested. A testfor “/2/query/…” is included as it's a bit special.
Expose changing job priority via LUXI
A new LUXI request is added, in both Python and Haskell.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
workerpool: Add method to change task's priority
Using the task ID a pending task's priority can be changed. This will beused to change the priority of jobs in the workerpool.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
workerpool: Preserve task number when deferring
When a task is deferred it should receive the same task ID upon beingreturned to the pool.
jqueue: Allow changing of job priority
This is due to a feature request. Sometimes one wants to change thepriority of a job after it has been submitted, e.g. after submitting animportant job only to later notice many other pending jobs which will beprocessed first. Priority changes only take effect at the next lock...
workerpool: Change data structure for priority change
To prepare for the addition of a new function allowing changing apending task's priority, the internal data structure is slightlychanged. The (optional) task ID is stored as part of the task entry. A...
RunCmd: Expose "postfork" callback
The “_postfork_fn” parameter was only used for tests until now. Toimplement a good locking scheme, remote commands must also make use ofthis callback to release a lock when the command was successfullystarted (but did not yet finish)....
jqueue/mcpu: Determine priority using callback
Instead of being given the priority for acquiring locks by means of aparameter, mcpu will now call back. This is in preparation forimplementing a command to change a job's priority on the fly and allowsto change it while locks are being acquired (taking effect on the next...
Merge branch 'devel-2.6'
Merge branch 'stable-2.6' into devel-2.6
rapi.testutils: Return headers from mock utility
A newly added test for RAPI will also verify the returned headers. Atest in ganeti.rapi.client_unittest.py is split into smaller stand-alonetests.
Add missing tests for commit f0d2286
Commit f0d2286 changed the logic ofgnt_instance._ConvertNicDiskModifications to also allow a parameternamed “modify”. Unfortunately the corresponding unittest was notupdated. An “if”/“else” condition is also merged....
Add utility to check if file is executable
This replaces direct calls to “os.access” and“os.path.exists”/“os.path.isfile”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
utils.io: Improve handling of double and single slashes
Up until now “IsBelowDir("/", …)” would never return True. The reasonwas that an additional slash was added to the root path resulting in“//", which is “implementation-defined” in posix and treated specially...
jqueue: Return jobs to queue when shutting down
When a job is still waiting for locks and the queue is shutting down,they should be returned and not actually start processing. Until nowjobs which transitioned from “queued” to “waiting” were alreadyconsidered to be running as far as the shutdown code was concerned....
Remove duplicate workerpool test
Commit 52c47e4e (July 2010) added the exact test twice, probably due toa copy & paste error.
cfgupgrade: Write file for file storage paths
When file storage is used this file is now mandatory.
Make Paramiko an optional dependency for listrunner
With the move away from “setup-ssh”, Paramiko is no longer necessary toconfigure SSH on nodes.
ssh: Add function to get all of user's SSH files
This new function returns the file paths for all of a user's SSH-relatedfiles (RSA, DSA and authorized_keys).
RunCmd: Support standard input file descriptor
This patch changes “utils.RunCmd” to accept a file-like object or anumeric file descriptor which will be used as the command's standardinput. One use-case will be to pass all necessary data to“prepare-node-join”....
prepare-node-join: Swap private and public keys
Other places, such as “ssh.GetUserFiles”, use a structure where theprivate key comes before the private key. Until now prepare-node-joindid the opposite, that is the public key came first. To avoid confusion...
prepare-node-join: Use public key directly for auth…_keys
A public key already includes the necessary prefix (“ssh-rsa” or“ssh-dss”), so there is no need to add it again.
ssh.GetUserFiles: Parameter to disable directory check
Without this parameter, either an error would be raised or “.ssh” wouldhave to be created. Now it is possible to retrieve the paths withoutrequiring the “.ssh” directory to exist.
Remove unused cache implementation
Note that this commit has no Makefile.am changes, as the files werenot actually used. So it's better to actually remove them.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
bdev: Add verification for file storage paths
An earlier version of this patch series verified all paths in cmdlib inthe master daemon. With this change all that verification code is movedto bdev to run inside the node daemon. The checks are much stricter...
Add initial implementation of prepare-node-join
This is a new tool as per the design document “design-ssh-setup”. Itreceives a JSON data structure on its standard input and configures theSSH daemon and root's SSH keys accordingly. Unit tests are included....
ssh.GetUserFiles: RSA support, unit tests
This patch changes “ssh.GetUserFiles” to support two different kinds ofSSH keys, RSA and DSA. Before it would always use DSA. Newly writtenunit tests are included.
Compare significant fields only for simple SSH keys
For simple SSH keys, that is those without options such as“command="…"”, only the first two parts need to be compared. The thirdfield is a free-form comment.
This patch changes the comparison used in...
test/*.py: Replace '' with ""
There might be more, but at least replace all these low-hanging fruits.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Group.hs: add 'allTags'; adjust loaders and test data for it
This commit adds a Group.allTags field to store the tags of node groups,and teaches each loader backend in HTools to populate it (additionally, theIAllocator class in lib/cmdlib.py now includes tags for groups too). Test...
htools-excl.test: add test case for exclusion tags in hbal
In preparation for future modifications in the exclusion tags field, add atest that verifies that exclusion tags are being honored: in a test clusterwith two instances of the same exclusion group in each node, hbal should...
gnt-job cancel: Confirmation and selection of jobs
New parameters, “--pending”, “--queued” and “--waiting”, are added toselect all jobs in the respective state. If one of those options is usedand “--force” is not given, the user is asked to confirm the operation....
Replace custom algorithm in constants unittest
There is no need for the “_IsUniqueSequence” function anymore, it caneasily be replaced by utils.FindDuplicates. Also, pass the message as akeyword parameter and use the more commonly used assert* functions....
Add new constant for pending job status
This constant contains the job status' “queued”, “waiting” and“cancelled”.
vcluster: Don't virtualize /etc/hosts path
/etc/hosts is a bit special as it's a system-wide file and the virtualcluster/node root doesn't apply. The modification of /etc/hosts shouldbe disabled in virtual clusters. If it isn't, however, the vclusterfunctions would raise an exception complaining about a path outside of...
jqueue: Add new in-memory attribute for archived jobs
This attribute is set to True for jobs which were restored from anarchived file. A new filter will act on this field.
query: Report data type for unary operators
All data kinds (used to restrict the data collected) referenced in afilter can be requested once it's been “compiled”. However, the kindsof fields used in boolean expressions (e.g. ["?", "xyz"]) were notrecorded. This patch changes the code accordingly and provides a unit...
Add basic unit tests for "gnt-cluster epo"
This patch adds some unit tests for “gnt-cluster epo”. Not everything iscovered, but at least the bug fixed in the previous patch is.
jstore: Nicer error message on non-numeric file content
An error like “invalid literal for int() with base 10” can be quiteconfusing.
Merge ganeti-master-cleaner back into ganeti-cleaner
As I wrote during/after the review on commit 2958c56, “ganeti-cleaner:Separate queue cleaning code”, while I appreciated the permissionseparation, I didn't like too much the file-based approach:
- it is a very simple script, and lots of the code is duplicated...
Explicitly ask for the default iallocator in commands
Now "gnt-instance recreate-disks" uses the default iallocator when "." isspecified as the iallocator. For uniformity, the same behavior applies tothese commands: gnt-node evacuate gnt-instance migrate...
bdev: Add functions to verify file storage paths
- LoadAllowedFileStoragePaths: Loads a list of allowed file storage paths from a file- CheckFileStoragePath: Checks a path against the list of allowed paths
The unit test for “utils.IsBelowDir” is updated with cases which weren't...
utils.FilterEmptyLinesAndComments: Return list
We don't use generators often and lists are easier to re-use.
Improve disk wipe unit test
- Don't hardcode node name in some places- Don't define functions inside functions- Simplify code for testing with and without offset, this is now in two separate tests
Wipe added space when growing disks
This patch adds code to wipe newly added disk space when growing disksusing “gnt-instance grow-disk”. “New disk space” is defined as the deltabetween the old block device size (not necessarily equal to the amountrecorded in the configuration) and the new recorded size. Extra caution...
Add unit tests for cmdlib._WipeDisks
This is in preparation for adding disk wipe on growing disks.
Add unit test for FilterEmptyLinesAndComments
I somehow forgot this in the previous patch.
cli: Exit with status 0 for --help
This patch is somewhat longer than was anticipated. Before, commands like“gnt-instance --help” would exit with the status code 1 (failure). Withthis patch, those commands exits with 0 (success) while unknown commandsstill print the usage screen and exit with status 1....
ganeti-cleaner: Separate queue cleaning code
This code does not need to run as root, therefore it's better to splitit out. It is now run with the same permissions as the master daemon.
Stop hardcoding root user
Some parts of the code still use a hardcoded user name: root. This patchreplaces all with a constant specified at build time. The end goal is tomake it possible to run a Ganeti cluster without any special privileges(of course this will prevent some functionality from working)....
Populate the instance dict with opcode infos
This is needed so we can load it as an opcode. However, this informationshould not be available on the client, so we inject that info when wereceive the request.
Provided unittest to verify behaviour.
Signed-off-by: René Nussbaumer <rn@google.com>...
Adding RAPI client for instance multi allocation
It was very easy to refactor CreateInstance to repurpose some part of itfor the instance multi allocation request. So we do this.
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Add tests for bash completion
Given the number of settings that need to be controlled a plain bashscript was a better choice over shelltestrunner. Just a few completionsare attempted, but among them should be the most critical ones (e.g.“gnt-instance add --node …”....
Fix shelltests for ganeti-*
They were never run.
Adding RAPI resource for multi-allocation
This is straightforward.
Adding the new opcode for multi-allocation
Skeleton for the multi-allocation opcode
Adding some fundamental unittests for iallocator
This test covers the bug fixes found in the previous two patches
Add new module for virtual clusters
This module will take care of managing paths for virtual clusters.Unittests are included (100% coverage).
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
ShellWriter: Add parameter to disable indentation
This will be used to write a more compact bash completion script.
Migrate test/ganeti.*.py from constants to pathutils
File system paths moved from constants to pathutils.
rpc: Remove duplicated logic, fix unittests
Commit 5fce6a89 changed RpcRunner._InstDict to add the disk parameterson all encoded instances. It didn't remove a special case in“_InstDictOspDp”. Update and fix unittests as well.
cmdlib: Handle locking.ALL_SET correctly when copying locks
When locks are copied “locking.ALL_SET” must be handled separately(ALL_SET has the value None). Reported by Constantinos Venetsanopouloswho saw failover for RDB-based instances not working.
Refactor IAllocator code
The IAllocator class was handling all the requests on its own, passingin parameters on top level which works, but is hard to maintain and notflexible.
With the upcoming change to the IAllocator for MultiAllocate we can'tuse the toplevel parameters anymore. Therefore, we refactor the code...
Make the slots functionality more modular
As we will introduce another set of containers using the slots trickwe abstract away as much as possible to separate bases classes. Thechild classes then adapt them for their needs. This leads to less code...
Move _CalculateGroupIPolicy and _ComputeDiskSize
This makes it possible to better untangle the IAllocator code, whichwould otherwise lead to a recursive import structure as we need thosehelpers there as well.
Move shelltests from test/ to htest/shelltests/
This is another rather trivial patch, moving all the (htools)shelltests to their own directory.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Move htools test files from test/data/htools to htest/data
Following up on the program moves, we now move the test data files.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Agata Murawska <agatamurawska@google.com>
Implement support for query only clients in Rapi
This implements the same query=True|False functionality as inGetClient for cli.py, however since the RAPI code is much moreunit-tested (and the unit-test clients are mocked, for the most part,without support for addresses) we have to do many adaptations in the...
Move necessary constants from rpc & rpc_defs to constants
As timeout constants will now be used also on Haskell side, they havebeen moved from rpc_defs and rpc to constants. The same is true forconnection timeout for connecting to nodes.
Signed-off-by: Agata Murawska <agatamurawska@google.com>...
Add test for checking that all gnt-* subcommands run OK
This is a bit of a shell munging trickery, but works for now. Makingit more generic can be done later.
Switch job IDs to numeric
This has been a long-standing cleanup item, which we've alwaysrefrained from doing due to the high estimated effort needed.
In reality, it turned out that after some infrastructure improvements(the previous patches), the actual job queue-related changes are quite...
Make jqueue unittests more generic
This patch removes/abstracts some hardcoded values in the jqueueunittests. Currently we have a per-resource type name field, but thename field values are hardcoded at the call sites, instead of beingabstracted into separate variables....
Merge branch 'stable-2.6'
Add a shell test for hbal and split instances
This is not perfect, as we only test that hbal completes successfullyand that it show a score improvement, but it's better than nothing.
Add newline at the end of shelltest files
Signed-off-by: Agata Murawska <agatamurawska@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Add forgotten unittest changes for instance_os_add
The previous patch which fixed disk parameters didn't adapt theunittests so it lead to failing QA.
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Agata Murawska <agatamurawska@google.com>
Style fixes in shelltests
Shelltestrunner tests for hcheck
Simple tests for hcheck using shelltestrunner. Among other, we checkthat we can run hcheck on multi-group cluster.
Update the hooks documentation
Also provide some extended unittests to catch those cases.
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>Reviewed-by: Agata Murawska <agatamurawska@google.com>