ganeti.query_unittest.py: add tests for group queries
Signed-off-by: Adeodato Simo <dato@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
ganeti.query_unittest.py: test lock fields too
Additionally, change TestQueryFields.testSomeFields() to handle lists offields shorter than 20 elements.
Merge branch 'devel-2.3'
Convert “gnt-debug locks” to query2
Locks can now be queried using “Query(what="lock", …)” over LUXI.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
cfgupgrade: Check master name, clarify question
- Check hostname and abort if it doesn't match contents of “ssconf_master_node”, can be overridden using “--ignore-hostname” parameter.- Clarify confirmation question and don't mention instances anymore....
RAPI: Add resource to grow instance disk
baserlib: Add function for filling opcodes
This function makes use of the opcode parameters which now livedirectly in the opcode. A number of RAPI resources can now be simplified.
RAPI: Add resource to modify cluster
Improve opcode summary tests
Test full summary instead of just format.
Migrate code verifying opcode parameters to base class
This allows the function to be used in other places as well.An optional parameter is added to control whether defaultvalues should be set. Unittests are added, providing fullcoverage.
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
Improve tests for OP_ID
… by detecting duplicates.
cmdlib: Remove opcode parameters
Remove the parameter definitions and use those from the opcode classesinstead. Small style changes are also made (empty lines, wrapping).
opcodes: Add opcode parameter definitions
This is the first step for migrating them from cmdlib. A metaclass isused to define “__slots__” upon class creation time (not instantiation).
query2: Add new field status “offline”
This allows “gnt-node list” to show the difference between modes markedoffline and nodes with e.g. RPC errors (“(nodata)”). node1 is themaster, node2's node daemon crashed and node3 is marked offline:
$ gnt-node list -o name,offline,dtotal,dfree...
Add unittests for ht module
Conflicts: man/gnt-instance.sgml: Trivial merge in gnt-instance.rst...
cli: Change “<…>” in query output to “(…)”
This should reduce the amount of damage in case of accidential copy &paste.
jqueue: Fix cancelling while in waitlock in queue
Since the recent change to leave jobs in the “waitlock” status (commit5fd6b6947), cancelling a job while it's back in the queue would break.This patch handles these cases and adds a unittest.
Remove utils.FormatTimestampWithTZ
Long story short: time.strftime("%Z", time.localtime()) doesn't work,even though it's documented to be equivalent to time.strftime("%Z").
$ TZ=America/Sao_Paulo python -c 'import time; printtime.strftime("%Z"), time.strftime("%Z", time.localtime())'...
Ensure temp files from RunCmd tests are removed
utils: Timezone fixes and tests
- Update docstrings to explicitely mention Epoch- Fix timezone bug in FormatTimestampWithTZ, where it would use GMT/UTC when it should use the local timezone- Add unittests for time formatting functions
Move QueryFields to query module
Also replace “sorted” with “utils.NiceSort” now that it supports a keyfunction.
utils.NiceSort: Use sorted(), add keyfunc, unittests
This patch changes utils.NiceSort to use the built-in “sorted()” andgets rid of the intermediate list. Instead of wrapping the itemsourselves, a key function is used. The caller can specify another key...
Conflicts: man/ganeti-cleaner.sgml: Removed man/ganeti-confd.sgml: Removed man/ganeti-masterd.sgml: Removed...
jqueue: Keep jobs in “waitlock” while returning to queue
Iustin Pop reported that a job's file is updated many times while itwaits for locks held by other thread(s). After an investigation it wasconcluded that the reason was a design decision for job priorities to...
Improve jqueue unittests
- Verify job file updates- Ensure queue lock is released while executing opcode
cli: Add infrastructure for query2
A new function for formatting the query results is added,``FormatTable``. This was determined to be easier and safer thanmodifying the existing ``GenerateTable`` function while keepingbackwards compatibility for code not yet converted. The new code makes...
Add some unit tests for ConfigWriter.AddNodeGroup()
qlang: Add function to build simple filter
This will be used in clients to build the filters for query2.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
query: Handle items missing timestamps
In upgraded configurations, some items might miss the “ctime” and/or“mtime” values and need to be handled specially.
query: Add definition for instance queries
Group operations: update ganeti.rapi.client with all node group operations
Group operations: expose add/remove/rename in RAPI
utils: Add function to find duplicates in sequence
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Adeodato Simo <dato@google.com>
locking: add nodegroup lock level
This also changes masterd to initialize the ganeti's manager with thecurrent list of nodegroup uuids, and updates unittests
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Querying node groups: RAPI support
This implements /groups and /groups/%s RAPI end points.
Signed-off-by: Adeodato Simo <dato@google.com>Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Add OpQuery opcode
Add node query definition
This includes a bunch of helper functions which can be helpful for otherqueries, too. Unittests are included.
Add utility class for definition-based data queries
While not immediately visible, using definition-based queries will allowfor a number of optimizations and simplifications. This patch justcontains the utility class used for executing queries.
Add simple query filter parser
This parser reads only the format described by the query2design document: either an empty filter or an OR operatorwith equality checks as operands.
RAPI: Implement OS parameters for instance reinstallation
Dictionaries are hard to encode into query strings, therefore the“/2/instances/[instance_name]/reinstall” resource is changed to acceptits parameters via the request body. The old query string parameters are...
Move “rapi_users” file into separate directory
This reduces the number of notifications in “ganeti-rapi”. Until now itwas notified for every change in …/lib/ganeti and had to check whetherthe users file was affected. A symlink is always created in cfgupgrade...
impexpd: Implement support for IPv6
Improve unittests for the utils module
This just a random collection of unittest improvements. Coverageincreases from 73% to 76%.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Fix breakage from previous commit e695efb
Wrong file name in Makefile.am and wrong naming (.unittest vs_unittest).
Add missing paramter and unittests for this case
While git-am'ing the blockdev_prefix patch, I modified the hv_xen.pyfile but forgot to amend the patch :(, so here it is.
Furthermore, to detect such inconsistencies in the future, a unittest isadded that cross-checks the default parameters in constants.py and the...
asyncnotifier tests: simplify the setup code
First change is to replace "range(len(self.NOTIFIERS)))" withself.NOTIFIERS, as range(len(range(N))) ≡ range(N).
Sencond change is switch from an explicit indexing to use of zip.Unittests still pass :)
Signed-off-by: Iustin Pop <iustin@google.com>...
Add consistency test for mcpu dispatch table
Adding convenience getter for ndparams
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Fix utils unittest TestRunCmd.testTimeoutKill
Currently the test executes "/bin/sh -c 'trap "" TERM …" via the shell,which means we have two shells involved (the parent one and the childone, which does the trap). For some weird reason, this works for older...
Adding oob_program in the same style as beparams
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Temporarily remove the interactive tests
make check < /dev/null breaks these tests. Until we have change RunCmdto override stdin, we need to remove them.
Support timeouts in RunCmd
Further investigations have to be done for merging some of these bitstogether with import-export daemon which uses similiar logic.
Adding unittests for RunningTimeout
Switch man pages from SGML to RST
This patch changes the Makefile rules to build the man pages from theRST sources instead of the SGML ones.
Note: pandoc quotes by default the ‘@’ character in manpage output, sowe explicitely undo this before writing the .in file....
Add unittests for TemporaryReservationManager
And fix an error message.
Add support and checks for version in LUXI
A new constant, LUXI_VERSION, is used to verify the peer's version. Theversion is optional, so old(er) clients and servers talking to peers notsupporting it won't break. Example with mismatching library:
$ gnt-instance list...
Add test for modifiable locking levels
GanetiLockManager, remove default values
The nodes and instances parameters to the constructor are mandatoryanyway, as a value of None will fail when creating the LockSet. Ratherthan fixing this adding code lines, since we never used the defaultvalue, let's remove them and require that the parameters are passed....
Add a "safe" file wrapper over WriteFile
Add functions to read and compare file 'ID's
utils: Add function to find items in dictionary using regex
This basically extracts a small piece of code from ganeti-rapi and putsit into a utility function. RAPI resources are found using a dictionaryin which the keys can either be static strings or compiled regular...
Extract base class from SingleFileEventHandler
The base class can contain code useful to other inotify users.As it is “SingleFileEventHandler” can not be used in ganeti-rapi,therefore it'll use its own small inotify handler class basedon this base class....
http.auth.ReadPasswordFile: Don't read file directly
Reading the file before this function allows for better errorreporting.
jqueue: Fix bug when cancelling jobs
If a job was cancelled while it was waiting for locks, an assertionwould've failed. This patch fixes the problem and provides a unittest to check for this situation.
Use only one version of WritePidFile
This patch merges the pid file handling used for ganeti-* daemons andimpexp daemons. The latter version is used, since it's more reliable:uses locked pid files as opposed to checking 'live' processes.
Merge branch 'devel-2.2'
Add simple unittest for utils.CommaJoin
cli: Move parsing of --net option to separate function
This function will also be used in tools/move-instance.
Abstract OS name/variant functions
Currently, the computation of the 'pure' name or the variant ishardcoded and spread around the functions that need it. This is notnice, and in the future we'd spread it even more with more usage ofvariants/pure os names....
Move the uuid regexp to utils.py
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
jqueue: Use priority for acquiring locks
jqueue: Use timeout when acquiring locks
As already noted in the design document, an opcode's priority isincreased when the lock(s) can't be acquired within a certain amount oftime, except at the highest priority, where in such a case a blockingacquire is used....
mcpu: Adjust lock acquire strategy
The changes to job queue processing require some changes on this class'interface. LockAttemptTimeoutStrategy might move to another place, but that'llbe done in a later patch.
jqueue: Rename current_op to better reflect what it actually is
jqueue: Add unittest for _QueuedJob.CalcStatus
jqueue: Change model from per-job to per-opcode processing
In order to support priorities, the processing of jobs needs to bechanged. Instead of processing jobs as a whole, the code is changed toprocess one opcode at a time and then return to the queue. See the...
Always enable logging for unittests
By enabling all log levels, we ensure all calls are fully evaluated.There was one case in the workerpool where a call to “logging.debug”was wrong, but not caught in unittests because debug logging wasdisabled.
The optional environment variable “LOGTOSTDERR” can be set to...
Add small jqueue unittests
cli: Add option definition for priority
Add support for job priority to opcodes and job queue objects
This allows clients to submit opcodes with a priority. Except for beingtracked by the job queue, it is not yet used by any code.
Unittests for jqueue._QueuedOpCode and jqueue._QueuedJob are provided for...
Add job priority constants
locking: Implement priorities in SharedLock and LockSet
For proper support of job priorities, jobs' locks need to respectpriorities. Otherwise it could happen that a job with a lower prioritycould get a lock before a job with a higher priority (depending on...
The config now should also belong to confd group and readable by it
jqueue: Use separate function for encoding errors
Comes with unittest.
Adding a runtime configuration library
This is used to expand the users/group names just once atinitial call.
Show list of pending acquires in “gnt-debug locks”
This is accomplished by keeping a list of waiting threads insteadof just their number inside the lock-internal condition. A fewother tweaks to the output format are also made.
hansmi helped me with merging the conflict. Thanks
Conflicts: lib/workerpool.py
Adding a paramiko fingerprint format helper
And provide unittests for them
workerpool: Add support for task priority
To add job priorities, the worker pool underlying the job queue mustsupport priorities per task. This patch adds them to the worker pool.
Fix race condition in locking unittest
While writing unittests for the new lock monitor, I made a small mistake anddidn't synchronize acquiring locks properly. This patch takes care of thisissue by adding additional synchronization.
Add simple lock monitor
This patch adds an initial implementation of a lock monitor, accessiblefor the user through “gnt-debug locks”. It currently shows all resourcelocks: BGL, nodes and instances. Config and job queue locks could beshown too, but wouldn't be of much help. The current owner(s) and mode...
workerpool: Allow setting task name
With this patch, the task name is added to the thread name and will show up inlogs. Log messages from jobs will look like “pid=578/JobQueue14/Job13 mcpu:289DEBUG LU locks acquired/cluster/BGL/shared”.
Use one function to parse “--fields” option values
Make family argument in FormatAddress optional
By doing this we delegate the task of finding the correct address familyto the FormatAddress method.
Signed-off-by: Manuel Franceschini <livewire@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Support IPv6 node add