Fix exception re-raising in Python Luxi clients
Commit e687ec01 (present in 2.5 since the 2.5 beta 3) did consistencyfixes across the code-base. Unfortunately this was done without enoughchecks on the actual meaning of one of the fixes, which means error...
Merge branch 'devel-2.4' into stable-2.5
KVM: support version reported by 1.0
This of course was working for all the rcs, but broke with 1.0 itself.
In addition: - split between running kvm --version and parsing its output - unittest parsing for various known --help outputs - updated NEWS file...
jqueue: Fix deadlock between job queue and dependency manager
When an opcode is about to be processed its dependencies areevaluated using “_JobDependencyManager.CheckAndRegister”. Dueto its nature that function requires a lock on the manager'sinternal structures. All of this happens while the job queue...
Add UnescapeAndSplit unittest for multi-escapes
This would have caught the bug in the first place. Argh,hand-generated test cases!
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
LUGroupAssignNodes: Fix node membership corruption
Note: This bug only manifests itself in Ganeti 2.5, but since theproblematic code also exists in 2.4, I decided to fix it there.
If a node was assigned to a new group using “gnt-group assign-nodes” the...
Fail if node/group evacuation can't evacuate instances
If an instance can't be evacuated, only a message would be printed. Withthis change the operation always aborts. Newly added unittests check forthis behaviour.
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
RAPI: Make node evacuation actually work
Commit e1f23243 changed te LU and opcode for node evacuation to receivea “mode” parameter (among other things). Commit de40437a changed theRAPI code accordingly, but did so for an earlier version of the firstpatch. Obviously this couldn't work, so here's the fix....
RAPI: Fix resource for replacing disks
Commit d1c172deb4f inadvertently changes the“/2/instances/[instance_name]/replace-disks” resource to use bodyparameters. There were no QA tests and the issue wasn't noticed.
This patch re-introduces support for query parameters and adds a QA...
Fix issue when verifying cluster files
If a cluster has any non-master-candidate nodes, those don't contain allfiles (e.g. config.data). With commit aef59ae764dc (March 31st, 2011)the logic was changed and subsequently verifying a cluster with non-mcnodes would complain....
LUClusterVerifyGroup: Spread SSH checks over more nodes
When verifying a group the code would always check SSH to all nodes inthe same group, as well as the first node for every other group. On bigclusters this can cause issues since many nodes will try to connect to...
Merge branch 'devel-2.4' into devel-2.5
Conflicts: NEWS (trivial) configure.ac (trivial) daemons/ensure-dirs.in (deleted)
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
utils: Fix UnescapeAndSplit parsing bug
If a value passed to UnescapeAndSplit ended with a backslash anexception would be raised:
$ gnt-instance modify -H mem=x\\ inst1.example.com[…] e2 = slist.pop(0)IndexError: pop from empty list
Adding missing test data for commit 7a380ddfc
Fix a parsing issue with DRBD 8.3.11 in the Linux Kernel
In the Linux kernel commit 4b0715f096 introduced a display bug into/proc/drbd which broke our regex.
The bug was first introduced into Linux 2.6.39-rc1. This bug is stillunfixed as of today.
This patch adapt the regular expression to workaround this bug for the...
ensure-dirs: Check mode and owner before changing
This avoids many calls to chmod(2) and chown(2), and thereby ctimeupdates.
Since I had to update the unittests anyway I untangled the code a bit,split it into more separate functions and added some more tests....
ensure-dirs: Refine error handling on stat(2)
The “_stat_fn” function is renamed to “_lstat_fn” to reflect itsfunction. The try/except block just wraps calling lstat(2) and nothingelse.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Clarify job ID-related type checks, add unittests
Instead of a rather complicated expression only “JobId” is output. JobID lists (like generated by “SubmitManyJobs”) are limited to two-itemlists. Unittests are added.
Added helper functions in netutils and related constants
Added the following functions to netutils:- IsValidInterface- GetInterfaceIpAddresses- _GetIpAddressesFromIpOutput
Added the following static methods to netutils.IPAddress:- GetAddressFamilyFromVersion...
rlib: Expose node group tags
Commit 1ffd26739d3 added support for tagging node groups. Also add acheck for exposed fields.
Detect globbing patterns as query arguments
Short: this patch enables the use of “gnt-instance list '*.site'”.
Detailed description: This patch changes the command line interface codeto try to deduce the kind of filter from the arguments to a “list”command. If it's a list of plain names an old-style name filter is used....
Implement globbing operator for filters
The operators “=*” and “!*” do globbing in filters, e.g.:
$ gnt-instance list --no-headers -o name 'name =* "*.site"'inst1.site.example.com
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Bump version to 2.5.0~beta1
cleaner: Remove watcher's instance status file after 21 days
utils.ReadFile: Add pre-read callback
This will be used by the watcher to store the file's fstat(2). It mustbe done from the filehandle.
Fix unittest failure after list_owned changes
We just need an object that has a list_owned method.
ganeti-cleaner: Remove old watcher state files
Watcher state files can stay around if node groups are removed. Withthis patch they're removed after 21 days.
Add primary/second nodes' group as query fields
These will be very useful for ganeti-watcher as it needs to retrieveinstances by group.
Add ht-based result checks to opcodes
This adds the infrastructure necessary to check opcode results usinght-based functions. Checks are added for two opcodes.
Merge branch 'devel-2.4'
Reopen log file only once after SIGHUP
Commit b6fa9a44 added a re-openable log handler. The log file isreopened when a daemon is sent a HUP signal. Due to a bug in the code,fixed by this patch, the log file would be reopened for every single logmessage thereafter....
Implement instance failover via RAPI
No idea why this was missed before.
Export job dependencies through lock monitor
This makes them visible to the user. Example:
$ gnt-debug locks -o name,pendingName Pendingjob/890 job:891,892job/892 job:894
Make lock monitor more versatile
With this change it'll be possible to register other lock informationproviders. One usecase for this are job dependencies, which can be shownin the output of “gnt-debug locks”, too.
The lock monitor is changed to accept more than one return value from...
Rename *_STATUS_WAITLOCK to …_WAITING
This patch renames the {JOB,OP}_STATUS_WAITLOCK constants to {JOB,OP}_STATUS_WAITING, as per design document for chained jobs.
Fix locking issue with job dependencies
When jobs waiting for a dependency are notified, they're re-added to thequeue. This would require owning the queue lock in exclusive mode, butsince the function doing so is called from within the job/opcodeprocessor, it only holds the lock in shared mode....
jqueue: Implement submitting multiple jobs with dependencies
With this change users of the “SubmitManyJobs” interface can userelative job dependencies. Relative job IDs in dependencies are resolvedbefore handing the job off to the workerpool.
jqueue: Add “writable” flag to memory objects
Basically only one instance of the job, the one being processed,should be serialized to disk and replicated to other nodes. Withthis flag assertions can be added in various places.
Implement chained jobs
An overview is available in the design document for this change,doc/design-chained-jobs.rst.
When a job enters the job processor, the current opcode's dependenciesare evaluated. If a referenced job has not yet reached the desired...
Adding a wrapper around connecting to kvm console
The wrapper will connect to the console, and check in the background ifthe instance is paused, unpausing it as necessary.
Signed-off-by: Stephen Shirley <diamond@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
cli.GetOnlineNodes: Support node group filter, use query2
This patc changes cli.GetOnlineNodes to use query2, which does thefiltering in the master daemon, and adds a new parameter to filter bynode group.
Unittests were added for the old implementation and then adopted to...
ht: Add new check for numbers
Places which receive floats can usually also deal with integers, e.g.OpTestDelay. Tests are added and the new check function is used for theaforementioned opcode and verifying query results.
Change RAPI for new node evacuation opcode
The change is not backwards compatible, see the updated NEWS file.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Adding basic abstraction layer for caching
This includes an own simple cache implementation and aninterface to a memcache instance.
Fix _checkRsaPrivateKey for newer key generation
Keys generated under debian sid just read "BEGIN PRIVATE KEY" ratherthan "BEGIN RSA PRIVATE KEY".
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Fix incomplete merge
Commit 66bd7445 changed the semantics of _JobProcessor on finishedjobs, and updated the related unittests in the 2.4 branch. It was thenmerged to master, however on master there was an additional test forthis case, which was not updated....
jqueue: Fix potential race condition when cancelling queued jobs
When a job was cancelled, its status would be changed and the filewritten again. Since this was a final status, the job file could bemoved anytime for archival. If the job was still in the queue, however,...
gnt-node migrate: Use LU-generated jobs
Until now LUNodeMigrate used multiple tasklets to evacuate all primaryinstances on a node. In some cases it would acquire all node locks,which isn't good on big clusters. With upcoming improvements to the LUsfor instance failover and migration, switching to separate jobs looks...
ht: Add checks for anything, regexp, job ID, container items
The check for container items is useful for tuples and/or lists withnon-uniform values. The “anything” check can be used when any valueshould be accepted for an item.
The job ID check, which uses the regexp check, will be used for...
GetEntResolver: Make it possible to resolve uid/gid to name
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
utils.algo: Add InvertDict to invert a dict
Improve hooks documentation unittest
Also check for the opcode ID.
Split LUClusterVerify into LUClusterVerify{Config,Group}
With this change, LUClusterVerifyConfig becomes a "light" LU that onlyverifies the global config and other, master-only settings, and the bulk ofnode/instance verification is done by LUClusterVerifyGroup, which only acts...
ht: Add strict check for dictionaries
This allows checking specific dictionary items, unlike TDictor TDictOf.
SharedLock: Implement downgrade from exclusive to shared mode
If a job needs to modify a resource and then wait for a result, it mustacquire the resource lock in exclusive mode. In some cases it would bepossible to only have a shared lock for waiting. Until now it was not...
Re-indent test/mocks.py using two spaces
No idea where those four spaces came from, but they must've been therefor a while.
cmdlib: Remove acquired_locks attribute from LUs
The “acquired_locks” attribute in LUs is used to keep a list of acquiredlocks at each lock level. This information is already known in the lockmanager, which also happens to be the authoritative source. Removing the...
opcodes: Add function for compact summary
Depending on the opcode and its parameters, the existing “Summary”function can give a rater long summary. For displaying the summary inlogs and in the lock monitor, it should be shorter. Hence this newfunction is added to just use the opcode ID with common prefixes...
locking: Make parameter to condition's wait() positional
It is always used in the locking code. Unittests are updated.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
SetEtcHostsEntry: maintain existing ordering
Currently RemoveEtcHostsEntry keeps the ordering, but SetEtcHostsEntrynot, as it will always write the new entry at the end of file. Ipersonally dislike this as it "uglifies" my custom host files, so thispatch makes it update the record instead in-place so to say instead of...
Fix WriteFile with unicode data
Unicode is fun, indeed:
len(buffer("abc"))
3
len(buffer(u"abc"))
12
So we can't pass unicode data to buffer(), as the result will be towrite the in-memory (usually UTF-32) representation to disk.
Signed-off-by: Iustin Pop <iustin@google.com>...
RAPI: Add support for tagging node groups
qlang: Add function to distinguish filters from names
qlang: Add parser for query filter language
With this parser, command line utilities will be able to provide filtersthrough query2 in a simplistic language. Example filters:
name "node3.example.com" master or (name "node4.example.com") be/memory == 128 and name =~ /^web/i...
Add instance query field for OS parameters
These were not available as a query field before. Update unittestsand description text for the other “..params” fields.
utils.WriteFile: Close file before renaming
Issue 154 (http://code.google.com/p/ganeti/issues/detail?id=154)reported an “Operation not supported” error when writing instanceexports to a mounted CIFS filesystem. Experimentation showed the errorto only occur when using rename(2) on an opened file. Various references...
Merge branch 'stable-2.4' into devel-2.4
Increase the lock timeouts before we block-acquire
This has been observed to cause problems on real clusters via thefollowing mechanism:
- a long job (e.g. a replace-disks) is keeping an exclusive lock on an instance- the watcher starts and submits its query instances opcode which...
utils: Add function generating regex for DNS name globbing
The intent of this function is to be able to provide a globbing operatoror query filters. One should be able to say, for example, something tothe effect of “gnt-instance shutdown '*.site'”.
Also rename a variable in MatchNameComponent....
query: Add implementation of regex match operator
So far this operator was not implemented. This patch adds an additionalvalue preparation function to the function table for binary operators,used to compile the regular expression. Unittests are included....
Add a simple wrapper over utils.Retry
The new wrapper makes moving legacy code to utils.Retry or addingretries in existing code simpler.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
test.ganeti.process_unittest: Fix race condition
There was a race condition on heavily loaded testsystem causing randomlyto fail the timeout unittests as the signal handler is not yet setup butthe timeout has already hit.
Therefore we introduce a workaround to wait until a program reached a...
RAPI client: Remove support for version 0 instance creation requests
Implement submitting jobs from logical units
The design details can be seen in the design document(doc/design-lu-generated-jobs.rst).
Rewrite of ensure-dirs in python
I provided unittest to test the important pieces of the infrastructure.The one remaining function (ResuriveEnsure) is not easy to unittestbut also not critical if it fails to operate correctly.
Signed-off-by: René Nussbaumer <rn@google.com>...
RAPI client: Tidy and test WaitForJobCompletion
- Use constants- Don't sleep if no delay is given- Mark function as deprecated: it uses polling instead of waiting for changes (but the latter needs authentication); it can still be used- Add unittests...
RAPI client: Add job status constants
Split BuildHooksEnv of LUs
Commit dd7f677623 added another call to BuildHooksEnv to providepost-phase status variables. Since BuildHooksEnv also built the nodelists, that meant they have to be built twice. First a rather strictcheck was used, but it turned out to be more tricky. Commit b423c51336...
Remove restrictive hook node list check
Commit dd7f67762 added a restrictive check for the node lists returnedby BuildHooksEnv, leading to errors with some LUs, one of which wasfixed in commit 0dfa2c227. As it turns out, other LUs have similarissues, some not easy to fix. This patch disables the restrictive check...
hooks: Provide variables with post-opcode values
When a hook is called, it is provided with a number of variablesdescribing the status of the instance/node/etc. before the operation.Some opcodes provide extra variables to see modified values from hooks,...
locking: Fix race condition in lock monitor
In some rare cases it can happen that a lock is re-created very soonafter deletion, while the old instance hasn't been destructed yet. Insuch a case the code would detect a duplicate name and raise anexception....
RAPI: Add support for querying resources
- Access is only permitted for authenticated clients (queries can return sensitive data)- Filters can be specified when sending a PUT request- Updates RAPI client, documentation and tests
Add support for query resources in RAPI URIs
constants: Rename QR_OP_*, add QR_VIA_RAPI
Commit 28b71a76 added a list of resources which can be queried usingLUXI. Unfortunately the variable was named “QR_OP_LUXI”, which can beconfusing. This patch renames “QR_OP_QUERY” to “QR_VIA_OP”, “QR_OP_LUXI”...
qlang: Remove unused ReadSimpleFilter
utils: Export NiceSortKey function
The ability to split a string into a list of strings and integers can behandy elsewhere and is necessary for sorting query results by names.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>...
Convert OsDiagnose to query
cfgupgrade: Fix critical bug overwriting RAPI users file
The cfgupgrade tool was designed to be idempotent, that means it couldbe run several times and still give produce the correct result. Ganeti2.4 moved the file containing the RAPI users to a separate directory...
query: Add operator for truth
The “?” operator is the equivalent of “if var” in Python.
Fix test rapi request
This was wrong even before, as it contained both a bridge and a link,which are incompatible when passed together. Fix by passing only thelink field.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
query: Fix bug when names are specified
If the client/caller would specify names through the use of a filter,the result would be sorted. This is a regression over earlier Ganetiversions and verified in QA. This patch adds an optional parameter tocontrol the sorting and provides unittests....
query: Add support for filters
This patch adds a “compiler” for query filters, converting them to acallable function used while preparing the query result. In addition, ahints call allows some analysis to be done on the query (e.g. referencednames), making data collection more efficient....
Add query field flags
Some fields (e.g. “name”) should be treated specially when comparing forequality. Hypervisor names should use normal rules, but for node names,“node2” should be equivalent with “node2.example.com”.
To make these differences, a new field for flags is added to the query...
Query: Add additional test to node.powered for OOB handling
This patch covers the testing of _GetNodePower of query.py not only forunavailable but also for OOB support and returning the rightnode.powered.