Add unittest for utils._FingerprintFile
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Merge branch 'stable-2.1' into devel-2.1
Add unittests for cli.GenerateTable
Fix long-standing race condition bug in locking unittest
Every once in a while we saw failures in the locking unittest, butweren't able to determine the cause. This patch fixes it by usingthreading.Event instead of plain threading.Condition to notify another...
testutils: Print name of test program before running it
While the name can be looked up in Makefile.am, thisis useful when an automated test just hangs and needsto be killed.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Add an UnescapeAndSplit function
In many cases, where we accept (usually from the command line) a list ofparameters, we remove the use of the separator as an component of any ofthe elements.
This patch adds a new function that can split strings of the form...
workerpool: Make worker ID alphanumeric
Having a proper name instead of just a number makes debuggingeasier.
gnt-cluster verify: Warn if node time diverges too far
The warning will be generated if the clocks diverge by morethan 150 seconds. Due to the way the RPC system works, wecannot get exact time differences, e.g. if one of thequeried nodes is broken. The comparision is done using a...
Remove testJsonIndent unittest
It can't work on older distributions where simplejsondoesn't have indentation support.
Revert "Get rid of utils.CommaJoin"
This reverts commit 6915bc28fe053e92aa16cf2d974d205f1140219c based on thread onganeti-devel.
Conflicts:
lib/cmdlib.py (due to the error code classification, trivial)
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Handle EEXIST in utils.RenameFile
This should fix an issue I've seen exactly once during testing. It might havebeen caused by parallel RPC calls to archive jobs.
[…] ganeti-noded:112 ERROR Error in RPC call […] File "/usr/lib/python2.4/site-packages/ganeti/backend.py", line 2365, in JobQueueRename...
config.Add{Node,Instance}: get the ec id
This is ok because adding a node or instance cannot happen in a query.
We get the ec id from the LU and pass it to _EnsureUUID, which willthen for now not use it.
Signed-off-by: Guido Trotter <ultrotter@google.com>...
Cache JSON encoders and sort keys
The sort_keys argument is supported since simplejson 1.3.
Ignore log messages in unittests
http.auth: Add new function to verify passwords
This new function supports two schemes for passwords:- Old-style cleartext passwords- Hashed passwords according to RFC2617 (H(A1))
Schemes are differentiated by their prefix, a concept alsoused in OpenLDAP. Cleartext passwords can no longer start...
Provide feedback from redistributing configuration
This is particularily useful for “gnt-cluster redist-conf”, butalso for all other cases where the configuration files arerewritten on other nodes.
$ gnt-cluster redist-conf… Copy of file /var/lib/ganeti/config.data to node … failed: Error while...
Fix serializer unittests
Commit d22b29997cd broke the serializer unittests with certainversions of simplejson. This patch removes sort_keys againand implements a slightly more efficient way of detectingsimplejson functionality. The serializer unittests no longer...
mcpu: Use new timeout class for timeout
Test LockSet.acquire return value for timeout
mcpu: Change lock attempt timeout calculation
With this patch all timeouts are pre-calculated. The interface ofthe _LockTimeoutStrategy class is also changed a bit; NextAttemptnow returns a new instance.
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
mcpu: Implement lock timeouts
The timeout is always between ~0.1 and ~10.0 seconds. A smallvariation of ±5% is added to prevent different jobs fromfighting each other. After 10 attempts to acquire the locks witha timeout, a blocking acquire is made.
Lock status reporting will be improved in a separate patch....
locking.LockSet: Implement acquire timeouts
The timeout passed to LockSet.acquire() is measured over all lock acquires. IfLockSet.acquire fails to acquire all requested locks within the specifiedamount of time, all locks are released again and the acquire fails....
Get rid of utils.CommaJoin
- We never remember to use it (5 uses vs 21 " ,".join())- It's longer to write than " ,".join()- The added value of the apostrophe in the string is not very much
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Add case_sensitive keyword to MatchNameComponent
Now featuring unit testing, and more deterministic results on somecorner cases.
Replace all xrange() with range()
More locking tests race conditions fixes
There were more race conditions. By adding a notify function toSharedLock.acquire we can prevent them.
SingleActionPipeCondition =~ s/Action/Notify/
With this patch we simplify usage on the SingleActionCondition (whichwasn't a condition at all) by making it a real condition. This way wecan just wait() on it, or notifyAll() as we would on a normal one. The...
testNotification: add more checking about order
Abstract base condition test cases
This way they can be used to test different condition classes.
Move the "done" queue inside _ThreadedTestCase
All (ok, all but one) _ThreadedTestCase users have a done Queue, so wemove its building in the _ThreadedTestCase setUp
locking.SharedLock: Fix bug in delete function
SharedLock.__acquire_unlocked uses keyword parameters. Just passingthe timeout would set the “shared” parameter.
Rename LockSet.acquire parameter “blocking” to “timeout”
Also remove the “blocking” parameter from LockSet.remove andGanetiLockManager.remove. There's no point in implementing timeouts on removalunless we need them.
Try to fix locking unittests
Our automated test system found a few problems in the new lockingunittests. This patch should fix them, although I wasn't able toreproduce the problem. All are race conditions.
Add _PipeCondition class
_PipeCondition is a condition implemented using pipe(2) and poll(2).It allows the implementation of timeouts without using a busy-wait loopwith time.sleep.
Unlike Python's built-in threading.Condition class and to save filedescriptors and an internal queue, it can only be used to notify...
Add _SingleActionPipeCondition class
This class will be used as a basic block for pipe(2)-basedconditions. Upon initialization it creates a pipe and can benotified once (hence the “single action” in the name). Acallable helper class is used to wait for notifications....
SharedLock: implement timeouts
This patch greatly simplifies the SharedLock code and implementstimeouts for the acquire() and delete() functions. A wrapper aroundPython's threading.Condition class must be used to ensure threadsafety when check whether there are any waiters left....
Merge remote branch 'origin/master' into mogu
Add initial confd client unittests
Some basic tests for the confd client library
Merge commit 'origin/next'
Fix utils.MatchNameComponent for full matches
While ‘test1’ matches both ‘test1’ and ‘test1.example’, it has a full,exact match and we should return it if that is the case.
Add unittests for confd constants
Checking that the fourcc code is indeed a fourcc code, and that requestcodes and reply statuses are distinct.
Wrap lines over 80 characters
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Luca Bigliardi <shammash@google.com>
Move LoadModule function to ganeti.build
It should only be used at build-time.
Add simple unittest for manpages
Add simple unittest for remote API docs
Add simple unittest for hooks documentation
Run unittests in a temporary directory
Remove extra argument from HooksMaster class
The mcpu.py:HooksMaster class needs to have a proc attribute/argument toinit in ordet to call its LogWarning method. However, this is availablefrom the 'lu' attribute, so we can remove this dependency.
Signed-off-by: Iustin Pop <iustin@google.com>...
Fix /proc/drbd parsing in presence of gaps
In case there are gaps in /proc/drbd related to the minors sequence,this can lead to empty lines (see the attached data file). In this case,we currently fail to parse the file correctly.
The patch fixes this by skipping empty lines in...
Add utils.FormatTime and a simple unittest
We don't format with subsecond-precision.
Serializer, remove salt_verifier functionality
The salt needs to be returned anyway, so we don't have to add anotherkey for the sender to recognize which request an answer is answering, soall that infrastructure is useless. :(
Add constants for local disk status
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
TestParameterNames: also check nic parameters
ConfigObject.ToDict() only export non-None values
The method is changed to a normal loop, to avoid calling getattr()twice. Also getstate is changed to just use ToDict() by default.
This should also make getstate work for objects which have tooverride the ToDict function because they contain other objects....
Fix unittests broken by commit 2bb5c9115f
File "../test/ganeti.hooks_unittest.py", line 239, in setUp self.lu = FakeLU(FakeProc(), self.op, self.context, None)File "…/ganeti/cmdlib.py", line 92, in init self.LogStep = processor.LogStepAttributeError: FakeProc instance has no attribute 'LogStep'...
Add enabled hypervisors to TestConfigRunner
This parameter is now mandatory for the cluster config to work.
Get rid of the default_hypervisor slot
Currently we have both a default_hypervisor and an enabled_hypervisorslist. The former is only settable at cluster init time, while the lattercan be changed with cluster modify.
This becomes cumbersome in a few ways: at cluster init time for example...
HMAC authenticated json messages
This patch includes HMAC authenticated json messages to the serializer.The new interface works on any json-encodable data type, and can sign itwith a private key and an optional salt. The same private key must beused upon message loading to verify the message....
check_ident_key_val, handle no_ and - prefixes
If an ident member of an IdentKeyVal relationship starts with no_ or -,handle it the same way we do for a key. Some unittests are added tocheck that check_ident_key_val behaves as expected.
This patch also changes ForceDictType to, for now, fail on such an...
_SplitKeyVal with no data return an empty dict
If an empty string is passed to _SplitKeyVal, we should return {},rather than {'': True}. Also test for the correct behavior.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Merge branch 'next' into branch-2.1
Fix hooks unittests after RPC result changes
Patch "Simplify the RPC result framework in backend.py" changed allRPCs, and thus the hooks results where also changed. This needs changesto the hooks unittests too.
Convert hooks_runner rpc to new style result
This also converts (and fixes) unittests and mock objects to deal withthis change, and the custom hook verifier in cmdlib.LUClusterVerify.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Remove utils.CheckDict since it's no longer used
Only the custom handling in rpc.py for call_node_info used this, so nowit can be removed.
config unittests: use nic params
The target test cluster should have nic params, rather than a defaultbridge.
Add NIC.CheckParameterSyntax
This function will be used to check the NIC parameters for validity.Unittests are included.
IsNormAbsPath and users, use "normalized" term
We used to refer to normalized paths as "normal" which might beconfusing. This fixes the syntax in all current IsNormAbsPath users andin the docstring.
Add utils.IsNormAbsPath function
Currently most of the time we check for absolute path, but that doesn'tprotect us from some invalid paths. In some places we should be morestrict, and this function should help us to.
Export more instance information in hooks
Currently we miss in hooks the instance's hypervisor, hypervisorparameters and backend parameters. This forces hooks to query back intoganeti, which is dangerous due to possible luxi sockets exhaustion.
This patch adds these three as INSTANCE_HYPERVISOR, INSTANCE_HV_*,...
Fix the SafeEncoding behaviour
Currently we have bad behaviour in SafeEncode: - binary strings are actually not handled correctly (ahem) - the encoding is not stable, due to use of string_escape
For this reason, we replace the use of string_escape with part of the...
Fix compatibility with DRBD 8.3
DRBD 8.3 changes two more things compared to 8.2: - /proc/drbd format changed in multiple ways; the part we're interested is the ‘st:’ to ‘ro:‘ change (in the changelog named as “Renamed 'state' to 'role'” - “drbdsetup /dev/drbdN show” changed the ‘device’ stanza from:...
Convert IOErrors for /proc/drbd into our errors
If /proc/drbd can't be opened, this raises an IOError, but all theerror-handling behaviour in backend treats only BlockDeviceErrors. Thiscreates a plain failure in cluster verify and in other RPC calls.
This patch simply converts EnvironmentErrors into BlockDeviceErrors, and...
Apply the right permissions to /etc/hosts
In the current Ganeti version when modifying /etc/hosts we mistakenlygive it the permissions of the temporary file we create to define itscontent, which is by default 0600. This breaks most non-rootapplications, and thus must be corrected. This patch forces the mode to...
Instance parameters: force typing
We want all the hv/be parameters to have a known type, rather than arandom mix of empty string, boolean values, and None, so we declare thetype of each variable and we enforce/convert it.
- Add some new constants for enforceable value types...
Fix unittest encoding breakage
Due to the fact that we sanitize now the output from environmentscripts, the unittest needs to be adjusted. My bad for not checking it.
Reviewed-by: imsnah
RAPI: rlib1 removal
The resources we still need moved to rlib2.
Reviewed-by: iustinp
RAPI: Implement /2 resource
RAPI: Deprecate version Rapi version1
It is impossible to keep backward compatibility due tosignificant changes in the Ganeti core.
Introduce more configuration consistency checks
This patch enhances the duplicate DRBD minors checks (currently just afew) and adds automatic checks of configuration consistency atconfiguration file writing time.
In order to do so and show meaningful error messages, the...
DRBD: check for in-use minor during Create
In order to prevent errors with old, in-use DRBD minors, we check andabort at create time if our minor is already in use. For this we need toalso modify DRBD8Status to be able to parse cs:Unconfigured devices....
Add a TailFile function
This patch adds a tail file function, to be used for parsing and returning inthe job log OS installation failures.
Reviewed-by: ultrotter
Unify some unittest functions
This patch adds unified temporary file handling to thetestutils.GanetiTestCase class, which adds easy creation and automatedcleanup of temporary files.
The patch allows a simpler handling in a couple of test cases butrequires all child classes to call the parent setUp and tearDown...
ganeti.http: Add support for private data in HTTP requests
Reviewed-by: amishchenko
Add rename function automatically creating directories if needed
Unfortunately, os.makedirs in Python 2.4 is not safe against multipleprocesses creating the same directory tree at the same time. This isonly fixed in Python 2.5 and up. Adding more checks in our code doesn't...
Add simple unittests for ganeti.http
More complex unittests will need some refactoring in the HTTP code.
Rename all HTTP classes to camel case
It should be consistent.
ganeti.http: Remove ApacheLogfile class
We don't need it anymore and it wouldn't work as it is, anyway.
Fix hooks_unittest with new rpc call structure
RAPI:Fix root list and unittest for it.
Enable auto-unit formatting in script output
This patch enables by default the old 'human-readable' option, but in aslightly different model.
The option is now called "units" and takes either: - 'h' for automatic formatting - 'm', 'g' or 't' for mebi/gibi/tebibytes...
Move FieldSet class to utils.py
Since we can use the FieldSet class in cli.py to nicely format disksizes and such, we move it to utils.py and also move its associatedunittest. I didn't remove the cmdlib.py unittest file as that's not thegood direction :)...
Skip another testWaitingXblocksY
The waiting sharer blocks exclusive is another not-possible to testright now.
Sorry for missing it the first time; there are no other testWaiting...tests.
Add a FieldSet class for variable parameter sets
This patch adds a _FieldSet class that can be used for the new variableparameter sets: e.g. the sda_size will change to disk/0.size (orsimilar) and we need to both check validity and extract the index of the...
Skip the waitingExclusiveBlockSharer test
Until we have a way to properly test this, we'll skip the test.
Convert the locking unittests to repetition-test
Currently the locking tests are using timeouts to ensure an event will'never happen'. However, this is suboptimal.
The patch converts all of these to instead: not wait, but sequence theoperations logically and expect that they execute as such. In case of...
Remove zombie special case from IsProcessAlive
Based on the discussion on the mailing list, we remove the specialcasing of zombie processes.
Note we don't use kill -0 since that has a different meaning than "checkprocess is alive", so this function is still linux-specific....
Add cleanup of processes to utils.KillProcess
In case the process we want to signal is our own process, and the signalwe send is a deadly one, we should also cleanup after the process.
This patch adds a new parameter waitpid to this function that does this,...
Implement working directory for utils.RunCmd
In order to replace usage of the shell just for 'cd %s; ...' we need tobe able to specify the working directory for childs.
The patch also changes the default working directory (when notspecified) to "/", as opposed to the current dir; since the daemos run...
Modify utils.RunCmd to write output to file
Currently we launch processes via the shell in a few places only toredirect standard output and error to a log file ("&> $file"). It isbetter to do such redirection from within RunCmd itself.
This patch splits RunCmd in two parts, the setup and the execution part,...