utils.RunCmd: Test case with reset_env set and setting variables
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
utils.CreateBackup: Use human-readable instead of seconds since Epoch
Seconds since the Epoch are not easily readable by a human. Using aformatted timestamp makes it easier (e.g.“….backup-2010-03-12_14_02_43.…”). This patch also makes OS logfiles usethis formatted timestamp....
Add unittest for daemon-util
This test doesn't cover everything, but it's better than nothing.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Improve ganeti.serializer unittests
Add unittests for ganeti.errors
Verify cluster certificates in LUVerifyCluster
When using pyOpenSSL 0.7 or above, LUClusterVerify will start to show awarning 30 days before a certificate expires. 7 days before thecertificate expires, the warning becomes an error. Once expired,LUVerifyCluster will always report an error. The latter is also supported...
utils: Add function to extract X509 cert validity
X509 uses ASN1 GENERALIZEDTIME or UTCTIME to store certificate validity.pyOpenSSL 0.7 and above allow us to retrieve both “notBefore” and“notAfter” as strings. Parsing them turned out to be a challenge since...
Adding unittests for objects.Cluster.FillHV
This adds tests for the stacking of objects.Cluster.FillHV to verifythat the override is working as expected.
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Provide unittests for http.auth
To simplify writing unittests, one data structure class in http.server isalso changed. According to the coverage utility, this provides 95%coverage.
http.auth: Fix bug with checking hashed passwords
When username and password were sent for a resource not requiringauthentication, it wouldn't be accepted if the user in question had ahashed password. The reason was that the function GetAuthRealm used to...
Support passing in file object in utils.FileLock
This way we can re-use file objects opened in other places. Also add moreunittests.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Support arguments in utils.RunInSeparateProcess
Add a function to validate and normalize hostnames
This differs slightly from the specification, by allowing names to startwith digits, not checking the length of individual components, andallowing underscores.
Signed-off-by: Iustin Pop <iustin@google.com>...
ListVisibleFiles: require normalized path names
This patch changes ListVisibleFiles to raise ProgrammerError if it'spassed a non-absolute/non-normalized path name, and adds unittests forthis behaviour.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
utils: Add a PathJoin function
This will replace os.path.join since it is not safe for directorytraversal issues.
Add multi-key support to the serializer
Signed-off-by: Balazs Lecz <leczb@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Add unittests for ganeti.opcodes
According to “coverage”, this covers 99% of the code.
Implement utils.RunParts and use it for hooks
This function is a generic pythonic version of runparts. We currentlyuse it in the backend HooksRunner, but we'll use it for runningdifferent directories as well.
Signed-off-by: Guido Trotter <ultrotter@google.com>...
Add reset_env option to RunCmd
This allows to run a command with only the passed in environment, ratherthan just updating the default one with it.
Now with unit testing.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Move RunInSeparateProcess to ganeti.utils
This function could be useful in other places and thisway we can easily unittest it.
Add function to reset tempfile module after fork
On fork, the tempfile module's pseudo random generator isnot reset. If several processes (e.g. two children or parentand child) try to create a temporary file, they'll conflict.This function can be used to reset the name generator which...
Add unittest for utils._FingerprintFile
Merge branch 'stable-2.1' into devel-2.1
Add unittests for cli.GenerateTable
Fix long-standing race condition bug in locking unittest
Every once in a while we saw failures in the locking unittest, butweren't able to determine the cause. This patch fixes it by usingthreading.Event instead of plain threading.Condition to notify another...
testutils: Print name of test program before running it
While the name can be looked up in Makefile.am, thisis useful when an automated test just hangs and needsto be killed.
Add an UnescapeAndSplit function
In many cases, where we accept (usually from the command line) a list ofparameters, we remove the use of the separator as an component of any ofthe elements.
This patch adds a new function that can split strings of the form...
workerpool: Make worker ID alphanumeric
Having a proper name instead of just a number makes debuggingeasier.
gnt-cluster verify: Warn if node time diverges too far
The warning will be generated if the clocks diverge by morethan 150 seconds. Due to the way the RPC system works, wecannot get exact time differences, e.g. if one of thequeried nodes is broken. The comparision is done using a...
Remove testJsonIndent unittest
It can't work on older distributions where simplejsondoesn't have indentation support.
Revert "Get rid of utils.CommaJoin"
This reverts commit 6915bc28fe053e92aa16cf2d974d205f1140219c based on thread onganeti-devel.
Conflicts:
lib/cmdlib.py (due to the error code classification, trivial)
Handle EEXIST in utils.RenameFile
This should fix an issue I've seen exactly once during testing. It might havebeen caused by parallel RPC calls to archive jobs.
[…] ganeti-noded:112 ERROR Error in RPC call […] File "/usr/lib/python2.4/site-packages/ganeti/backend.py", line 2365, in JobQueueRename...
config.Add{Node,Instance}: get the ec id
This is ok because adding a node or instance cannot happen in a query.
We get the ec id from the LU and pass it to _EnsureUUID, which willthen for now not use it.
Cache JSON encoders and sort keys
The sort_keys argument is supported since simplejson 1.3.
Ignore log messages in unittests
http.auth: Add new function to verify passwords
This new function supports two schemes for passwords:- Old-style cleartext passwords- Hashed passwords according to RFC2617 (H(A1))
Schemes are differentiated by their prefix, a concept alsoused in OpenLDAP. Cleartext passwords can no longer start...
Provide feedback from redistributing configuration
This is particularily useful for “gnt-cluster redist-conf”, butalso for all other cases where the configuration files arerewritten on other nodes.
$ gnt-cluster redist-conf… Copy of file /var/lib/ganeti/config.data to node … failed: Error while...
Fix serializer unittests
Commit d22b29997cd broke the serializer unittests with certainversions of simplejson. This patch removes sort_keys againand implements a slightly more efficient way of detectingsimplejson functionality. The serializer unittests no longer...
mcpu: Use new timeout class for timeout
Test LockSet.acquire return value for timeout
mcpu: Change lock attempt timeout calculation
With this patch all timeouts are pre-calculated. The interface ofthe _LockTimeoutStrategy class is also changed a bit; NextAttemptnow returns a new instance.
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
mcpu: Implement lock timeouts
The timeout is always between ~0.1 and ~10.0 seconds. A smallvariation of ±5% is added to prevent different jobs fromfighting each other. After 10 attempts to acquire the locks witha timeout, a blocking acquire is made.
Lock status reporting will be improved in a separate patch....
locking.LockSet: Implement acquire timeouts
The timeout passed to LockSet.acquire() is measured over all lock acquires. IfLockSet.acquire fails to acquire all requested locks within the specifiedamount of time, all locks are released again and the acquire fails....
Get rid of utils.CommaJoin
- We never remember to use it (5 uses vs 21 " ,".join())- It's longer to write than " ,".join()- The added value of the apostrophe in the string is not very much
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Add case_sensitive keyword to MatchNameComponent
Now featuring unit testing, and more deterministic results on somecorner cases.
Replace all xrange() with range()
More locking tests race conditions fixes
There were more race conditions. By adding a notify function toSharedLock.acquire we can prevent them.
SingleActionPipeCondition =~ s/Action/Notify/
With this patch we simplify usage on the SingleActionCondition (whichwasn't a condition at all) by making it a real condition. This way wecan just wait() on it, or notifyAll() as we would on a normal one. The...
testNotification: add more checking about order
Abstract base condition test cases
This way they can be used to test different condition classes.
Move the "done" queue inside _ThreadedTestCase
All (ok, all but one) _ThreadedTestCase users have a done Queue, so wemove its building in the _ThreadedTestCase setUp
locking.SharedLock: Fix bug in delete function
SharedLock.__acquire_unlocked uses keyword parameters. Just passingthe timeout would set the “shared” parameter.
Rename LockSet.acquire parameter “blocking” to “timeout”
Also remove the “blocking” parameter from LockSet.remove andGanetiLockManager.remove. There's no point in implementing timeouts on removalunless we need them.
Try to fix locking unittests
Our automated test system found a few problems in the new lockingunittests. This patch should fix them, although I wasn't able toreproduce the problem. All are race conditions.
Add _PipeCondition class
_PipeCondition is a condition implemented using pipe(2) and poll(2).It allows the implementation of timeouts without using a busy-wait loopwith time.sleep.
Unlike Python's built-in threading.Condition class and to save filedescriptors and an internal queue, it can only be used to notify...
Add _SingleActionPipeCondition class
This class will be used as a basic block for pipe(2)-basedconditions. Upon initialization it creates a pipe and can benotified once (hence the “single action” in the name). Acallable helper class is used to wait for notifications....
SharedLock: implement timeouts
This patch greatly simplifies the SharedLock code and implementstimeouts for the acquire() and delete() functions. A wrapper aroundPython's threading.Condition class must be used to ensure threadsafety when check whether there are any waiters left....
Merge remote branch 'origin/master' into mogu
Add initial confd client unittests
Some basic tests for the confd client library
Merge commit 'origin/next'
Fix utils.MatchNameComponent for full matches
While ‘test1’ matches both ‘test1’ and ‘test1.example’, it has a full,exact match and we should return it if that is the case.
Add unittests for confd constants
Checking that the fourcc code is indeed a fourcc code, and that requestcodes and reply statuses are distinct.
Wrap lines over 80 characters
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Luca Bigliardi <shammash@google.com>
Move LoadModule function to ganeti.build
It should only be used at build-time.
Add simple unittest for manpages
Add simple unittest for remote API docs
Add simple unittest for hooks documentation
Run unittests in a temporary directory
Remove extra argument from HooksMaster class
The mcpu.py:HooksMaster class needs to have a proc attribute/argument toinit in ordet to call its LogWarning method. However, this is availablefrom the 'lu' attribute, so we can remove this dependency.
Fix /proc/drbd parsing in presence of gaps
In case there are gaps in /proc/drbd related to the minors sequence,this can lead to empty lines (see the attached data file). In this case,we currently fail to parse the file correctly.
The patch fixes this by skipping empty lines in...
Add utils.FormatTime and a simple unittest
We don't format with subsecond-precision.
Serializer, remove salt_verifier functionality
The salt needs to be returned anyway, so we don't have to add anotherkey for the sender to recognize which request an answer is answering, soall that infrastructure is useless. :(
Add constants for local disk status
TestParameterNames: also check nic parameters
ConfigObject.ToDict() only export non-None values
The method is changed to a normal loop, to avoid calling getattr()twice. Also getstate is changed to just use ToDict() by default.
This should also make getstate work for objects which have tooverride the ToDict function because they contain other objects....
Fix unittests broken by commit 2bb5c9115f
File "../test/ganeti.hooks_unittest.py", line 239, in setUp self.lu = FakeLU(FakeProc(), self.op, self.context, None)File "…/ganeti/cmdlib.py", line 92, in init self.LogStep = processor.LogStepAttributeError: FakeProc instance has no attribute 'LogStep'...
Add enabled hypervisors to TestConfigRunner
This parameter is now mandatory for the cluster config to work.
Get rid of the default_hypervisor slot
Currently we have both a default_hypervisor and an enabled_hypervisorslist. The former is only settable at cluster init time, while the lattercan be changed with cluster modify.
This becomes cumbersome in a few ways: at cluster init time for example...
HMAC authenticated json messages
This patch includes HMAC authenticated json messages to the serializer.The new interface works on any json-encodable data type, and can sign itwith a private key and an optional salt. The same private key must beused upon message loading to verify the message....
check_ident_key_val, handle no_ and - prefixes
If an ident member of an IdentKeyVal relationship starts with no_ or -,handle it the same way we do for a key. Some unittests are added tocheck that check_ident_key_val behaves as expected.
This patch also changes ForceDictType to, for now, fail on such an...
_SplitKeyVal with no data return an empty dict
If an empty string is passed to _SplitKeyVal, we should return {},rather than {'': True}. Also test for the correct behavior.
Merge branch 'next' into branch-2.1
Fix hooks unittests after RPC result changes
Patch "Simplify the RPC result framework in backend.py" changed allRPCs, and thus the hooks results where also changed. This needs changesto the hooks unittests too.
Convert hooks_runner rpc to new style result
This also converts (and fixes) unittests and mock objects to deal withthis change, and the custom hook verifier in cmdlib.LUClusterVerify.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Remove utils.CheckDict since it's no longer used
Only the custom handling in rpc.py for call_node_info used this, so nowit can be removed.
config unittests: use nic params
The target test cluster should have nic params, rather than a defaultbridge.
Add NIC.CheckParameterSyntax
This function will be used to check the NIC parameters for validity.Unittests are included.
IsNormAbsPath and users, use "normalized" term
We used to refer to normalized paths as "normal" which might beconfusing. This fixes the syntax in all current IsNormAbsPath users andin the docstring.
Add utils.IsNormAbsPath function
Currently most of the time we check for absolute path, but that doesn'tprotect us from some invalid paths. In some places we should be morestrict, and this function should help us to.
Export more instance information in hooks
Currently we miss in hooks the instance's hypervisor, hypervisorparameters and backend parameters. This forces hooks to query back intoganeti, which is dangerous due to possible luxi sockets exhaustion.
This patch adds these three as INSTANCE_HYPERVISOR, INSTANCE_HV_*,...
Fix the SafeEncoding behaviour
Currently we have bad behaviour in SafeEncode: - binary strings are actually not handled correctly (ahem) - the encoding is not stable, due to use of string_escape
For this reason, we replace the use of string_escape with part of the...
Fix compatibility with DRBD 8.3
DRBD 8.3 changes two more things compared to 8.2: - /proc/drbd format changed in multiple ways; the part we're interested is the ‘st:’ to ‘ro:‘ change (in the changelog named as “Renamed 'state' to 'role'” - “drbdsetup /dev/drbdN show” changed the ‘device’ stanza from:...
Convert IOErrors for /proc/drbd into our errors
If /proc/drbd can't be opened, this raises an IOError, but all theerror-handling behaviour in backend treats only BlockDeviceErrors. Thiscreates a plain failure in cluster verify and in other RPC calls.
This patch simply converts EnvironmentErrors into BlockDeviceErrors, and...
Apply the right permissions to /etc/hosts
In the current Ganeti version when modifying /etc/hosts we mistakenlygive it the permissions of the temporary file we create to define itscontent, which is by default 0600. This breaks most non-rootapplications, and thus must be corrected. This patch forces the mode to...
Instance parameters: force typing
We want all the hv/be parameters to have a known type, rather than arandom mix of empty string, boolean values, and None, so we declare thetype of each variable and we enforce/convert it.
- Add some new constants for enforceable value types...
Fix unittest encoding breakage
Due to the fact that we sanitize now the output from environmentscripts, the unittest needs to be adjusted. My bad for not checking it.
Reviewed-by: imsnah
RAPI: rlib1 removal
The resources we still need moved to rlib2.
Reviewed-by: iustinp
RAPI: Implement /2 resource
RAPI: Deprecate version Rapi version1
It is impossible to keep backward compatibility due tosignificant changes in the Ganeti core.