Improve import/export daemon tests
- Progress messages- Log output on errors- Listen timeout tests
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Retry connection in import-export daemon
Until now, exactly one attempt was made to establish a connection.If it failed, the whole import/export for the disk in questionaborted. Retrying will make it more reliable.
Unfortunately the listening part can't be made completely...
Merge branch 'devel-2.1'
Conflicts: lib/utils.py: Trivial
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
masterd: Log PID, UID and GID of connected client
This can be very useful if client programs run as non-root.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
cmdlib: Add utility for instance data import/export
Interpreting the backend's import/export daemon status is a bit tricky.This utility code keeps track of multiple transfers at the same time.Users can supply callback functions to react to events.
Timeouts are currently hardcoded. Intra-cluster instance moves will likely...
Add a basic unittest for uidpool.RequestUnusedUid
Signed-off-by: Balazs Lecz <leczb@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Add separate module for backported language functionality
utils.py, where they were before, is already huge.
utils: Add function for partial application of function arguments
The function's code was mostly copied from Python's documentationand it's equivalent to “functools.partial” in Python 2.5 and above.
utils: Add function to read locked PID file
This is useful in combination with utils.StartDaemon and will be used forreading the import/export daemon's PID file.
Fix RAPI client methods return values
Ensure that all RAPI client methods that should return job ids actuallyreturn job ids.
Signed-off-by: David Knowles <dknowles@google.com>Reviewed-by: Iustin Pop <iustin@google.com>Signed-off-by: Iustin Pop <iustin@google.com>
Conflicts: doc/security.rst trivial lib/cli.py trivial
Signed-off-by: Balazs Lecz <leczb@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
uidpool: test the separator= argument
Also, fix the string quoting style.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Balazs Lecz <leczb@google.com>
Add daemon for instance import and export
This backend daemon for instance import and export will be used totransfer instance data to other machines. It is implemented in a genericway to support different ways of data input and output. The third-partyprogram “socat”, which is already used by the KVM hypervisor abstraction,...
Add basic unittests for uid_pool
Signed-off-by: Balazs Lecz <leczb@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
utils: Add class to split string stream into lines
This will be used by the new import/export daemon to splitoutput by its child processes into lines.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Merge remote branch 'devel-2.1'
objects.Cluster: add method to get hv defaults
Currently the FillHV method is the one that does the cluster hvparams +os hvparams merger. However, in some cases we need to do just this,without adding the instance hvparams on top.
This patch adds a function to compute just this (hv + os hvp...
Rename the confd_client unittest (to confd.client)
This is to keep same naming across all tests (modules separate with dot,followed by _unittest.py).
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
ConfdClient.SendRequest: allow max coverage
This patch changes the coverage parameter to allow specification of maxcoverage (via -1), versus auto-computation (default, 0) and manualspecification.
Unittests are updated for this case too.
Signed-off-by: Iustin Pop <iustin@google.com>...
Fix utils.WaitForFdCondition inner retry loop
Commit dfdc4060 added WaitForFdCondition which uses utils.Retry withouthandling timeout exceptions. This breaks any nested retry loops.
This patch fixes the above function, and also changes utils.Retry todetect and warn future similar cases. In addition, we add a few small...
utils: Move wrapper code around os.makedirs into separate function
Fix unittest for the rapi client library
Wrong escape, so we make sure to use proper escapes (we want thebackslashes to be embedded, not interpreted). Also change " to ' to beeasier to read.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>...
Adding RAPI client library.
Signed-off-by: David Knowles <dknowles@google.com>Reviewed-by: Iustin Pop <iustin@google.com>Signed-off-by: Iustin Pop <iustin@google.com>(modified slightly the unittest to account for missing httplib2 library)
Remove unused import from test file
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Add RPC calls to create and remove X509 certificates
Certificates and keys generated using these functions will be used forinter-cluster instance moves. As per design, the private key should neverleave the node.
utils: Add functions to sign and verify X509 certs using HMAC
Certificates exchanged via an untrusted third party should besigned to ensure they haven't been modified.
Merge remote branch 'origin/devel-2.1'
utils.RunCmd: Test case with reset_env set and setting variables
utils.CreateBackup: Use human-readable instead of seconds since Epoch
Seconds since the Epoch are not easily readable by a human. Using aformatted timestamp makes it easier (e.g.“….backup-2010-03-12_14_02_43.…”). This patch also makes OS logfiles usethis formatted timestamp....
Add unittest for daemon-util
This test doesn't cover everything, but it's better than nothing.
Improve ganeti.serializer unittests
Add unittests for ganeti.errors
Verify cluster certificates in LUVerifyCluster
When using pyOpenSSL 0.7 or above, LUClusterVerify will start to show awarning 30 days before a certificate expires. 7 days before thecertificate expires, the warning becomes an error. Once expired,LUVerifyCluster will always report an error. The latter is also supported...
utils: Add function to extract X509 cert validity
X509 uses ASN1 GENERALIZEDTIME or UTCTIME to store certificate validity.pyOpenSSL 0.7 and above allow us to retrieve both “notBefore” and“notAfter” as strings. Parsing them turned out to be a challenge since...
Adding unittests for objects.Cluster.FillHV
This adds tests for the stacking of objects.Cluster.FillHV to verifythat the override is working as expected.
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Provide unittests for http.auth
To simplify writing unittests, one data structure class in http.server isalso changed. According to the coverage utility, this provides 95%coverage.
http.auth: Fix bug with checking hashed passwords
When username and password were sent for a resource not requiringauthentication, it wouldn't be accepted if the user in question had ahashed password. The reason was that the function GetAuthRealm used to...
Support passing in file object in utils.FileLock
This way we can re-use file objects opened in other places. Also add moreunittests.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Support arguments in utils.RunInSeparateProcess
Add a function to validate and normalize hostnames
This differs slightly from the specification, by allowing names to startwith digits, not checking the length of individual components, andallowing underscores.
ListVisibleFiles: require normalized path names
This patch changes ListVisibleFiles to raise ProgrammerError if it'spassed a non-absolute/non-normalized path name, and adds unittests forthis behaviour.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
utils: Add a PathJoin function
This will replace os.path.join since it is not safe for directorytraversal issues.
Add multi-key support to the serializer
Add unittests for ganeti.opcodes
According to “coverage”, this covers 99% of the code.
Implement utils.RunParts and use it for hooks
This function is a generic pythonic version of runparts. We currentlyuse it in the backend HooksRunner, but we'll use it for runningdifferent directories as well.
Signed-off-by: Guido Trotter <ultrotter@google.com>...
Add reset_env option to RunCmd
This allows to run a command with only the passed in environment, ratherthan just updating the default one with it.
Now with unit testing.
Add utility function to start daemon
The currently available functions in this direction (utils.RunCmd andutils.Daemonize) both can not be used to start an external daemonprocess. This function takes many steps to ensure the daemon is startedproperly and does careful error checking. Unittests are included....
Use OpenSSL module instead of binary to generate certs
This saves us one dependency and saves us from complicated handling ofexternal files if we need key and certificate separated from each other.
At the same time, the number of bits used for RSA keys is increased from...
utils: Add function to set O_NONBLOCK on file descriptor
utils: Add function to set FD_CLOEXEC on file descriptor
This will be used by another patch adding a utility function to start a daemon.
Move RunInSeparateProcess to ganeti.utils
This function could be useful in other places and thisway we can easily unittest it.
Add function to reset tempfile module after fork
On fork, the tempfile module's pseudo random generator isnot reset. If several processes (e.g. two children or parentand child) try to create a temporary file, they'll conflict.This function can be used to reset the name generator which...
Add unittest for utils._FingerprintFile
Move function generating SSL certs into utils
Also add unittest.
Merge branch 'stable-2.1' into devel-2.1
Factorize LUXI parsing and handling code
Also fix a typo in http/__init__.py and add unittestsfor the LUXI parsing and formatting functions.
Add unittests for cli.GenerateTable
Fix long-standing race condition bug in locking unittest
Every once in a while we saw failures in the locking unittest, butweren't able to determine the cause. This patch fixes it by usingthreading.Event instead of plain threading.Condition to notify another...
testutils: Print name of test program before running it
While the name can be looked up in Makefile.am, thisis useful when an automated test just hangs and needsto be killed.
Add an UnescapeAndSplit function
In many cases, where we accept (usually from the command line) a list ofparameters, we remove the use of the separator as an component of any ofthe elements.
This patch adds a new function that can split strings of the form...
workerpool: Make worker ID alphanumeric
Having a proper name instead of just a number makes debuggingeasier.
gnt-cluster verify: Warn if node time diverges too far
The warning will be generated if the clocks diverge by morethan 150 seconds. Due to the way the RPC system works, wecannot get exact time differences, e.g. if one of thequeried nodes is broken. The comparision is done using a...
Remove testJsonIndent unittest
It can't work on older distributions where simplejsondoesn't have indentation support.
Revert "Get rid of utils.CommaJoin"
This reverts commit 6915bc28fe053e92aa16cf2d974d205f1140219c based on thread onganeti-devel.
Conflicts:
lib/cmdlib.py (due to the error code classification, trivial)
Handle EEXIST in utils.RenameFile
This should fix an issue I've seen exactly once during testing. It might havebeen caused by parallel RPC calls to archive jobs.
[…] ganeti-noded:112 ERROR Error in RPC call […] File "/usr/lib/python2.4/site-packages/ganeti/backend.py", line 2365, in JobQueueRename...
config.Add{Node,Instance}: get the ec id
This is ok because adding a node or instance cannot happen in a query.
We get the ec id from the LU and pass it to _EnsureUUID, which willthen for now not use it.
Cache JSON encoders and sort keys
The sort_keys argument is supported since simplejson 1.3.
Ignore log messages in unittests
http.auth: Add new function to verify passwords
This new function supports two schemes for passwords:- Old-style cleartext passwords- Hashed passwords according to RFC2617 (H(A1))
Schemes are differentiated by their prefix, a concept alsoused in OpenLDAP. Cleartext passwords can no longer start...
Provide feedback from redistributing configuration
This is particularily useful for “gnt-cluster redist-conf”, butalso for all other cases where the configuration files arerewritten on other nodes.
$ gnt-cluster redist-conf… Copy of file /var/lib/ganeti/config.data to node … failed: Error while...
Fix serializer unittests
Commit d22b29997cd broke the serializer unittests with certainversions of simplejson. This patch removes sort_keys againand implements a slightly more efficient way of detectingsimplejson functionality. The serializer unittests no longer...
mcpu: Use new timeout class for timeout
Test LockSet.acquire return value for timeout
mcpu: Change lock attempt timeout calculation
With this patch all timeouts are pre-calculated. The interface ofthe _LockTimeoutStrategy class is also changed a bit; NextAttemptnow returns a new instance.
mcpu: Implement lock timeouts
The timeout is always between ~0.1 and ~10.0 seconds. A smallvariation of ±5% is added to prevent different jobs fromfighting each other. After 10 attempts to acquire the locks witha timeout, a blocking acquire is made.
Lock status reporting will be improved in a separate patch....
locking.LockSet: Implement acquire timeouts
The timeout passed to LockSet.acquire() is measured over all lock acquires. IfLockSet.acquire fails to acquire all requested locks within the specifiedamount of time, all locks are released again and the acquire fails....
Get rid of utils.CommaJoin
- We never remember to use it (5 uses vs 21 " ,".join())- It's longer to write than " ,".join()- The added value of the apostrophe in the string is not very much
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Add case_sensitive keyword to MatchNameComponent
Now featuring unit testing, and more deterministic results on somecorner cases.
Replace all xrange() with range()
More locking tests race conditions fixes
There were more race conditions. By adding a notify function toSharedLock.acquire we can prevent them.
SingleActionPipeCondition =~ s/Action/Notify/
With this patch we simplify usage on the SingleActionCondition (whichwasn't a condition at all) by making it a real condition. This way wecan just wait() on it, or notifyAll() as we would on a normal one. The...
testNotification: add more checking about order
Abstract base condition test cases
This way they can be used to test different condition classes.
Move the "done" queue inside _ThreadedTestCase
All (ok, all but one) _ThreadedTestCase users have a done Queue, so wemove its building in the _ThreadedTestCase setUp
locking.SharedLock: Fix bug in delete function
SharedLock.__acquire_unlocked uses keyword parameters. Just passingthe timeout would set the “shared” parameter.
Rename LockSet.acquire parameter “blocking” to “timeout”
Also remove the “blocking” parameter from LockSet.remove andGanetiLockManager.remove. There's no point in implementing timeouts on removalunless we need them.
Try to fix locking unittests
Our automated test system found a few problems in the new lockingunittests. This patch should fix them, although I wasn't able toreproduce the problem. All are race conditions.
Add _PipeCondition class
_PipeCondition is a condition implemented using pipe(2) and poll(2).It allows the implementation of timeouts without using a busy-wait loopwith time.sleep.
Unlike Python's built-in threading.Condition class and to save filedescriptors and an internal queue, it can only be used to notify...
Add _SingleActionPipeCondition class
This class will be used as a basic block for pipe(2)-basedconditions. Upon initialization it creates a pipe and can benotified once (hence the “single action” in the name). Acallable helper class is used to wait for notifications....