reraise exceptions in async tests' error handlers
This makes sure that any unforeseen error raises an exception ratherthen just increasing a counter. It makes unittest debugging a loteasier.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
import/export daemon: Move command building into separate module
The import/export daemon code is already large. Moving some codeto a separate module will make it smaller and easier to test.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Merge branch 'devel-2.1'
Test for errors during inotify callback
- Create a new _MyErrorLoggingAsyncNotifier class which registers error counts, rather than logging them- Add an additional ERR notifier to test with- Check that no error was returned, for tests that weren't supposed to...
ErrorLoggingAsyncNotifier
This mixes AsyncNotifier with GanetiBaseAsyncoreDispatcher to provide anAsyncNotifier which will log errors, rather than bail out.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Balazs Lecz <leczb@google.com>
TestSingleFileEventHandler: abstract notifier type
Rather than hardcode that we have two notifiers, and notifier 0 is theterminating one, we abstract this with class level constants. This makesit easier to add more, with different features.
The only real change is that now the callback class takes as input the...
Mainloop: handle SIGINT as well (and terminate)
This is needed if daemons are in the foreground, and get ctrl+c-ed bythe user. Also add unittests to make sure the correct signals terminatethe mainloop.
Signed-off-by: Guido Trotter <ultrotter@google.com>...
import-export daemon: Allow changing compression method
For example, exports on the same node shouldn't be compressed.
Add asyncnotifier unittests
TestMainloop: test scheduler priorities as well
By using enterabs we can schedule events at the same time, which willthen be sorted by priority.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Fix race condition in mainloop unittest
Currently, in testDeferredCancel if the self._CancelEvent are enteredmore than 0.3 seconds after the _SendSig have been entered, the testcould fail. This is unlikely but may happen. To avoid it we useenterabs, to use absolute times....
TestMainloop.testReRun
Currently we never do that, but it's nice to know we can restart themainloop, after termination.
Add AsyncUDPSocket tests
Add initial mainloop unittests
RAPI changes for instance moves
Two new resources are added:- /2/instances/$name/prepare-export- /2/instances/$name/export
The documentation for the existing resource for creating instances is updatedfor remote imports. The RAPI client is extended for the new resources....
Implement opcode changes for remote-export
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Add opcode to prepare export
To prepare a remote export, the X509 key and certificate need to be generated.A handshake value is also returned for an easier check whether both clustersshare the same cluster domain secret.
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
ganeti-cleaner: Remove expired X509 certs
Importing/exporting an instance to a remote machine creates X509certificates which expire after some time. They need to be removed fromthe nodes as they become useless.
RAPI client: Rename Get{Node,Instance}Info, add new GetInstanceInfo
GetInstanceInfo should return the resource /2/instances/$name/info,but so far it returns /2/instances/$name. The same applies toGetNodeInfo, which returns /2/nodes/$name. Both names are stripped...
Add checks for master IP in cluster verify
This also updates a comment in the unittest for utils.py. We unittestthe new function for two things: correct reporting on real case (forlocalhost), and correct reporting with a mocked-out TcpPing that returns...
utils.IgnoreSignals
Remove duplicate code between a couple of asyncore related function byhaving a function in charge of handling EINTR errors. Unittests included.
Conflicts: daemons/ganeti-noded lib/daemon.py lib/rapi/baserlib.py lib/rapi/rlib2.py lib/utils.py
Signed-off-by: Luca Bigliardi <shammash@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
RAPI client: Implement instance creation
Currently this only supports the new instance creation request dataformat version 1, but support for the old version can be easilyimplemented.
Most arguments are optional and documented in the RAPI documentation....
RAPI: Add new request data format for instance creation
As mentioned in commit d975f482d, the current way of creating aninstance via RAPI is not very flexible. With this patch, a newinstance creation request data format is introduced and documented.Support can be detected by checking the list of features returned...
Add new /2/features RAPI resource
The /2/features RAPI resource can be used to detect optionalfeatures implemented by the RAPI server. This will be usedto recognize servers implementing a new request format forinstance creation requests.
Introduce Mlockall()
Add Mlockall() utility to lock current process' virtual adress space into RAM.
utils.ReadOneLineFile()
Read the first non-empty file line. When strict is set, abort if morethan one line is non-empty. Some unittests inspired by the reverted onesfrom commit b774bb106cc28d008e790ad2666eb64c76866fa0, and some new ones.
Remove oneline= parameter from utils.ReadFile
This partially reverts commit b774bb106cc28d008e790ad2666eb64c76866fa0.Unittests unrelated to that particular functionality but introduced inthat commit are left untouched. Since the temporary directory is now...
RAPI client: Don't check node role in client
Only the server knows which node roles can be set via RAPI.Constants are provided for convenience.
RAPI client: Update ReplaceInstanceDisks
- The RAPI client shouldn't check the replacement mode as only the server knows which ones are valid (constants are still provided for convenience)- Disks shouldn't be a list of names, but of indexes
RAPI client: Fix behaviour of “allocatable” storage flag
When modifying a storage unit, the “allocatable” flag should defaultto “no modification”. This replicates the behaviour of the commandline interface.
RAPI client: Encode boolean and None query values
Boolean values must be passed as 0 or 1. None should be an emptyvalue ("").
RAPI client: No longer check storage type locally
Only the RAPI server (actually masterd) knows which storage typesare valid. The exception can no longer be raised as the type isonly checked in the job.
RAPI client: Include HTTP error code in GanetiApiError exception
Having the HTTP error code allows users of the class to act differently basedon the error code.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: David Knowles <dknowles@google.com>
Support different HTTP error codes in RAPI client unittest mock
This allows testing error conditions (e.g. if a job wasn't found).
RAPI client: Fix error message for unsupported methods in unittest
RAPI client: Allow waiting for job changes
RAPI client: Rename “DeleteJob” to “CancelJob”
Jobs can't be deleted, but cancelled (even though the HTTP method“DELETE” is used).
RAPI client: Various code style changes
- Replace hardcoded values with constants- Code formatting- Exception messages without periods and fixed string formatting
RAPI client: Add missing unittest for RepairNodeStorageUnits
RAPI client: Always return job ID
Even removing tags returns a job ID.
Remove httplib2 dependency from ganeti.rapi.client
- It's possible to implement all functionality in ganeti.rapi.client using Python's standard modules httplib and urllib2- By doing so, proper SSL certificate verification is implemented- Adjust some of the code to Ganeti's code style (this is not yet...
utils.ReadFile: Add parameter to read only one line
Reading only one line is useful for reading passwords from files. Thisalso adds unittests for utils.ReadFile.
cli: Make PollJob generic to support other protocols
By separating the LUXI-specific code and stdio-related codeinto separate classes, we can make cli.PollJob protocol-agnostic, allowing it to be used with RAPI.
This patch also adds unittests for cli.PollJob....
Fix a unittest name and docstring
The docstring has been copied by the name, but the name reports amisspell of the function it is actually testing.
Fix a unittest docstring
utils.Retry: pass up timeout arguments
If Retry has to fail with RetryTimeout, it might be useful to pass theRetry argument to RetryTimeout, to help debugging outside the Retrycycle. We also define a RetryTimeout.RaiseInner() helper, to re-raise anexception passed to RetryAgain. All served with a side of unit tests....
Add a few Retry unittests
Check that Retry actually times out, and actually retries.
Small fixes for RAPI client
- Always check SSL certificate for validity- Actually JSON-encode HTTP content
Signed-off-by: David Knowles <dknowles@google.com>Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Add optional “salt” argument to utils.{,Verify}Sha1Hmac
We're using salted hashes all over the place.
Use utility functions for HMAC
HMAC will be used in more places. Centralizing some parts can't hurt.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Move cmdlib._VerifyCertificate to utils
This function will also be useful for inter-cluster instancemoves for verifying certificates.
Improve import/export daemon tests
- Progress messages- Log output on errors- Listen timeout tests
Retry connection in import-export daemon
Until now, exactly one attempt was made to establish a connection.If it failed, the whole import/export for the disk in questionaborted. Retrying will make it more reliable.
Unfortunately the listening part can't be made completely...
Conflicts: lib/utils.py: Trivial
masterd: Log PID, UID and GID of connected client
This can be very useful if client programs run as non-root.
cmdlib: Add utility for instance data import/export
Interpreting the backend's import/export daemon status is a bit tricky.This utility code keeps track of multiple transfers at the same time.Users can supply callback functions to react to events.
Timeouts are currently hardcoded. Intra-cluster instance moves will likely...
Add a basic unittest for uidpool.RequestUnusedUid
Signed-off-by: Balazs Lecz <leczb@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Add separate module for backported language functionality
utils.py, where they were before, is already huge.
utils: Add function for partial application of function arguments
The function's code was mostly copied from Python's documentationand it's equivalent to “functools.partial” in Python 2.5 and above.
utils: Add function to read locked PID file
This is useful in combination with utils.StartDaemon and will be used forreading the import/export daemon's PID file.
Fix RAPI client methods return values
Ensure that all RAPI client methods that should return job ids actuallyreturn job ids.
Signed-off-by: David Knowles <dknowles@google.com>Reviewed-by: Iustin Pop <iustin@google.com>Signed-off-by: Iustin Pop <iustin@google.com>
Conflicts: doc/security.rst trivial lib/cli.py trivial
Signed-off-by: Balazs Lecz <leczb@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
uidpool: test the separator= argument
Also, fix the string quoting style.
Add daemon for instance import and export
This backend daemon for instance import and export will be used totransfer instance data to other machines. It is implemented in a genericway to support different ways of data input and output. The third-partyprogram “socat”, which is already used by the KVM hypervisor abstraction,...
Add basic unittests for uid_pool
Signed-off-by: Balazs Lecz <leczb@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
utils: Add class to split string stream into lines
This will be used by the new import/export daemon to splitoutput by its child processes into lines.
Merge remote branch 'devel-2.1'
objects.Cluster: add method to get hv defaults
Currently the FillHV method is the one that does the cluster hvparams +os hvparams merger. However, in some cases we need to do just this,without adding the instance hvparams on top.
This patch adds a function to compute just this (hv + os hvp...
Rename the confd_client unittest (to confd.client)
This is to keep same naming across all tests (modules separate with dot,followed by _unittest.py).
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
ConfdClient.SendRequest: allow max coverage
This patch changes the coverage parameter to allow specification of maxcoverage (via -1), versus auto-computation (default, 0) and manualspecification.
Unittests are updated for this case too.
Signed-off-by: Iustin Pop <iustin@google.com>...
Fix utils.WaitForFdCondition inner retry loop
Commit dfdc4060 added WaitForFdCondition which uses utils.Retry withouthandling timeout exceptions. This breaks any nested retry loops.
This patch fixes the above function, and also changes utils.Retry todetect and warn future similar cases. In addition, we add a few small...
utils: Move wrapper code around os.makedirs into separate function
Fix unittest for the rapi client library
Wrong escape, so we make sure to use proper escapes (we want thebackslashes to be embedded, not interpreted). Also change " to ' to beeasier to read.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>...
Adding RAPI client library.
Signed-off-by: David Knowles <dknowles@google.com>Reviewed-by: Iustin Pop <iustin@google.com>Signed-off-by: Iustin Pop <iustin@google.com>(modified slightly the unittest to account for missing httplib2 library)
Remove unused import from test file
Add RPC calls to create and remove X509 certificates
Certificates and keys generated using these functions will be used forinter-cluster instance moves. As per design, the private key should neverleave the node.
utils: Add functions to sign and verify X509 certs using HMAC
Certificates exchanged via an untrusted third party should besigned to ensure they haven't been modified.
Merge remote branch 'origin/devel-2.1'
utils.RunCmd: Test case with reset_env set and setting variables
utils.CreateBackup: Use human-readable instead of seconds since Epoch
Seconds since the Epoch are not easily readable by a human. Using aformatted timestamp makes it easier (e.g.“….backup-2010-03-12_14_02_43.…”). This patch also makes OS logfiles usethis formatted timestamp....
Add unittest for daemon-util
This test doesn't cover everything, but it's better than nothing.
Improve ganeti.serializer unittests
Add unittests for ganeti.errors
Verify cluster certificates in LUVerifyCluster
When using pyOpenSSL 0.7 or above, LUClusterVerify will start to show awarning 30 days before a certificate expires. 7 days before thecertificate expires, the warning becomes an error. Once expired,LUVerifyCluster will always report an error. The latter is also supported...
utils: Add function to extract X509 cert validity
X509 uses ASN1 GENERALIZEDTIME or UTCTIME to store certificate validity.pyOpenSSL 0.7 and above allow us to retrieve both “notBefore” and“notAfter” as strings. Parsing them turned out to be a challenge since...
Adding unittests for objects.Cluster.FillHV
This adds tests for the stacking of objects.Cluster.FillHV to verifythat the override is working as expected.
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>