Mlockall: decrease warnings if ctypes module is not present
Node daemon prints a lot of warnings if --no-mlock option is not specified andctypes module is not present.
With the following patch the warning is printed only at noded startup.
Signed-off-by: Luca Bigliardi <shammash@google.com>...
TestAsyncUDPSocket: remove dead code and add test
- _ThreadedClient was added on the idea of making this unittest concurrent, which was actually never done (we could test everything without it, so well)- handle_write() was never called without filling the send queue, and...
TestAsyncUDPSocket: test for oversized sends
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Luca Bigliardi <shammash@google.com>
AsyncUDPSocket: fix IgnoreSignals usage and test
This bug was found in the asyncore master patch series, but actuallyapplies to 2.1 for AsyncUDPSocket as well.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Fix and Improve TryToRoman unittest
1) Don't break when the roman module is not found2) Test that not finding the roman module doesn't make TryToRoman fail(currently that is the case)
Move roman conversion to compat
The new TryToRoman function provides optional easy to use romanconversion. Nunc cum demonstrationi unitati.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Fix IgnoreSignals on socket.error
Some confusion arose handling EINTR on this function: in python 2.6socket.error is an IOError, and thus: - It's an EnvironmentError - It has an .errno member
In 2.4 and 2.5 it's not, and so its errno variable must be extracted...
Make utils.EnsureDirs() ignore umask
EnsureDirs() should create directories with the exact mode requestedin the arguments, but it currently applies the umask.This patch makes it independent from the umask.
Signed-off-by: Balazs Lecz <leczb@google.com>...
Test for errors during inotify callback
- Create a new _MyErrorLoggingAsyncNotifier class which registers error counts, rather than logging them- Add an additional ERR notifier to test with- Check that no error was returned, for tests that weren't supposed to...
ErrorLoggingAsyncNotifier
This mixes AsyncNotifier with GanetiBaseAsyncoreDispatcher to provide anAsyncNotifier which will log errors, rather than bail out.
TestSingleFileEventHandler: abstract notifier type
Rather than hardcode that we have two notifiers, and notifier 0 is theterminating one, we abstract this with class level constants. This makesit easier to add more, with different features.
The only real change is that now the callback class takes as input the...
Mainloop: handle SIGINT as well (and terminate)
This is needed if daemons are in the foreground, and get ctrl+c-ed bythe user. Also add unittests to make sure the correct signals terminatethe mainloop.
Signed-off-by: Guido Trotter <ultrotter@google.com>...
Add asyncnotifier unittests
TestMainloop: test scheduler priorities as well
By using enterabs we can schedule events at the same time, which willthen be sorted by priority.
Fix race condition in mainloop unittest
Currently, in testDeferredCancel if the self._CancelEvent are enteredmore than 0.3 seconds after the _SendSig have been entered, the testcould fail. This is unlikely but may happen. To avoid it we useenterabs, to use absolute times....
TestMainloop.testReRun
Currently we never do that, but it's nice to know we can restart themainloop, after termination.
Add AsyncUDPSocket tests
Add initial mainloop unittests
RAPI client: Rename Get{Node,Instance}Info, add new GetInstanceInfo
GetInstanceInfo should return the resource /2/instances/$name/info,but so far it returns /2/instances/$name. The same applies toGetNodeInfo, which returns /2/nodes/$name. Both names are stripped...
Add checks for master IP in cluster verify
This also updates a comment in the unittest for utils.py. We unittestthe new function for two things: correct reporting on real case (forlocalhost), and correct reporting with a mocked-out TcpPing that returns...
utils.IgnoreSignals
Remove duplicate code between a couple of asyncore related function byhaving a function in charge of handling EINTR errors. Unittests included.
RAPI client: Implement instance creation
Currently this only supports the new instance creation request dataformat version 1, but support for the old version can be easilyimplemented.
Most arguments are optional and documented in the RAPI documentation....
RAPI: Add new request data format for instance creation
As mentioned in commit d975f482d, the current way of creating aninstance via RAPI is not very flexible. With this patch, a newinstance creation request data format is introduced and documented.Support can be detected by checking the list of features returned...
Add new /2/features RAPI resource
The /2/features RAPI resource can be used to detect optionalfeatures implemented by the RAPI server. This will be usedto recognize servers implementing a new request format forinstance creation requests.
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
Introduce Mlockall()
Add Mlockall() utility to lock current process' virtual adress space into RAM.
Signed-off-by: Luca Bigliardi <shammash@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
utils.ReadOneLineFile()
Read the first non-empty file line. When strict is set, abort if morethan one line is non-empty. Some unittests inspired by the reverted onesfrom commit b774bb106cc28d008e790ad2666eb64c76866fa0, and some new ones.
Remove oneline= parameter from utils.ReadFile
This partially reverts commit b774bb106cc28d008e790ad2666eb64c76866fa0.Unittests unrelated to that particular functionality but introduced inthat commit are left untouched. Since the temporary directory is now...
RAPI client: Don't check node role in client
Only the server knows which node roles can be set via RAPI.Constants are provided for convenience.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
RAPI client: Update ReplaceInstanceDisks
- The RAPI client shouldn't check the replacement mode as only the server knows which ones are valid (constants are still provided for convenience)- Disks shouldn't be a list of names, but of indexes
RAPI client: Fix behaviour of “allocatable” storage flag
When modifying a storage unit, the “allocatable” flag should defaultto “no modification”. This replicates the behaviour of the commandline interface.
RAPI client: Encode boolean and None query values
Boolean values must be passed as 0 or 1. None should be an emptyvalue ("").
RAPI client: No longer check storage type locally
Only the RAPI server (actually masterd) knows which storage typesare valid. The exception can no longer be raised as the type isonly checked in the job.
RAPI client: Include HTTP error code in GanetiApiError exception
Having the HTTP error code allows users of the class to act differently basedon the error code.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: David Knowles <dknowles@google.com>
Support different HTTP error codes in RAPI client unittest mock
This allows testing error conditions (e.g. if a job wasn't found).
RAPI client: Fix error message for unsupported methods in unittest
RAPI client: Allow waiting for job changes
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
RAPI client: Rename “DeleteJob” to “CancelJob”
Jobs can't be deleted, but cancelled (even though the HTTP method“DELETE” is used).
RAPI client: Various code style changes
- Replace hardcoded values with constants- Code formatting- Exception messages without periods and fixed string formatting
RAPI client: Add missing unittest for RepairNodeStorageUnits
RAPI client: Always return job ID
Even removing tags returns a job ID.
Remove httplib2 dependency from ganeti.rapi.client
- It's possible to implement all functionality in ganeti.rapi.client using Python's standard modules httplib and urllib2- By doing so, proper SSL certificate verification is implemented- Adjust some of the code to Ganeti's code style (this is not yet...
utils.ReadFile: Add parameter to read only one line
Reading only one line is useful for reading passwords from files. Thisalso adds unittests for utils.ReadFile.
cli: Make PollJob generic to support other protocols
By separating the LUXI-specific code and stdio-related codeinto separate classes, we can make cli.PollJob protocol-agnostic, allowing it to be used with RAPI.
This patch also adds unittests for cli.PollJob....
Fix a unittest name and docstring
The docstring has been copied by the name, but the name reports amisspell of the function it is actually testing.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Balazs Lecz <leczb@google.com>
Fix a unittest docstring
utils.Retry: pass up timeout arguments
If Retry has to fail with RetryTimeout, it might be useful to pass theRetry argument to RetryTimeout, to help debugging outside the Retrycycle. We also define a RetryTimeout.RaiseInner() helper, to re-raise anexception passed to RetryAgain. All served with a side of unit tests....
Add a few Retry unittests
Check that Retry actually times out, and actually retries.
Small fixes for RAPI client
- Always check SSL certificate for validity- Actually JSON-encode HTTP content
Signed-off-by: David Knowles <dknowles@google.com>Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
masterd: Log PID, UID and GID of connected client
This can be very useful if client programs run as non-root.
Add a basic unittest for uidpool.RequestUnusedUid
Signed-off-by: Balazs Lecz <leczb@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Add separate module for backported language functionality
utils.py, where they were before, is already huge.
utils: Add function for partial application of function arguments
The function's code was mostly copied from Python's documentationand it's equivalent to “functools.partial” in Python 2.5 and above.
Fix RAPI client methods return values
Ensure that all RAPI client methods that should return job ids actuallyreturn job ids.
Signed-off-by: David Knowles <dknowles@google.com>Reviewed-by: Iustin Pop <iustin@google.com>Signed-off-by: Iustin Pop <iustin@google.com>
uidpool: test the separator= argument
Also, fix the string quoting style.
Add basic unittests for uid_pool
Signed-off-by: Balazs Lecz <leczb@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
utils: Add class to split string stream into lines
This will be used by the new import/export daemon to splitoutput by its child processes into lines.
objects.Cluster: add method to get hv defaults
Currently the FillHV method is the one that does the cluster hvparams +os hvparams merger. However, in some cases we need to do just this,without adding the instance hvparams on top.
This patch adds a function to compute just this (hv + os hvp...
Rename the confd_client unittest (to confd.client)
This is to keep same naming across all tests (modules separate with dot,followed by _unittest.py).
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
ConfdClient.SendRequest: allow max coverage
This patch changes the coverage parameter to allow specification of maxcoverage (via -1), versus auto-computation (default, 0) and manualspecification.
Unittests are updated for this case too.
Signed-off-by: Iustin Pop <iustin@google.com>...
Fix utils.WaitForFdCondition inner retry loop
Commit dfdc4060 added WaitForFdCondition which uses utils.Retry withouthandling timeout exceptions. This breaks any nested retry loops.
This patch fixes the above function, and also changes utils.Retry todetect and warn future similar cases. In addition, we add a few small...
utils: Move wrapper code around os.makedirs into separate function
Fix unittest for the rapi client library
Wrong escape, so we make sure to use proper escapes (we want thebackslashes to be embedded, not interpreted). Also change " to ' to beeasier to read.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>...
Adding RAPI client library.
Signed-off-by: David Knowles <dknowles@google.com>Reviewed-by: Iustin Pop <iustin@google.com>Signed-off-by: Iustin Pop <iustin@google.com>(modified slightly the unittest to account for missing httplib2 library)
Remove unused import from test file
utils.RunCmd: Test case with reset_env set and setting variables
utils.CreateBackup: Use human-readable instead of seconds since Epoch
Seconds since the Epoch are not easily readable by a human. Using aformatted timestamp makes it easier (e.g.“….backup-2010-03-12_14_02_43.…”). This patch also makes OS logfiles usethis formatted timestamp....
Add unittest for daemon-util
This test doesn't cover everything, but it's better than nothing.
Improve ganeti.serializer unittests
Add unittests for ganeti.errors
Verify cluster certificates in LUVerifyCluster
When using pyOpenSSL 0.7 or above, LUClusterVerify will start to show awarning 30 days before a certificate expires. 7 days before thecertificate expires, the warning becomes an error. Once expired,LUVerifyCluster will always report an error. The latter is also supported...
utils: Add function to extract X509 cert validity
X509 uses ASN1 GENERALIZEDTIME or UTCTIME to store certificate validity.pyOpenSSL 0.7 and above allow us to retrieve both “notBefore” and“notAfter” as strings. Parsing them turned out to be a challenge since...
Adding unittests for objects.Cluster.FillHV
This adds tests for the stacking of objects.Cluster.FillHV to verifythat the override is working as expected.
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Provide unittests for http.auth
To simplify writing unittests, one data structure class in http.server isalso changed. According to the coverage utility, this provides 95%coverage.
http.auth: Fix bug with checking hashed passwords
When username and password were sent for a resource not requiringauthentication, it wouldn't be accepted if the user in question had ahashed password. The reason was that the function GetAuthRealm used to...
Support passing in file object in utils.FileLock
This way we can re-use file objects opened in other places. Also add moreunittests.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Support arguments in utils.RunInSeparateProcess
Add a function to validate and normalize hostnames
This differs slightly from the specification, by allowing names to startwith digits, not checking the length of individual components, andallowing underscores.
ListVisibleFiles: require normalized path names
This patch changes ListVisibleFiles to raise ProgrammerError if it'spassed a non-absolute/non-normalized path name, and adds unittests forthis behaviour.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
utils: Add a PathJoin function
This will replace os.path.join since it is not safe for directorytraversal issues.
Add multi-key support to the serializer
Signed-off-by: Balazs Lecz <leczb@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Add unittests for ganeti.opcodes
According to “coverage”, this covers 99% of the code.
Implement utils.RunParts and use it for hooks
This function is a generic pythonic version of runparts. We currentlyuse it in the backend HooksRunner, but we'll use it for runningdifferent directories as well.
Add reset_env option to RunCmd
This allows to run a command with only the passed in environment, ratherthan just updating the default one with it.
Now with unit testing.
Move RunInSeparateProcess to ganeti.utils
This function could be useful in other places and thisway we can easily unittest it.
Add function to reset tempfile module after fork
On fork, the tempfile module's pseudo random generator isnot reset. If several processes (e.g. two children or parentand child) try to create a temporary file, they'll conflict.This function can be used to reset the name generator which...
Add unittest for utils._FingerprintFile
Merge branch 'stable-2.1' into devel-2.1
Add unittests for cli.GenerateTable
Fix long-standing race condition bug in locking unittest
Every once in a while we saw failures in the locking unittest, butweren't able to determine the cause. This patch fixes it by usingthreading.Event instead of plain threading.Condition to notify another...
testutils: Print name of test program before running it
While the name can be looked up in Makefile.am, thisis useful when an automated test just hangs and needsto be killed.
Add an UnescapeAndSplit function
In many cases, where we accept (usually from the command line) a list ofparameters, we remove the use of the separator as an component of any ofthe elements.
This patch adds a new function that can split strings of the form...
workerpool: Make worker ID alphanumeric
Having a proper name instead of just a number makes debuggingeasier.
gnt-cluster verify: Warn if node time diverges too far
The warning will be generated if the clocks diverge by morethan 150 seconds. Due to the way the RPC system works, wecannot get exact time differences, e.g. if one of thequeried nodes is broken. The comparision is done using a...
Remove testJsonIndent unittest
It can't work on older distributions where simplejsondoesn't have indentation support.
Revert "Get rid of utils.CommaJoin"
This reverts commit 6915bc28fe053e92aa16cf2d974d205f1140219c based on thread onganeti-devel.
Conflicts:
lib/cmdlib.py (due to the error code classification, trivial)
Handle EEXIST in utils.RenameFile
This should fix an issue I've seen exactly once during testing. It might havebeen caused by parallel RPC calls to archive jobs.
[…] ganeti-noded:112 ERROR Error in RPC call […] File "/usr/lib/python2.4/site-packages/ganeti/backend.py", line 2365, in JobQueueRename...
config.Add{Node,Instance}: get the ec id
This is ok because adding a node or instance cannot happen in a query.
We get the ec id from the LU and pass it to _EnsureUUID, which willthen for now not use it.
Cache JSON encoders and sort keys
The sort_keys argument is supported since simplejson 1.3.
Ignore log messages in unittests