Move ShellWriter class to utils
Also add unittest.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Rename test for utils.IgnoreProcessNotFound
Usually our tests are named “Test…”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Manuel Franceschini <livewire@google.com>
jqueue: Factorize code waiting for job changes
By splitting the _WaitForJobChangesHelper class into multiple smallerclasses, we gain in several places:
- Simpler code, less interaction between functions and variables- Easy to unittest (close to 100% coverage)...
Merge remote branch 'origin/devel-2.1'
Conflicts: test/ganeti.rapi.client_unittest.py: Trivial test/ganeti.rapi.rlib2_unittest.py: Trivial...
RAPI client: Implement old instance creation request format
Commit 8a47b4478 implemented instance creation in the RAPI client,but it left out support for the old instance creation request format.This patch now implements the old format as good as possible. This...
rlib2: Use constants for disk and NIC parameters
These constants were added in commit bd061c35, but the parsing codewas not updated. This also fixes a bug where a NIC's MAC addresswasn't used.
Use reserved documentation IPs and domains
Use RFC 5737 IP addresses and RFC 2606 domain names in allunittests, docs, qa and docstrings.
Signed-off-by: Manuel Franceschini <livewire@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Provide feedback function for all LU methods
By exposing mcpu's _Feedback function (now renamed to “Log”) to LU's,methods like ExpandNames can also write to the job execution log.
Add function to format all job log messages
Just calling utils.SafeEncode on the log message failed when itwasn't of the type ELOG_MESSAGE and not a string. Now non-messagelog entries are formatted using repr().
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
Confd IPv6 support
This patch series basically adds a new parameter 'family' to the constructorsof daemon.AsyncUDPSocket and confd.client.ConfdUDPClient. This enables theusers of these two classes to support IPv6.
In ganeti-confd.ConfdAsyncUDPClient a method to check the address families of...
Introduce lib/netutils.py
This patch moves network utility functions to a dedicated module.
_CheckIAllocatorOrNode unit tests
Add unit tests to check the function of _CheckIAllocatorOrNode
Signed-off-by: Apollon Oikonomopoulos <apollon@noc.grnet.gr>Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Merge branch 'devel-2.1'
Signed-off-by: Luca Bigliardi <shammash@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Mlockall: decrease warnings if ctypes module is not present
Node daemon prints a lot of warnings if --no-mlock option is not specified andctypes module is not present.
With the following patch the warning is printed only at noded startup.
Signed-off-by: Luca Bigliardi <shammash@google.com>...
Add utils.GetMounts()
Signed-off-by: Balazs Lecz <leczb@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Add drbd_usermode_helper to configuration
BaseDRBD: provide a way to query usermode_helper parameter
Rework the "type" system
This patch merges the _OP_REQP and _OP_DEFS class attributes into a_OP_PARAMS list, which holds both. The associated unittest checks thatall opcode attributes are declared and checked, and that no LU uses theold fields (could be removed later)....
RAPI client: Switch to pycURL
Currently the RAPI client uses the urllib2 and httplib modules fromPython's standard library. They're used with pyOpenSSL in a very fragileway, and there are known issues when receiving large responses from a RAPIserver....
DRBD IPv6 support
Support IPv6 configuration for 'drbdsetup show' parser and add unittestsconcerning that case. Renames some data files to use consistent namesclarifying their usage.
Signed-off-by: Manuel Franceschini <livewire@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
IPv6 support for utils.TcpPing()
Add function to retrieve family of an ip address
Signed-off-by: Manuel Franceschini <livewire@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Introduce utils.IsValidIP{4,6}()
This patch introduces functions to check for valid IPv4 and IPv6addresses and converts IsValidIP() to return True if it is either a IPv4or a IPv6 address.
For now we do not change the functional behavior and replace IsValidIP...
User assertFalse instead of assert_(not ...)
Rename some constants to facilitate IPv6 support
AsyncTerminatedMessageStream: send_message
This function adds the ability for a AsyncTerminatedMessageStream tohave a thread-safe message delivery function.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
AsyncTerminatedMessageStream: limit message count
Currently the message stream can process any number of messages inparallel (if they get dispatched to different threads or processes).In order to limit their number we only handle messages and read from...
Add test script for cfgupgrade
This should catch cases where we update the configuration version, butforget to adjust cfgupgrade accordingly.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
count the number of tasks done in the wp unittest
Currently there's no way to know if something actually gets done.After this check we actually test that the threads do their job.
Workerpool.AddManyTasks: check tasks type
Each task has to be a sequence, or the RunTask call will fail.
Change ganeti-cleaner unittest to not use random values
Using random values in unittests isn't good. This one broke exactlywhen building the 2.2.0~beta0 release. I suspect there were duplicatejob IDs generated (due to $large being not so large).
WorkerPool.AddManyTasks
Useful if we want to add many tasks at once, without contention with theprevious one we added starting.
ListVisibleFiles: do not sort output
Among all users, turns out just one may need the output to be sorted.All the others can cope without.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Use import/export magic for backup/import and inter-cluster moves
This should prevent bugs in our code from accidentally overwritingdisks.
import/export daemon: Add support for a magic prefix
This “magic” value will be used to ensure that we don't accidentiallyconnect to the wrong daemon (e.g. due to a bug), comparable to DRBD'sper-disk secret. Just depending on the SSL certificate isn't enough...
import/export: Limit max length of socat options
import/export: Validate remote host/port
The hostname and port received from the remote cluster shouldbe validated, just in case.
utils: Add function to validate service name
Handle ESRCH when sending signals
Upon sending signals, ESRCH can be reported when the target nolonger exists.
ListVisibleFiles: do optional sorting
Improve import-export unittest a bit
- Increase timeouts from 10 to 30 seconds (this still breaks when the machine is busy, e.g. using bonnie++)- Depend on only one timeout per test instead of three- Reset variables before each test
Test client timeout for import-export daemon
Generate import-export unittest certs in parallel
Generating certificates can be slow.
_BaseCondition: allow saving/restoring state
SharedLock _acquire_restore and _release_save
If a shared lock is used inside a condition, we need to make sure thatit's reacquired in the same way as it was originally, after the wait.
Add unittest for ganeti-cleaner
import/export daemon: Record amount of data transferred
This reports the amount of data transferred and the throughput (averagedover 60 seconds) to the master daemon. While not yet fully implemented,once the export scripts report the expected data size, we can even provide...
import/export: Show progress updates to user
With this patch, we show progress updates approx. once per minute.
utils: Add function to format seconds
import/export unittest: Test large(r) transfer
import/export unittest: Improve logging and fix one race condition
Apart from improved logging, one race condition is fixed. Ifthe destination's status file became available, the port wouldbe returned immediately, even if it was still “None”. Most ofthe time it worked, but not always. Now an additional check...
daemon.AsyncAwaker
This new asyncore dispatcher can be used to force a thread running theasyncore loop to awake from the select, by signaling it on one of itsselected sockets.
Test the new streaming daemon classes
Unittests cover AsyncStreamServer and AsyncTerminatedMessageStream withboth tcp and unix sockets.
TestAsyncUDPSocket: remove dead code and add test
- _ThreadedClient was added on the idea of making this unittest concurrent, which was actually never done (we could test everything without it, so well)- handle_write() was never called without filling the send queue, and...
TestAsyncUDPSocket: test for oversized sends
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Luca Bigliardi <shammash@google.com>
Conflicts: test/ganeti.daemon_unittest.py...
AsyncUDPSocket: fix IgnoreSignals usage and test
This bug was found in the asyncore master patch series, but actuallyapplies to 2.1 for AsyncUDPSocket as well.
utils: Add function to check whether process handles a signal
This will be used to avoid a race condition between starting a program (ddfor import/export) and sending signals to it.
Fix and Improve TryToRoman unittest
1) Don't break when the roman module is not found2) Test that not finding the roman module doesn't make TryToRoman fail(currently that is the case)
Move roman conversion to compat
The new TryToRoman function provides optional easy to use romanconversion. Nunc cum demonstrationi unitati.
Fix IgnoreSignals on socket.error
Some confusion arose handling EINTR on this function: in python 2.6socket.error is an IOError, and thus: - It's an EnvironmentError - It has an .errno member
In 2.4 and 2.5 it's not, and so its errno variable must be extracted...
reraise exceptions in async tests' error handlers
This makes sure that any unforeseen error raises an exception ratherthen just increasing a counter. It makes unittest debugging a loteasier.
Move hash functions to the compat module
Since the hash functions' changed their module name between python 2.4and 2.6, and we have to do an try/import/except trick, we'll do it justonce, for both hash functions, and in compat.py. This also fixes a use...
Fix {Ignore, RetryOn}Signals on socket.error
Some confusion arose handling EINTR on those functions: in python 2.6socket.error is an IOError, and thus: - It's an EnvironmentError - It has an .errno member
Make utils.EnsureDirs() ignore umask
EnsureDirs() should create directories with the exact mode requestedin the arguments, but it currently applies the umask.This patch makes it independent from the umask.
Signed-off-by: Balazs Lecz <leczb@google.com>...
import/export daemon: Move command building into separate module
The import/export daemon code is already large. Moving some codeto a separate module will make it smaller and easier to test.
Test for errors during inotify callback
- Create a new _MyErrorLoggingAsyncNotifier class which registers error counts, rather than logging them- Add an additional ERR notifier to test with- Check that no error was returned, for tests that weren't supposed to...
ErrorLoggingAsyncNotifier
This mixes AsyncNotifier with GanetiBaseAsyncoreDispatcher to provide anAsyncNotifier which will log errors, rather than bail out.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Balazs Lecz <leczb@google.com>
TestSingleFileEventHandler: abstract notifier type
Rather than hardcode that we have two notifiers, and notifier 0 is theterminating one, we abstract this with class level constants. This makesit easier to add more, with different features.
The only real change is that now the callback class takes as input the...
Mainloop: handle SIGINT as well (and terminate)
This is needed if daemons are in the foreground, and get ctrl+c-ed bythe user. Also add unittests to make sure the correct signals terminatethe mainloop.
Signed-off-by: Guido Trotter <ultrotter@google.com>...
import-export daemon: Allow changing compression method
For example, exports on the same node shouldn't be compressed.
Add asyncnotifier unittests
TestMainloop: test scheduler priorities as well
By using enterabs we can schedule events at the same time, which willthen be sorted by priority.
Fix race condition in mainloop unittest
Currently, in testDeferredCancel if the self._CancelEvent are enteredmore than 0.3 seconds after the _SendSig have been entered, the testcould fail. This is unlikely but may happen. To avoid it we useenterabs, to use absolute times....
TestMainloop.testReRun
Currently we never do that, but it's nice to know we can restart themainloop, after termination.
Add AsyncUDPSocket tests
Add initial mainloop unittests
RAPI changes for instance moves
Two new resources are added:- /2/instances/$name/prepare-export- /2/instances/$name/export
The documentation for the existing resource for creating instances is updatedfor remote imports. The RAPI client is extended for the new resources....
Implement opcode changes for remote-export
Add opcode to prepare export
To prepare a remote export, the X509 key and certificate need to be generated.A handshake value is also returned for an easier check whether both clustersshare the same cluster domain secret.
ganeti-cleaner: Remove expired X509 certs
Importing/exporting an instance to a remote machine creates X509certificates which expire after some time. They need to be removed fromthe nodes as they become useless.
RAPI client: Rename Get{Node,Instance}Info, add new GetInstanceInfo
GetInstanceInfo should return the resource /2/instances/$name/info,but so far it returns /2/instances/$name. The same applies toGetNodeInfo, which returns /2/nodes/$name. Both names are stripped...
Add checks for master IP in cluster verify
This also updates a comment in the unittest for utils.py. We unittestthe new function for two things: correct reporting on real case (forlocalhost), and correct reporting with a mocked-out TcpPing that returns...
utils.IgnoreSignals
Remove duplicate code between a couple of asyncore related function byhaving a function in charge of handling EINTR errors. Unittests included.
Conflicts: daemons/ganeti-noded lib/daemon.py lib/rapi/baserlib.py lib/rapi/rlib2.py lib/utils.py
Signed-off-by: Luca Bigliardi <shammash@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
RAPI client: Implement instance creation
Currently this only supports the new instance creation request dataformat version 1, but support for the old version can be easilyimplemented.
Most arguments are optional and documented in the RAPI documentation....
RAPI: Add new request data format for instance creation
As mentioned in commit d975f482d, the current way of creating aninstance via RAPI is not very flexible. With this patch, a newinstance creation request data format is introduced and documented.Support can be detected by checking the list of features returned...
Add new /2/features RAPI resource
The /2/features RAPI resource can be used to detect optionalfeatures implemented by the RAPI server. This will be usedto recognize servers implementing a new request format forinstance creation requests.
Introduce Mlockall()
Add Mlockall() utility to lock current process' virtual adress space into RAM.
utils.ReadOneLineFile()
Read the first non-empty file line. When strict is set, abort if morethan one line is non-empty. Some unittests inspired by the reverted onesfrom commit b774bb106cc28d008e790ad2666eb64c76866fa0, and some new ones.
Remove oneline= parameter from utils.ReadFile
This partially reverts commit b774bb106cc28d008e790ad2666eb64c76866fa0.Unittests unrelated to that particular functionality but introduced inthat commit are left untouched. Since the temporary directory is now...
RAPI client: Don't check node role in client
Only the server knows which node roles can be set via RAPI.Constants are provided for convenience.