import/export: Validate remote host/port
The hostname and port received from the remote cluster shouldbe validated, just in case.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
utils: Add function to validate service name
Handle ESRCH when sending signals
Upon sending signals, ESRCH can be reported when the target nolonger exists.
Cache a few bits of status in jqueue
Currently each time we submit a job we check the job queue size, and thedrained file. With this change we keep these pieces of information inmemory and don't read them from the filesystem each time.
Significant changes include:...
ListVisibleFiles: do optional sorting
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Fix a TODO in _QueuedJob
Rather than raising Exception use GenericError and explain a bit betterwhat happened.
Remove unused parameter from function
This also removes the relevant pylint disable.No point in keeping unused parameters around: if/when we need them it'seasy to add it back.
Optimize _GetJobIDsUnlocked
Currently we sort the list of job queue files twice (once inutils.ListVisibleFiles with sort and then later with NiceSort). We applythe _RE_JOB_FILE regular expression twice (once in _ListJobFiles andonce in _ExtractJobID). This simplifies the code a little, and a couple...
jqueue: Rename _queue_lock to _queue_filelock
The name clarifies the difference between this and the internal lock.Also explain a bit better what it is.
jstore._ReadNumericFile: use utils.ReadFile
Remove locking._CountingCondition
This class is unused and untested. We must have forgot it around.
Remove the job queue drain rpc call
This call was introduced but never used. In two years.Since it's just creating/removing a file it can also be in simpler ways,without a special rpc call, if/when we need it again. In the meantime,let's give it to history....
_BaseCondition: allow saving/restoring state
SharedLock _acquire_restore and _release_save
If a shared lock is used inside a condition, we need to make sure thatit's reacquired in the same way as it was originally, after the wait.
Submit[*each*]Pending job
This is useful so we can test both SubmitJob and SubmitManyJobs.
cfgupgrade: Local variable for cluster-domain-secret filename
This is necessary to allow cfgupgrade to work on a non-standard directory.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
import/export: Allow script to predict size
Once we have a size for an export (in the context of theimport/export daemon), we can provide the user with apercentage and ETA.
backend: Enable export size prediction
Show formatted ETA for disk sync and import/export
import/export daemon: Record amount of data transferred
This reports the amount of data transferred and the throughput (averagedover 60 seconds) to the master daemon. While not yet fully implemented,once the export scripts report the expected data size, we can even provide...
import/export: Show progress updates to user
With this patch, we show progress updates approx. once per minute.
Introduce harcdoded timeouts for each RPC call
This patch adds a table with per-opcode timeouts. They were chosen in anempiric, rather than scientific, way - see the comments in lib/rpc.py.
The patch also shows how custom timeouts can be used - call_test_delay...
http client: support per-request read timeout
Currently, the read timeout is hardcoded in theHttpClientRequestExecutor class. The patch changes the timeout so thatit's a per-request property, and makes the rpc.Client class pass oneexplicitly in. Furthermore, we modify the rpc.RpcRunner class to support...
Let ganeti-rapi run under a different user/group
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Make it possible to call utils.Daemonize with uid and gid to run as
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Merge branch 'devel-2.1'
_ExecuteKVMRuntime: fix hv parameter fun
When executing the kvm runtime we were currently accessing a mix of theparameters as configured currently on the instance and the ones it wasstarted with. We were doing it without a precise criteria, but quite by...
Update FinalizeMigration docstring
This is used not only for aborted migrations, so the docstring shouldreflect that.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
LUGrowDisk: fix operation on down instances
Currently it's impossible to grow a disk if an instance is shutdown,because the disk could not be assembled. Now we take care of assemblingit, and shutting it down after.
Signed-off-by: Guido Trotter <ultrotter@google.com>...
Allow disk operation to act on a subset of disks
If the disks= parameter is passed, we can assemble/wait forsync/shutdown only some disks belonging to an instance, rather than all.
This is useful to only activate/sync/shutdown the affected disk whengrowing it....
utils: Add function to format seconds
import/export unittest: Improve logging and fix one race condition
Apart from improved logging, one race condition is fixed. Ifthe destination's status file became available, the port wouldbe returned immediately, even if it was still “None”. Most ofthe time it worked, but not always. Now an additional check...
daemon.AsyncAwaker
This new asyncore dispatcher can be used to force a thread running theasyncore loop to awake from the select, by signaling it on one of itsselected sockets.
daemon.AsyncStreamServer
This is a new asyncore server which handles listening stream sockets bycalling a non-implemented function for each connection it accepts. It'sthe stream-oriented cousing of the AsyncUDPSocket.
daemon.AsyncTerminatedMessageStream
This is the counterpart of the AsyncStreamServer can be used to handleconnected sockets returned from connected clients if the protocol is aterminator separated message stream. Nothing in this class is serverspecific though: it can be used as a client as well, if the client is...
ganeti-watcher should attempt to fix ganeti-rapi
Update ganeti-watcher so that it tests the master's RAPI port with asimple test (in this case GetVersion). If it fails, make one attemptat restarting ganeti-rapi and retest.
- daemons/ganeti-watcher: Test rapi and make one attempt at restarting it....
Add RemoveDir utility function
Backported from master, 72087dcd5b06c0127e2ec3bf8c80f7f54da3fb01
Signed-off-by: Balazs Lecz <leczb@google.com>Reviewed-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Merge remote branch 'origin/devel-2.1'
Conflicts: test/ganeti.daemon_unittest.py...
AsyncUDPSocket: fix IgnoreSignals usage and test
This bug was found in the asyncore master patch series, but actuallyapplies to 2.1 for AsyncUDPSocket as well.
Explicitely return None from IgnoreSignals
Same result, but what happens is clearer.
Add KVM chroot feature
This patch adds a new boolean hypervisor parameter to the KVM hypervisor,named 'use_chroot'.If it's turned on for an instance, than KVM is started in "chroot mode":Ganeti creates an empty directory for the instance and passes the path...
utils: Add function to check whether process handles a signal
This will be used to avoid a race condition between starting a program (ddfor import/export) and sending signals to it.
Distribute cluster domain secret
The cluster domain secret file was not distributed to other nodes.
FormatUidPool: provide optional roman conversion
The convert= option of compat.tryToRoman is used to do optionalconversion without duplicating formatting code.
Move roman conversion to compat
The new TryToRoman function provides optional easy to use romanconversion. Nunc cum demonstrationi unitati.
ssconf: error out when writing oversized files
Since we impose a maximum limit when reading ssconf files, let's errorout when trying to write them too big, so we don't pretend everything isok, and make mistakes when we actually read partial files.
Add a new opcode timestamp field
Since the current start_timestamp opcode attribute refers to the initalstart time, before locks are acquired, it's not useful to determine theactual execution order of two opcodes/jobs competing for the same lock.
This patch adds a new field, exec_timestamp, that is updated when the...
Fix IgnoreSignals on socket.error
Some confusion arose handling EINTR on this function: in python 2.6socket.error is an IOError, and thus: - It's an EnvironmentError - It has an .errno member
In 2.4 and 2.5 it's not, and so its errno variable must be extracted...
Move hash functions to the compat module
Since the hash functions' changed their module name between python 2.4and 2.6, and we have to do an try/import/except trick, we'll do it justonce, for both hash functions, and in compat.py. This also fixes a use...
Fix {Ignore, RetryOn}Signals on socket.error
Some confusion arose handling EINTR on those functions: in python 2.6socket.error is an IOError, and thus: - It's an EnvironmentError - It has an .errno member
RAPI client should convert urllib2.URLError to GanetiApiError
Signed-off-by: Tom Limoncelli <tlim@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
KVM: Migration bandwidth and downtime control
Introduce 2 new hypervisor options, migration_bandwidth and migration_downtimeand implement KVM migration bandwidth and downtime control.
migration_bandwidth controls KVM's maximal bandwidth during migration, in...
Make utils.EnsureDirs() ignore umask
EnsureDirs() should create directories with the exact mode requestedin the arguments, but it currently applies the umask.This patch makes it independent from the umask.
Signed-off-by: Balazs Lecz <leczb@google.com>...
import/export daemon: Move command building into separate module
The import/export daemon code is already large. Moving some codeto a separate module will make it smaller and easier to test.
import/export daemon: Move some I/O processing code to module
The code parsing the child process' output is moved to a separateclass in the impexpd module. As more programs are added, it'llbecome more complex and should be separated.
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
Signed-off-by: Balazs Lecz <leczb@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Fix two race conditions in reboot instance
If the instance crashes between backend.InstanceReboot checks the listof running instances and the execution of hv_xen.RebootInstance,ini_info will be None. And if the instance doesn't reboot fast enough,new_info will be None. Both cases lead to “TypeError: unsubscriptable...
Support for latin friendly output in node list
SingleFileEventHandler: Remove try/except blocks
Since now we use the SingleFileEventHandler together with an errorhandling asyncore dispatcher, we don't need the internal try/exceptanymore.
ErrorLoggingAsyncNotifier
This mixes AsyncNotifier with GanetiBaseAsyncoreDispatcher to provide anAsyncNotifier which will log errors, rather than bail out.
daemon.GanetiBaseAsyncoreDispatcher
Abstract a few common functionalities between all ganeti asyncoredispatchers: - Handle errors by logging them, and then continue - By default check sockets only for readability
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Balazs Lecz <leczb@google.com>
Mainloop: handle SIGINT as well (and terminate)
This is needed if daemons are in the foreground, and get ctrl+c-ed bythe user. Also add unittests to make sure the correct signals terminatethe mainloop.
SingleFileEventHandler: update comments
The comments in the SingleFileEventHandler are still confd-specific.Update them to make them generic for any single-file monitoring.
Allow control of import/export compression method
For exports to/imports from the same machine, compression willnot be used anymore.
Put common import/export daemon options into object
The X509 key name and CA are passed from cmdlib all the way tothe backend import/export daemon. With the addition of an optionto choose the compression method, another parameter would haveto be passed all the way. By moving these options to a separate...
import-export daemon: Allow changing compression method
For example, exports on the same node shouldn't be compressed.
Make ConfdInotifyEventHandler a library function
Cut&Paste, plus the following changes: - The class is renamed to SingleFileEventHandler - The monitored filename must be passed in and doesn't default to the ganeti cluster config file - A small docstring is added to the class...
Comment on AsyncNotifier upstream's availability
Since we contributed AsyncNotifier back to the upstream pyinotifyproject, we'll be able, one day, to remove the ganeti version of thatcode. For now we still need it to support older distributions, buthaving a note about when we'll be able to remove it is nice....
Remove errors.ConfdFatalError
This exception is caught, but never thrown. It became useless when wemoved confd from on/off to enabled/disabled, but always running on allnodes. Removing its definition and the code catching it can do no harm.
RAPI changes for instance moves
Two new resources are added:- /2/instances/$name/prepare-export- /2/instances/$name/export
The documentation for the existing resource for creating instances is updatedfor remote imports. The RAPI client is extended for the new resources....
Implement opcode changes for remote-import
Implement opcode changes for remote-export
Add opcode to prepare export
To prepare a remote export, the X509 key and certificate need to be generated.A handshake value is also returned for an easier check whether both clustersshare the same cluster domain secret.
Conflicts: lib/luxi.py - trivial
Abstract the LUXI eom into a constant
Currently the EOM terminator is hardcoded on the server side, and iscustomizable in the Transport object (with the default being the same asthe value found in the server), but not in the luxi client.
With this patch we move the value to constants, and remove the "fake"...
KVM: vhost net acceleration support
This will only work on patched or newer (>= 2.6.34) kernels and with apatched version of qemu-kvm.
Add function to read cluster domain secret
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
LUExportInstance: Remove instance only if export was successful
Until now, the instance was always removed (if asked for by theuser). In case of export errors however, it shouldn't.
RAPI client: Handle urllib2.HTTPError and raise GanetiApiError
This allows users of the RAPI client to catch GanetiApiError for all HTTPerrors.
RAPI: /2/{nodes,instances}/$name should return 404 for unknown items
Currently they return a 500 Server Error, not really usefulfor detecting nonexistent items.
Return disk_template from LUQueryInstanceData
Inter-cluster instance moves need the disk template. As they runLUQueryInstanceData to get an instance's details, the disk templatemust be returned.
RAPI client: Rename Get{Node,Instance}Info, add new GetInstanceInfo
GetInstanceInfo should return the resource /2/instances/$name/info,but so far it returns /2/instances/$name. The same applies toGetNodeInfo, which returns /2/nodes/$name. Both names are stripped...
RAPI client: Log request to be made
Add checks for master IP in cluster verify
This also updates a comment in the unittest for utils.py. We unittestthe new function for two things: correct reporting on real case (forlocalhost), and correct reporting with a mocked-out TcpPing that returns...
RetryOnSignal: handle socket error as well
Remove unused import from daemon.py
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Luca Bigliardi <shammash@google.com>
Add a forgotten comment about overriding a method
AsyncUDPSocket.handle_error
By overriding the default asyncore handle_error (which closes thesocket) with our own version, which logs what happened but tries toproceed, we can get rid of a couple of try/except blocks. The resultingchurn is deindentation of the internal code....
utils.IgnoreSignals
Remove duplicate code between a couple of asyncore related function byhaving a function in charge of handling EINTR errors. Unittests included.
Conflicts: daemons/ganeti-noded lib/daemon.py lib/rapi/baserlib.py lib/rapi/rlib2.py lib/utils.py
Signed-off-by: Luca Bigliardi <shammash@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Fix some pylint warnings
Disable warnings for:- except Exception,- use of __errno_location,- redeclaration of handleError()
Signed-off-by: Luca Bigliardi <shammash@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Fix logging string format
Fix this pylint warning:[W6501, Mlockall] Specify string format arguments as logging function parameters
Fix Filehandler / FileHandler typo
Fix typo spotted by pylint:E1101:2095:LogFileHandler.handleError: Module 'logging' has no 'Filehandler' member