import-export daemon: Allow changing compression method
For example, exports on the same node shouldn't be compressed.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Merge branch 'devel-2.1'
Make ConfdInotifyEventHandler a library function
Cut&Paste, plus the following changes: - The class is renamed to SingleFileEventHandler - The monitored filename must be passed in and doesn't default to the ganeti cluster config file - A small docstring is added to the class...
Remove errors.ConfdFatalError
This exception is caught, but never thrown. It became useless when wemoved confd from on/off to enabled/disabled, but always running on allnodes. Removing its definition and the code catching it can do no harm.
Signed-off-by: Guido Trotter <ultrotter@google.com>...
Conflicts: lib/luxi.py - trivial
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
ganeti-cleaner: Remove expired X509 certs
Importing/exporting an instance to a remote machine creates X509certificates which expire after some time. They need to be removed fromthe nodes as they become useless.
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
Abstract the LUXI eom into a constant
Currently the EOM terminator is hardcoded on the server side, and iscustomizable in the Transport object (with the default being the same asthe value found in the server), but not in the luxi client.
With this patch we move the value to constants, and remove the "fake"...
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
ganeti-cleaner: Write log file with removed files
Logfiles can be useful for debugging.
ganeti-masterd: Improve error logging for client requests
Ganeti errors should also be logged with a backtrace.
ganeti-noded: add the --no-mlock option
While mlock on noded is definitely good in most situations, there aresome - namely my laptop - where it has no benefit, and uses preciousnon-swappable memory. To avoid this we make it optional, with a new--no-mlock option. Note that only the main node daemon and its http...
Conflicts: daemons/ganeti-noded lib/daemon.py lib/rapi/baserlib.py lib/rapi/rlib2.py lib/utils.py
Signed-off-by: Luca Bigliardi <shammash@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Use console_logging in node daemon
Node daemon logs will be printed on system console if writing on the log filefails.
Lock node daemon children in memory
Lock node daemon in memory
Fix pylint 0.20.0 warnings
These seem to be wrong, possibly a regression in pylint.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Retry connection in import-export daemon
Until now, exactly one attempt was made to establish a connection.If it failed, the whole import/export for the disk in questionaborted. Retrying will make it more reliable.
Unfortunately the listening part can't be made completely...
Use new class for wakeup file descriptor in import/export daemon
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Move some code into separate class in import/export daemon
Conflicts: lib/utils.py: Trivial
masterd: Log PID, UID and GID of connected client
This can be very useful if client programs run as non-root.
Remove two unused RPC functions
Both of these functions, “snapshot_export” and “instance_os_import”,have been replaced by the instance import/export daemon.
Add RPC call to send SIGTERM to import/export daemon
This will be used to stop the daemon without doing complete cleanup (yet).
Rename import/export RPC calls to match others
Add RPC calls to import and export instance data
These RPC calls can be used to start, monitor and stop the instance dataimport/export daemon.
Add daemon for instance import and export
This backend daemon for instance import and export will be used totransfer instance data to other machines. It is implemented in a genericway to support different ways of data input and output. The third-partyprogram “socat”, which is already used by the KVM hypervisor abstraction,...
Fix new pylint errors
Make watcher request the max coverage
Since the actions are potentially destructive, we should try to get aconsistent view of the cluster, so it's better to get the most coveragepossible.
Watcher: automatic shutdown of orphan resources
This patch changes the watcher so that it maintains (on all nodes) thelist of instances and DRBD devices by shutting down ones that confddaemons indicate should not be running on this node.
Signed-off-by: Iustin Pop <iustin@google.com>...
Merge remote branch 'devel-2.1'
Send "501 Not Implemented" back when method not found
Before this was "400 Bad Request" and thus it didn't reflectthe reality.
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Watcher: do not warn for missing hooks dir
If the hooks dir does not exist, do not warn needlessly. This is similarto commit a9b7e346 (for backend.py).
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Watcher: fix some doc typos
ganeti-confd: Call pyinotify flags correctly
The "apparently pylint was right" commit.
Although the pyinotify constants work on old distributions, they fail onnew ones, with new python. Fixing this by calling them in a way thatworks everywhere.
Some epydoc fixes
Add a hint to masterd for inconsistent clusters
Add RPC calls to create and remove X509 certificates
Certificates and keys generated using these functions will be used forinter-cluster instance moves. As per design, the private key should neverleave the node.
Merge remote branch 'origin/devel-2.1'
Conflicts: lib/bootstrap.py: Trivial lib/constants.py: Trivial
Rightname confd's HMAC key
Currently, the ganeti-confd's HMAC key is called “cluster HMAC key” orsimply “HMAC key” everywhere. With the implementation of inter-clusterinstance moves, another HMAC key will be introduced for signing criticaldata. They can not be the same, so this patch clarifies the purpose of the...
daemon-util: Generate daemon path in separate function
daemon-util: Use “return” instead of “exit” in all functions
This is important if they're called directly within daemon-util.
daemon-util: Add function to start and stop all daemons
This is, to some degree, duplicated code from the init script. However,the init script has to conform to standards of the underlying Linuxdistributions, while these functions will be called by Ganeti itself. By...
ganeti.initd: Move all daemon names from init script to daemon-util
The list of daemon names will be used in daemon-util, too.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
ganeti.initd: Move code checking daemon exit code to daemon-util
This is again for re-using code.
ganeti.initd: Move code checking config to daemon-util
This allows for more code re-use. daemon-util will also be used to startall daemons.
daemon-util: Require dashes in commands
Even though the script uses underscores (_) internally, the externalcommands are supposed to be written using dashes (-).
http.auth: Fix bug with checking hashed passwords
When username and password were sent for a resource not requiringauthentication, it wouldn't be accepted if the user in question had ahashed password. The reason was that the function GetAuthRealm used to...
Switch from os.path.join to utils.PathJoin
This passes a full burnin with lots of instances, and should be safe aswe mostly to join a known root (various constants) to a run-timevariable.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
watcher: Acquire lock early and give more friendly message
By opening the lock file early, other programs can lock thestate file to prevent ganeti-watcher from restarting daemons.Using the pause feature is inherently prone to race conditions.
Before a traceback was logged when the lock file couldn't...
Merge remote branch 'origin/devel-2.0' into devel-2.1
Conflicts: NEWS: Trivial configure.ac: Trivial...
Move watcher's EnsureDaemon function to utils
This is going to be used from the nbma repository, to ensure that thenld daemon is running.
Add watcher hooks
These hooks are run on all nodes, after the "base" daemons are started.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Abstract starting the node daemons
We're using a separate function for this, as we're going to add somefunctionality to this feature.
ganeti-watcher: remove unused Indent function
Catch disk activation errors in watcher
If activating disks fails for some reason, the watcher didn'tcatch the exception. With this patch it's caught and logged.
Disable warning for not calling ProcessEvent init
This class doesn't need its constructor to be called.
Move RunInSeparateProcess to ganeti.utils
This function could be useful in other places and thisway we can easily unittest it.
ganeti-noded: Fix bug when export didn't succeed for all disks
snap_disks can contain boolean values. They weren't handled correctly.The error message was “Error while executing backend function: Invalidobject passed to FromDict: expected dict, got <type 'bool'>”....
Merge branch 'stable-2.1' into devel-2.1
Implement debug level across OS-related RPC calls
This doesn't implement the full functionality, we need to add the debuglevel to the opcodes too, but at least won't require changing the RPCcalls during the 2.1 series.
ganeti-cleaner: does 'echo 0' instead of 'exit 0'
This is from Debian Bug#568105.
ganeti-cleaner in one case erroneously calls "echo 0" instead of "exit0". As ganeti-cleaner is called in the daily cronjob, this leads to adaily mail containing just "0" with an unconfigured default install....
Merge remote branch 'origin/stable-2.1' into devel-2.1
ganeti-watcher: ensure confd is running as well
Ganeti-confd should be running on all 2.1 nodes.
node daemon: allow working with broken queue dir
In case the queue dir cannot be create/initialized, currentlyganeti-noded exits. This means that a read-only filesystem or apermission error breaks all node daemon functionality, includingpowercycle. This is not good for the usual failure case for nodes....
daemon-utils: remove usage of here-docs
In some versions of bash, here-docs and here-strings use temporaryfiles, which means daemon-util needs a writable temporary filesystem.Since echo is a bash builtin anyway, it's simple to switch to it andremove this dependency....
Fix lint warnings after my last changes
Remove http.HttpJsonConverter
With the move of the content-type handling to the various users of the HTTPlayer, this class isn't really useful anymore.
http.server: No longer enfore JSON encoding for body
The HTTP layer shouldn't care about the contents of the request data orresponses. This requires further changes in the RAPI code to handle clientrequests correctly.
ganeti-rapi: Remove serial number/ETag code
It was never used as intended.
Factorize LUXI parsing and handling code
Also fix a typo in http/__init__.py and add unittestsfor the LUXI parsing and formatting functions.
daemon-util: Fix quoting issue
This patch fixes a quoting issue in daemon-util:
The reason was that the generated variables were not quoted properly and...
workerpool: Make worker ID alphanumeric
Having a proper name instead of just a number makes debuggingeasier.
Introduce a Luxi call for GetTags
This changes from submitting jobs to get the tags (in cli scripts) toqueries, which (since the tags query is a cheap one) should be muchfaster.
The tags queries are already done without locks (in the generic querypaths for instances/nodes/cluster), so this shouldn't break tags query...
Fix unused imports or add silences where needed
In some cases pylint doesn't parse the import correctly, so we addsilences; but there are also many cases of unused imports, which wesimply remove.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Olivier Tharan <olive@google.com>
confd: add a TODO and a pylint disable
Two variables are used in a strange way in ExecConfd. Until that isclarified, add a TODO and a silence for the unused variable warning.
Further pylint disables, mostly for Unused args
Many of our functions have to follow a given API, and thus we have tokeep a given signature, but pylint doesn't understand this. Therefore,we silence this warning.
The patch does a few other cleanups.
daemons: handle arguments correctly and uniformly
Of all daemons, only rapi did abort when given argument. None of ourdaemons use any arguments, but they accepted them blindly. This is avery bad experience for the user.
This patch adds checking and exiting in all daemons, in a uniform way....
Remove more unused variables
This removes unused variables in the rest of the code (outside lib/).
Add targeted pylint disables
This patch should have only:
- pylint disables- docstring changes- whitespace changes
Rename an ConfdInotifyEventHandler init argument
'file' is a builtin keyword/type. Like many others, it should not beused as a variable/argument name.
No code is actually passing in this argument so renaming it is simple.
Fix use of the logging functions
The logging functions expand the arguments themselves, thus it's saferto let them do it rather than manual string formatting.
Also re-wraps one comment.
Change pyinotify import for broader compatibility
On some distributions pyinotify is installed in a different way, and theactual module just contains an internal pyinotify entry, which is theactual library. On others the main pyinotify module contains the library...
daemon.GenericMain: Don't use dict for SSL paths, improve CLI options
Pass SSL certificate and key paths from ganeti-* instead of using a dict. Thepatch also improves the --ssl-{key,cert} options by giving the default in--help output and changes the validation a bit....
daemon.GenericMain: Don't use list of multithreaded daemons
Passing it in as a parameter seems more logical.
Remove quotes from CommaJoin and convert to it
This patch removes the quotes from CommaJoin and converts most of thecallers (that I could find) to it. Since CommaJoin does str(i) for i inparam, we can remove these, thus simplifying slightly a few calls....
Use “daemon-util” to reload SSH keys
Processor: support a unique execution id
When the processor is executing a job, it can export the execution id toits callers. This is not supported for Queries, as they're not executedin a job.
config.Add{Node,Instance}: get the ec id
This is ok because adding a node or instance cannot happen in a query.
We get the ec id from the LU and pass it to _EnsureUUID, which willthen for now not use it.
Add new “daemon-util” script to start/stop Ganeti daemons
Until now, Ganeti started and stopped its own daemons using custom functions.To start, the daemon was just executed and then sent the appropriate signals tostop it again. Init scripts would have to pay attention to the PID file and...