watcher: Acquire lock early and give more friendly message
By opening the lock file early, other programs can lock thestate file to prevent ganeti-watcher from restarting daemons.Using the pause feature is inherently prone to race conditions.
Before a traceback was logged when the lock file couldn't...
Make SSH_CONFIG_DIR customizable
This patch adds ability to customize ssh config directory with --with-ssh-config-dir(instead of hardcoded /etc/ssh value). This is useful in Linux distributions withcustom ssh config directories (/etc/openssh in ALTLinux, for example)....
Merge branch 'stable-2.1' into devel-2.1
Signed-off-by: Guido Trotter <ultrotter@google.com>...
Add NLD constants to Ganeti
This avoids the need for them to be injected in the nbma repository.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Make pylint happy
I was using a too old version which doesn't got all those. Thispatch is fixing the new lint errors.
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Merge remote branch 'origin/devel-2.0' into devel-2.1
Conflicts: NEWS: Trivial configure.ac: Trivial...
Fix two potentially endless loops in http library
The first can be problematic if poll(2) returns POLLHUP|POLLERR on asocket. Before it would be only be respected for SOCKOP_RECV, but sincethey can also occur on other socket operations, esp. in combination with...
Move watcher's EnsureDaemon function to utils
This is going to be used from the nbma repository, to ensure that thenld daemon is running.
Add multi-key support to the serializer
Signed-off-by: Balazs Lecz <leczb@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Adding tool for automated cluster-merger
This is the implementation of docs/design-cluster-merger.rst. It allowsthe automatic merging of one or more clusters into the invoking cluster.
While this version is tested and working it still needs some tweaking...
Fix bug in LUQueryConfigValues
LUQueryConfigValues supports multiple output fields. If the client askedfor the watcher pause status, it would not get a list, but simply thevalue.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Add watcher hooks
These hooks are run on all nodes, after the "base" daemons are started.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Abstract starting the node daemons
We're using a separate function for this, as we're going to add somefunctionality to this feature.
ganeti-watcher: remove unused Indent function
Fix typo in LUVerifyCluster when checking node time
The first argument to _ErrorIf should always be True in this case.
Add make target to generate unittest coverage report
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Add unittests for ganeti.opcodes
According to “coverage”, this covers 99% of the code.
Catch disk activation errors in watcher
If activating disks fails for some reason, the watcher didn'tcatch the exception. With this patch it's caught and logged.
Disable warning for not calling ProcessEvent init
This class doesn't need its constructor to be called.
Implement utils.RunParts and use it for hooks
This function is a generic pythonic version of runparts. We currentlyuse it in the backend HooksRunner, but we'll use it for runningdifferent directories as well.
Change backend hooks runner to use RunCmd
And save lots of lines of code, in the process
Add reset_env option to RunCmd
This allows to run a command with only the passed in environment, ratherthan just updating the default one with it.
Now with unit testing.
Make it possible to pass custom private key path to SshRunner.Run
Signed-off-by: René Nussbaumer <rn@google.com>Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Handle EAGAIN in LUXI client
If too many clients try to connect to the master at the same time, some ofthem might fail if the master doesn't accept the connections fast enough.
Show message when job is waiting in queue or for locks
Jobs submitted via the standard command line utilities didn't give anyindication that anything is happening while they were waiting in the jobqueue (e.g. due to other jobs using all worker threads) or acquiring...
Update the IAllocator documentation
This should be rewritten from a 'change document' (e.g. "Ganeti onlysupports...") to a 'current implementation document', but in themeantime we can at least update it with the multi-evac changes.
Signed-off-by: Iustin Pop <iustin@google.com>...
Switch gnt-node evacuate to the new opcode
This switches gnt-node to the new opcode, and in the process alsoenables multi-node arguments for it.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Add LUNodeEvacuationStrategy
Add a new opcode for node evacuation
We add this as a new opcode since we don't want to alter the behaviourof current opcodes/lus.
Implement support for mevac in OpTestAllocator
Implement IAllocator multi-evacuate mode
This is a new mode that request a solution for the evacuation ofmultiple nodes. The external script will be fed a list of names, and isexpected to return a list of [instance, new_node(s)] lists, detailingthe evacuation path of each instance....
Accept both 'nodes' and 'result from iallocator
This patch switches the default result key from 'nodes' to 'result'. Theold name is still accepted for backwards-compatiblity, and should beremoved in later versions.
Change internal API for the IAllocator class
Currently the 'name' parameter in the constructor is required (as anon-keyword argument). Since the (to follow) node evac IAllocator modedoesn't have 'name' as a valid argument, we're moving this one into the...
Remove redundant code in IAllocator class
This moves the setting of the request member on the in_data, of therequest type, and of the branching basef on request type outside ofindividual functions and directly into the constructor.
Since the values we're using externally are identical to the...
bootstrap: Wait for node daemon when adding new node
Until now this was only done for the master node, thoughthe problem originally fixed in 8f215968 also occurs forother node daemons.
Reset tempfile module after fork where useful
Move RunInSeparateProcess to ganeti.utils
This function could be useful in other places and thisway we can easily unittest it.
Add function to reset tempfile module after fork
On fork, the tempfile module's pseudo random generator isnot reset. If several processes (e.g. two children or parentand child) try to create a temporary file, they'll conflict.This function can be used to reset the name generator which...
Fix ssh host key checking with no-key-check
In case we add a node with “--no-ssh-key-check”, this should overrideany default yes/ask values in the system-wide (or user) ssh key check.
Currently this only works in batch mode, whereas in non-batch we only...
Simplify a bit _GetWantedNodes
This should have been done in the _ExpandNodeName patch.
Fix a wrong docstring
There's no such thing as OpProgrammerError (I found this as I wrote itin code in another place, and pylint complained).
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
ganeti-noded: Fix bug when export didn't succeed for all disks
snap_disks can contain boolean values. They weren't handled correctly.The error message was “Error while executing backend function: Invalidobject passed to FromDict: expected dict, got <type 'bool'>”....
Remove boiler-plate code about node/instance names
Currently we have lots of duplication of the error-checking (and properexception raising) around node/instance name expansion. LUCreateInstanceis the only place where we have abstracted this.
This patch creates two functions (ExpandNodeName and ExpandInstanceName)...
Merge remote branch 'origin/stable-2.1' into devel-2.1
Add unittest for utils._FingerprintFile
Release all node locks during disk replace
This patch extends commit 7ea7bcf by releasing all node locks in diskreplace for the early release mode. The rationale behind this is:
- LUCreateInstance already releases all node locks while waiting for disk synchronization, and does an instance startup later...
Unify a few re.compile calls in DRBD
These are both cleanups and, in the case of _MassageProcData, switchingfrom a weaker RE to a stronger one (we now need cs: in the line,previosuly any line starting with \d+: was accepted).
Auto-enable early release for offline old nodes
In case the old node is offline, we won't be able to talk to it toremove the storage, and in most cases the node is poweredoff/unreachable.
In this case, it makes no sense to delay the storage release, so we...
Skip line-length warnings in man
Revert "Workaround man page output for long PREFIX dirs"
This reverts commit 83d9f4366f3aa9ae360e27bfe6619402793e9eb5.
man is still unable to wrap some long lines, so we simply revert this patch(and filter out the specific message in autotools/check-man)....
Run instance hooks on more nodes
This should fix issue 68: some hooks should be run on more nodes thancurrently. GrowDisk runs on both nodes, remove run the post hook on theinstance's nodes, and failover and migrate run the post hook on thesource node too....
Add {NEW,OLD}_{PRIMARY,SECONDARY} vars to hooks
Per issue 71, the migrate and failover need special variables forkeeping the nodes consistent during instance migrations.
Workaround man page output for long PREFIX dirs
A long PREFIX variable (to configure) will result in very longLOCALSTATEDIR, which when concatenated with lib/ganeti/ (and even moreitems under it) will go over the 80 char line length we enforce in theman checker....
Convert scripts to pass options to the JobExecutor
Pass debug mode to noded for OS-related calls
Add the options attribute to cli.JobExecutor
Update scripts to pass 'opts' to SubmitOpCode
Add debug mode to burnin
There are two entry points to job execution in burnin, ExecOp andExecOrQueue, and these are modified to call the new _SetDebug method onthe opcodes.
Implement generic CLI options->opcode updates
This patch changes SubmitOpCode and SubmitOrSend such that we have asingle function that does generic CLI options to opcode attributesfunction. This will allow, once all scripts pass the opts argument toSubmitOpCode, to pass the debug parameter or the dry-run one to the LUs....
Change the debug CLI option to integer/count
This changes from boolean to integer/count (for a future differentiationbased on the actual debug level). All the uses of the code only testit's boolean status, so it still works as an integer value.
Add a generic 'debug_level' attribute to opcodes
Also automatically fix opcodes which have this missing in the LU initroutine.
Fix bug introduced in commit 413b747
While commit 413b747 fixed the issue of poll(2) returning toosoon, it didn't work when the poll(2) call should've beenblocking. This is now fixed and verified.
Fix dumpers/loaders after slots cleanup
Commit 154b958 changed (correctly) the slots usage, but this brokedumpers/loaders since we relied directly on the own class slotsfield.
To compensate, we introduce a simple function for computing the slots...
Fix locking bug causing high CPU usage
Iustin Pop noticed unusually high CPU usage with 2.1's masterdaemon, even with very simple opcodes like OP_TEST_DELAY. Asit turns out, we inadvertently passed seconds as millisecondsto a call to poll(2). Due to the way the loop around the call...
Check built manpages for errors
One fix is necessary in gnt-cluster.sgml. Also adding “DELETE_ON_ERROR”target to remove output file if an error occurred while building it (inthis case the manpage).
This was reported by Iustin Pop in issue 87 and proposed check method...
Fix confd procotol design description
The protocol design for confd was missing a description of the fourcccode which we use to distinguish between different message types, if wewant to completely change the protocol. Adding them so that someoneimplementing it can find out....
Implement instance rename QA tests
Add an early release lock/storage for disk replace
This patch adds an early_release parameter in the OpReplaceDisks andOpEvacuateNode opcodes, allowing earlier release of storage and moreimportantly of internal Ganeti locks.
The behaviour of the early release is that any locks and storage on all...
Fix "gnt-instance rename" functionality
Commit 91e0748c (Unify the “--no-ip-check” option) broke the optionsvariable name for ‘--no-ip-check’ but since we don't have a QA test forinstance rename (only burnin test), this was not caught until Issue 86was opened....
TLReplaceDisks: Delay iallocator when evacuating node
When evacuating nodes, the iallocator was run for allinstances without taking planned changes into consideration.This patch delays part of CheckPrereq and running theiallocator for node evacuation....
Update NEWS file and bump version to 2.0.6
Implement debug level across OS-related RPC calls
This doesn't implement the full functionality, we need to add the debuglevel to the opcodes too, but at least won't require changing the RPCcalls during the 2.1 series.
Second try to fix LUVerifyCluster
My previous patch, commit 785d142, fixed the case where a node is markedoffline. With this patch it'll also handle other failures correctly.
LUVerifyCluster: Fix bug with offline nodes
[…] * Other Notes - NOTICE: 1 offline node(s) found. * Hooks ResultsFailure: command execution error:iteration over non-sequence
Commit a0c9776a introduced an error simulation mode to LUVerifyCluster.Due to a small mistake, offline nodes weren't skipped when checking the...
utils: Fix retry delay calculator
Before this patch, it would always sleep for at leastthe time specified as the upper limit. Now it actuallylimits the sleep time.
ganeti-cleaner: does 'echo 0' instead of 'exit 0'
This is from Debian Bug#568105.
ganeti-cleaner in one case erroneously calls "echo 0" instead of "exit0". As ganeti-cleaner is called in the daily cronjob, this leads to adaily mail containing just "0" with an unconfigured default install....
Bump RPC protocol version to 30
Bump version to 2.1.0~rc5
Makefile.am: Targets used directly should depend on BUILT_SOURCES
Merge branch 'devel-2.0' into devel-2.1
Conflicts: man/gnt-cluster.sgml (trivial)
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Make the snapshot decision based on disk type
… instead of disk size, which is not as reliable. This actuallysimplifies the code; but it still leaves the possibility of stackoverflows if the disk data structure is corrupted.
Three small typos in man pages
The credit goes again to Lintian.
Fix missing bridge for xen instances
Xen instances nic definitions miss the target bridge.
This bug was introduced in commit 503b97a9.
Signed-off-by: Alessandro Cincaglini <alessandro.ciancaglini@gmail.com>Reviewed-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>...
Fix flipping MC flag bug
Currently unofflining or undraining an already functional mastercandidate node, can cause it to demote itself. In order to avoid that weonly trigger the self-promotion check if the node is not currently acandidate.
ganeti-watcher: ensure confd is running as well
Ganeti-confd should be running on all 2.1 nodes.
Add capability to use syslog for logging
This patch adds a configure-time parameter that will set the defaultsused by all programs, and command-line parameters in the daemons thatallow overriding it.
Syslog 'yes' enables syslog in addition to file-based logging, 'only'...
node daemon: allow working with broken queue dir
In case the queue dir cannot be create/initialized, currentlyganeti-noded exits. This means that a read-only filesystem or apermission error breaks all node daemon functionality, includingpowercycle. This is not good for the usual failure case for nodes....
utils.FileLock: handle init errors properly
If the open of the lock file fails (due to whatever reason), 'self'won't have the 'fd' attribute, and thus we fail in Close/__del__, whichwill ruin proper error reporting:
IOError: [Errno 30] Read-only file system: '/var/lib/ganeti/queue/lock'...
daemon-utils: remove usage of here-docs
In some versions of bash, here-docs and here-strings use temporaryfiles, which means daemon-util needs a writable temporary filesystem.Since echo is a bash builtin anyway, it's simple to switch to it andremove this dependency....
Fix typo in ganeti-os-interface(7)
The variable is called “IMPORT_INDEX”, not “IMPORT_IDX”.
locking: add/fix @type information
This patch missing @type information for all public methods, modifiesone to conform to the rest, and removes some information from @paramwhen it's been expressed in @type.
Fix slots definitions
According to http://docs.python.org/reference/datamodel.html#slots
Conflicts: lib/backend.py - trivial merge...
Fix the mocks.py for 2.0 unittests
The recent change to use LogWarning with multiple arguments inmpcu.py/HooksMaster broke the (simple) mock we have in the tests.
Add a crude disable for DRBD barriers
Ideally we want to/will have per-device DRBD controls of disk/metadataflushes. In the meantime, we want at least a disable of the barrierfunctionality for cases where one has battery-backed caches.
Background: DRBD has four mechanism of handling ordered disk-writes....
LURemoveNode safety in face of wrong node list
LURemoveNode runs under the BGL, which means we're guaranteed that thelist of nodes as retrieved in CheckPrereq is still valid inBuildHooksEnv. However, we can make Ganeti handle failures in case thelocking is broken (or the node list has been modified otherwise) easily,...