Bash completion: Simplify option completion

This makes the output file quite a bit smaller and is needed
in preparation for non-static option completion (e.g. list of
nodes).

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Olivier Tharan <olive@google.com>

Bash completion: Support for --foo=bar option format

That is with the equal sign.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Bash completion: Small optimizations

$2 doesn't contain the correct value when completing something like
"--disk-template=…". Getting it via COMP_WORDS is better.

Short options (e.g. -I) can't have an equal sign.

Also add useful debugging commands for development.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Olivier Tharan <olive@google.com>

Fix QA test for “gnt-node evacuate”

Before the arguments weren't valid.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Confd: don't fail if the config doesn't load

Rather than quitting we'll just continue to poll the config at a slow
rate, hoping that sooner or later we'll get it back. This allows also
working on non-MC nodes, and smoothly transitioning from MC to non-MC,
other than dealing with a few temporary breakages.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

confd: start in polling mode

This allows us not to enable the inotify handler immediately, and thus
to make it easier for us should the config file not exist at all.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

ConfdProcessor: add disabled state

This is a state the processor will get in, if it fails to load the
config.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Move SimpleConfigReader creation to ConfdProcessor

This will be useful to make ConfdProcessor aware of a config failure,
without quitting confd.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

confd: s/confd_event_handler/inotify_handler/

In a case we don't encounter frequently (file modified but not
overwritten) the notify handler we use is called with a wrong name.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Implement timers in confd

Timers are used both for checking for inotify failures, and for polling,
should inotify notices become too frequent.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

constants: add confd reload and rate limit times

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

ConfdInotifyEventHandler.enable: use InotifyError

Rather than raising ConfdFatalError directly
ConfdInotifyEventHandler.enable raises InotifyError should it not be
able to configure inotify, allowing the caller to decide what to do.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Add errors.InotifyError

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

ConfdInotifyEventHandler, move to a callback

ConfdInotifyEventHandler used to reload the config whenever a
notification arrived. Moving to a callback system, so that
ConfdConfigurationReloader can be responsible for that functionality.

Additionally the inotify class no longer reenables itself automatically,
but just notifies the callback if it's been disabled, and waits for its
call to enable(). This allows, should ConfdConfigurationReloader decide
it wants to move to polling, to avoid having a double enable()/disable()
call.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Move creation of inotify handler to a new class

This class will be responsible for managing inotify notifications,
timers, and rate-limiting reloads. For now none of these features is
implemented. :)

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

ConfdInotifyEventHandler: add enable/disable

Make possible to enable and disable the inotify event handler.  The
inotify handler will remain enabled, unless explicitely told to disable
itself.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

SimpleConfigReader: handle stat() errors

Also, catch EnvironmentError, rather than IOError, when trying to read
the file.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Encode the actual exception raised by LU execution

Currently, the actual exception raised during an LU execution (one of
OpPrereqError, OpExecError, HooksError, etc.) is lost because the
jqueue.py code simply sets that to a str(err), and the code in cli.py
simply passes that string to OpExecError.

This patch moves to encoding the errors as per errors.EncodeError and
changes the cli code to parse and raise that (if possible).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Move the luxi error handling into errors.py

Currently the luxi error handling is hardcoded as special encoding on
the masterd-side and special decoding on the client side. This patch
moves it to errors.py such that other parts of the code can reuse the
same encoding.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Remove watcher pause file 1h past end time

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

ganeti-watcher: Don't run if paused

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

gnt-cluster: Add CLI for watcher pause

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Add file to pause watcher for a certain duration

This can be used during maintenance work.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

cli._Argument: Remove unused, optional “suggest” parameter

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

bash completion: Allow arguments with min < max arguments

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

ganeti-masterd: Master voting in separate process

One shouldn't fork a Python process after using threads. Master
voting is done before forking (utils.Daemonize), but it also uses
threads. Hence it's now called from a separate process.

This patch also fixes the check function to actually exit if
the current machine is not the master (introduced in 04ccf5e9).

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

ganeti-masterd: Add helper to run function in separate process

This will be used to do the master voting.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

GenerateSecret: specify the length of the secret

Add a parameter to specify the length of the bytes sequence represented by the
returned hex string. Defaulting to 20 so it's compatible with previous SHA1
hash.

Signed-off-by: Luca Bigliardi <shammash@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

cmdlib: Add some messages to LUExportInstance

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Style fixes for ganeti-*

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

ganeti-noded: Close listening socket in child

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Document cluster destroy hook

Add documentation about new cluster-destroy-post hook.

Signed-off-by: Luca Bigliardi <shammash@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

LUDestroyCluster: run hooks

Run post hooks on master node before it's removed.

Signed-off-by: Luca Bigliardi <shammash@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

LUDestroyCluster: hooks support

We're going to call hooks from its Exec method so LUDestroyCluster has to
support them.

Signed-off-by: Luca Bigliardi <shammash@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Run unittests in a temporary directory

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Fix building in a different directory

Also add build-bash-completion to EXTRA_DIST.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Add script to run build commands in temporary directory

Python always compiles imported modules. By running these targets
in a temporary directory we don't pollute the source tree.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Makefile: Use BUILT_SOURCES

These are always built before anything else.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Simplify manpage building

This eliminates one temporary directory in the process.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Makefile: Fix docbook2man error message

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Handle more errors in bash completion

Don't print error messages if cluster hasn't been initialized yet.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

HooksMaster: fix RunPhase logging

In case of complete failure results is empty, return immediately
(tnx unittests).

Signed-off-by: Luca Bigliardi <shammash@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

LURemoveNode: no logs running post on removed node

Do not log results from hooks ran on the removed node as now it's done by
RunPhase.

Signed-off-by: Luca Bigliardi <shammash@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

HooksMaster: logging hooks in RunPhase

Extend RunPhase so it will log hooks results in POST phase.

Signed-off-by: Luca Bigliardi <shammash@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Document post cluster initialization LU

Update hooks documentation with the new opcode (not deleting OP_INIT_CLUSTER
from obsolete operations because the name is different).

Signed-off-by: Luca Bigliardi <shammash@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

node-remove post on removed node

Run post phase of node-remove on the removed node as well.

Signed-off-by: Luca Bigliardi <shammash@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

HooksMaster: document raised exception

HooksAbort is raised, but not documented.

Signed-off-by: Luca Bigliardi <shammash@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

HooksMaster: list of nodes override

Allow the caller of HooksMaster.RunPhase() to specify an alternative list of
nodes.

Signed-off-by: Luca Bigliardi <shammash@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Add burnin support for instance moves

This patch adds support for instance moves in burnin. This means that
non-drbd instances finally get support for being moved.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Fix error output in LUSetClusterParams

Before:
  gnt-cluster modify --enabled-hypervisors=xen-hvm,xem-pvm
  Failure: command execution error:
  Enabled hypervisors contains invalid entries: set([u'xem-pvm'])

After:
  gnt-cluster modify --enabled-hypervisors=xen-hvm,xem-pvm
  Failure: command execution error:
  Enabled hypervisors contains invalid entries: 'xem-pvm'

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Add ArgHost class

It expands to the hostnames known by bash.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Re-introduce constants for most often used arguments

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Use script to generate bash completion

Completion for tools/burnin is not yet implemented. It needs to be
converted to definition-based options handling first.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Add tools directory to constants

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Convert “burnin” to definition-based options

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Add more auto-completion metadata

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Make all options use cli_option

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Change scripts to use new argument definitions

This can be used to generate the bash completion script automatically.
In the future it may allow for better command line validation as well.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

cli: Add new classes for argument definitions

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Add gnt-instance move

This uses the new LUMoveInstance. It supports only single-instance moves.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Add OPMoveInstance and LUMoveInstance

This patch adds a basic version of LUMoveInstance. It doesn't yet
support iallocator-mode and it's implemented in old-style (non-TL) mode.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Add arbitrary node support to create/remove disks

Currently the cmdlib _CreateDisks and _RemoveDisks always work on the
instance's (current/primary) nodes. This patch extends them to take an
additional target_node parameter denoting that the creation/removal
should ignore the current instance nodes and instead operate on the
given node.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Add disk copy support at backend and the rpc level

This uses a simple 'dd if=… | ssh $target dd of=…' method, like the
ExportSnapshot (which uses the OS export; here we want full disk-level
copy and not any FS-level changes).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Remove extra argument from HooksMaster class

The mcpu.py:HooksMaster class needs to have a proc attribute/argument to
init in ordet to call its LogWarning method. However, this is available
from the 'lu' attribute, so we can remove this dependency.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

cmdlib: Fix broken QueryInstanceData for plain instances

Fixes this bug:
Failure: command execution error:
Can't compute disk status for instX.domain.tld: No error information

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Reinsert simple timers in Mainloop

This time we use the standard python sched module, rather than doing it
all by ourselves. The scheduler in mainloop can be manipulated directly
by callers, to enter new events.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Convert ganeti-confd to Mainloop

Now that mainloop is asyncore-enabled we can easily do that.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Convert the http server/mainloop to asyncore

We can avoid most of the Mainloop.Run() code if we use asyncore
for delivering I/O events, and just concentrate on what's missing in
asyncore: singnal handling and timers. This way confd can be ported to
use Mainloop as well.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

design-2.1: specify hmac key permissions

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

design-2.1: clarify confd usage of serial numbers

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Convert daemon.Mainloop to @SignalHandled

This makes the function a lot simpler, since it used to install two
different signal handlers, which are now transparently handled by the
decorator. The code is unindented, but remains unchanged apart from the
part that checks the signal handlers (which is now collapsed toghether).

The option of not installing signal handlers, which was unused, is now
removed.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Convert ganeti-masterd to @utils.SignalHandled

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Add a new SignalHandled decorator

This decorator can be used to call a function holding with protection
from a given signal. The function should accept a dict of
utils.SignalHandler, indexed by signal number, and make use of it.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

SignalHandler, only accept list of signals

utils.SignalHandler can accept either a list of signals or a single
signal, but it's always called with a list. Removing the single signal
option.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Simplify IO waiting in Mainloop

IO is never unregistered in our current usage, so dropping that
functionality for now. Also putting the poller outside of the Run()
function allows us to avoid the double step of adding tuples to an
_io_wait_add queue and adding them later in the main loop.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Remove timers from Mainloop

Timers are currently unused, slightly buggy (for example timeout doesn't
get updated correctly), and unneeded in the current form. Confd will
have timeouts in more simple fashion, and is not a mainloop user
anyway.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

QA: call ganeti-confd --version

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

QA: remove the --default-hypervisor option

This option is gone since 1ac0f5e6d2f06a5bf12815e1b92d1338af32833c,
fixing the QA suite to deal with this.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

rapi: Add /2/instances/[instance_name]/info resource

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

cli: Merge ikv_option and keyval_option into cli_option

Merging them gets rid of two public functions in cli.py and optparse explicitly
supports more than one option type per class.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

burnin: Don't pass any disk when replacing 2nd node

Replacing the secondary node always replaces it for all disks. Commit
2945fd2d added an error message for this case.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Add support for querying the ctime/mtime

This patch adds querying of ctime/mtime for the cluster/nodes/instances.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Add ctime/mtime support to the main ConfigObjects

This patch adds ctime/mtime support to the “main” config objects - the
config data itself, and the cluster/nodes/instances objects.

These are not added on auto-upgrade, but rather should be migrated if it
makes sense when 2.0 is upgraded to 2.1.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Add utils.FormatTime and a simple unittest

We don't format with subsecond-precision.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

cli: Use ToStdout/ToStderr instead of print

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

rapi: Add /2/nodes/[node_name]/storage/repair resource

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Implement “gnt-node repair-volume”

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Add opcode to repair storage volumes

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Add RPC call for storage operations

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

storage: Add operation for “vgreduce --removemissing”

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

storage: Add new function to execute operations

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

cmdlib: Add function to list all instances on node

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

cmdlib: Use common function to list instances on node

This will be used by another new function, too.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

cmdlib: Make TLReplaceDisks._FindFaultyDisks global

It'll be used for repairing storage units.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

gnt-node: Use helper function to convert storage type

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Remove obsolete ConfigObject.__setitem__

__setitem__ is used to emulate container objects. We don't use this, and
the method is not used in a couple of normal operations (add/remove
instance, add/remove disk).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Implement instance recreate-disks

This can be used for a 'plain' type instance when the underlying storage
went away, to recreate the storage (and reinstall) instead of removing
the instance and readding it.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

TLReplaceDisks: Refuse changing secondary for specific disks

We move all disks anyway.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

rapi: Add /2/instances/[instance_name]/replace-disks resource

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

rlib2: Remove leftover DOC_URI constant

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>