Synnefo » snf-ganeti

Implement adding/removal of locks by declaration

With this patch LUs can declare locks to be added when they start and/or
removed after they finish. For now locks can only be added in the
acquired state, and removed if owned, and added locks default to be...

Fix typo in a locking.py comment

Reviewed-by: imsnah

Add GanetiLockManager.is_owned function

This is a public version of the private function we already had.
We don't just change the previous version because it had lots of users
in the library itself and in the testing code.

Reviewed-by: imsnah

jqueue: Add common RPC error handling function

We didn't decide yet what exactly it should do with failed nodes.

Reviewed-by: ultrotter

Add an atomic ConfigWrite.GetAllInstanceInfo()

In order to be able to query instance without locking them, we need the
same atomic query of multiple instances as for nodes.

Reviewed-by: ultrotter

Rewrite the 'only submit job' handling in scripts

The "sys.exit(0)" was not nice as you couldn't differentiate it from
other exit codes. We change this to a specially defined exception for
this, so that multi-opcode commands can handle this nicely.

Reviewed-by: imsnah

Add a way to export all node information at once

The patch adds a new function to export all node information at once
(i.e. atomically with respect to the configuration lock).

Reviewed-by: ultrotter

Export backend.GetMasterInfo over the rpc layer

We create a multi-node call so that querying all nodes for agreement
will be fast.

Reviewed-by: imsnah

Parallelize LUQueryInstanceData

Reviewed-by: iustinp

Parallelize LUReplaceDisks

This is the most complex parallelization so far. We have to lock one
instance (and its nodes) plus one more node if doing a remote replace,
or all nodes if doing a remote replace with iallocator.

Reviewed-by: iustinp

Processor: remove ChainOpCode

This function was incompatible with the new locking system, and its
usage has been removed from the code. For now LUs share code by calling
common module-private functions in cmdlib.py, in the future they will
use tasklets (when those will be implemented)....

LUReplaceDisks: remove use of ChainOpCode

The calls to OpActivateInstanceDisks and OpDeactivateInstanceDisks has
been replaced by _StartInstanceDisks and _SafeShutdownInstanceDisks
respectively. This is the last usage of ChainOpCode.

Reviewed-by: iustinp

Fix a typo in LogicalUnit.ExpandNames docstring

s/locking.LEVEL_INSTANCES/locking.LEVEL_INSTANCE/

Reviewed-by: iustinp

Fix LUReplaceDisks with iallocator

self._RunAllocator() sets self.op.remote_node, but doesn't return the
new remote node. If we set it to the return value of the function we
basically reset it to None, and iallocator is never run.

Reviewed-by: imsnah

Fix iallocator run

The rpc library returns a list, not a tuple, so we'll accept both.

Reviewed-by: iustinp

Parallelize LUGrowDisk

Reviewed-by: iustinp

Add primary_only flag to _LockInstancesNodes

As the name says when the flag is on (the default is off) only the
primary nodes are locked, as opposed to all of them.

Reviewed-by: iustinp

Add locking.ALL_SET constant and use it

Rather than specifying None in needed_locks every time, with a nice
comment saying to read what we mean rather than what we write, and that
None actually means All, in our magic world, we'll hide this secret
under the ALL_SET constant in the locking module, which has value, you...

Prevent mistakes using _GetWantedNodes

All the users of _GetWantedNodes have been converted to be concurrent
LUs, and thus cannot call this function with an empty list of nodes
anymore. This patch makes this restriction a part of the function
itself. This prevents mistakes in new concurrent LUs, and creates more...

Parallelize LUDiagnoseOS

Reviewed-by: iustinp

s/Chain(OpQueryExports)/rpc.call_export_list(...)/

Parallel opcodes are not (yet?) supported for chaining. Turns out
though that chaining is used only four times in the code, and twice it's
for querying exports. But what's the need to chain the full opcode, when...

Merge r1607 from branches/ganeti/ganeti-1.2

Use a default vnc_bind_address if None is specified

Reviewed-by: iustinp

merge r1548 from branches/ganeti/ganeti-1.2

Fix wrong wording of instance rename error message.

Reviewed-by: imsnah

merge r1540 from branches/ganeti/ganeti-1.2

Allow access to HVM serial console

Reviewed-by: imsnah

merge r1538 from branches/ganeti/ganeti-1.2

Check HVM device type on instance modify as well.

Reviewed-by: imsnah

Pass the force param to SetInstanceParms

It was already allowed in gnt-instance modify, but ignored.
It will be used to force skipping parameter checks.

This is a forward-port from branches/ganeti-1.2

Original-Reviewed-by: imsnah
Reviewed-by: iustinp

utils.SplitTime: Fix rounding of milliseconds

Reported by Iustin.

It used to return this:

utils.SplitTime(1234.999999999999)

(1234, 1000)

while it should've returned this:

utils.SplitTime(1234.999999999999)

(1235, 0)

Reviewed-by: ultrotter

Make WaitForJobChanges deal with long jobs

This patch alters the WaitForJobChanges luxi-RPC call to have a
configurable timeout, so that the call behaves nicely with long jobs
that have no update.

We do this by adding a timeout parameter in the RPC call, and returning...

Fix error message when masterd is not listening

Reported by Iustin.

Reviewed-by: iustinp

jqueue: Replace normal cache dict with weakref dict

A job should only exist once in memory. After the cache is cleaned,
there can still be references to a job somewhere else. If there
are multiple instances, one can get updated while a function is
waiting for changes on another instance. By using...

jqueue: Reset run_op_idx after job is done

It can be confusing otherwise.

Reviewed-by: ultrotter

Make sure that client programs get all messages

This is a large patch, but I can't figure out how to split it without
breaking stuff. The old way of getting messages by always getting the
last one didn't bring all messages to the client if they were added...

Parallelize LUQueryNodes

As for LUQueryInstances the first version just acquires a shared lock on all
nodes. In the future further optimizations are possible, as outlined by
comments in the code.

Reviewed-by: imsnah

LockSet: allow lists with duplicate values

If a list with a duplicate value is passed to a lockset what the code
now does is to try to acquire the lock twice, generating a
double-acquire exception in the SharedLock code. This is definitely an
issue. In order to solve it we can either forbit double values in a list...

LURebootInstance: move arg check in ExpandNames

The check for the reboot type can be done without any locks held, so
we'll move it to ExpandNames. Plus, we note in a FIXME that if the
reboot type is not full, we can probably just lock the primary node, and...

Use Linux-specific way to name master socket

By using this Linux-specific way we don't have to care about removing the
socket file when quitting or starting (after an unclean shutdown). For a
more detailed description, see the comment in the patch.

Reviewed-by: schreiberal

SshRunner: Add parameter to always accept peer's SSH key

This will be used to add nodes without user interaction, specifically
in QA tests.

Reviewed-by: ultrotter

SshRunner.Run: Pass all arguments to BuildCmd

This patch changes SshRunner.Run to pass all arguments to
SshRunner.BuildCmd. They had the same arguments before
and should stay that way. This change makes it easier
to add new or change existing arguments.
...

RunCmd: add optional environment overriding

If the user passes an env dict to RunCmd we'll override the environment
passed to the to-be-executed command with the values in the dict. This
allows us to pass arbitrary environment values to commands we run.
...

Add KVM hypervisor code

ht_kvm.py contains the code for ganeti to work under kvm.
This patch also modifies Makefile.am to ship that file, and
lib/hypervisor/__init__.py to import it, and add kvm to the
hypervisors map.

Reviewed-by: imsnah

Add --with-kvm-path configure option

This allows to configure a different path to the kvm binary. By default
/usr/bin/kvm is used, which is the one found in debian and ubuntu.

Reviewed-by: imsnah

Convert RunCmd to an epydoc docstring

Reviewed-by: imsnah

Fix race locking issue in noded

Noded didn't release the job queue lock after initialising it. This
patch makes sure to unlock once the work is done.

Reviewed-by: ultrotter

Add RPC call to wait for job changes

This way clients can react faster to status or message changes and
don't have to poll anymore.

Reviewed-by: ultrotter

Add functions to split time into tuple and merge it back

These will be used for job logs.

Reviewed-by: ultrotter

Don't always remove queue lock when queue is purged

The lock should only be removed if ganeti-noded is going to quit.
Otherwise it needs to be kept to prevent another process from creating
it again while we're still holding the (removed) lock. This is due to...

jqueue: Move archived jobs on all nodes

Otherwise one might have archived jobs back in the list after a master
failover.

Reviewed-by: iustinp

backend: Add function to check whether file is in queue dir

Another function will need to check whether its parameters
are job queue files.

Reviewed-by: iustinp

jstore: Change to not always require a lock

This way we can do locking when both noded and masterd are running
on the same machine, the latter holding an exclusive lock on the
queue.

Reviewed-by: iustinp

Disallow uploading job queue files through upload_file

The job queue is now updated through its own RPC functions.

Reviewed-by: iustinp