Synnefo » snf-ganeti

Implement checking for the master role in rapi

This patch moves the CheckMaster function from ganeti-masterd to ssconf
(most logical place, it cannot go in utils since we would have recursive
imports between ssconf and utils) and changes ganeti-rapi to also call...

Log thread name when debug output is enabled

Reviewed-by: iustinp

Fix constants typo

Reviewed-by: imsnah

Add a KillProcess function

We cannot depend on all environments to have a start-stop-daemon or
similar tool. We instead implement a KillProcess function that behaves
similar to “start-stop-daemon --retry”.

Note that the attached unittest can hang in foreground if the child...

Move ganeti-rapi core code to daemon

All other daemons have their main code in themselves and not in a module.
This patch does the same to ganeti-rapi by moving the code from
lib/rapi/RESTHTTPServer.py to daemons/ganeti-rapi.

Reviewed-by: iustinp

Implement job canceling on server side

Locking is not completeley right due to a deadlock when the job calls
UpdateJob after changing its status.

Reviewed-by: ultrotter

Add “canceled” status for opcodes

Reviewed-by: ultrotter

Convert set to a list in LUGetTags

The set triggers exception on a list-tags command and RAPI calls for tags
since it is not serializable by JSON.

Reviewed-by: iustinp

Implement job archiving on the server side

So far no error reporting to the client is done. Clients don't get
noticed if a job doesn't exist or couldn't be archived because of
its current status.

The internal cache is always cleaned when the preconditions didn't...

Move code formatting job ID into a base class

A later patch will add a memory based job storage class, hence this
code is going into a separate class. It also changes the number format
to always use at least 10 digits, allowing up to 9'999'999'999 jobs to...

Add utils.IsPidFileAlive function

This helper function reads a pid from a file containing it and checks
whether it refers to a live process.

Reviewed-by: iustinp

Generalization of bulk output mapping

Reviewed-by: iustinp

Fix logging with string job IDs

The job ID is now a string, hence logging must use %s instead of %d.

Reviewed-by: iustinp

Make job ID a string

The docstring says that _NewSerialUnlocked returns “a string
representing the job identifier”. Until now it returned an
integer and this patch changes it.

Reviewed-by: iustinp

Make the job storage init reuse a serial file

This will be needed for master failover. If we don't have a valid queue
directory, we need to reinitialize it, but we should keep the existing
serial number.

As such, we abstract the reading of the serial and if we find a valid...

Convert SetInstanceParams to concurrency

Grab a lock for the instance we're working on, and update its params.

Reviewed-by: iustinp

Convert LUConnectConsole to concurrency

For ConnectConsole we just need to lock the instance we're connecting
to. We make a few rpcs to its primary node, but node daemons can now
handle multiple queries and nodes cannot be removed till they have
instances on them anyway. Note that since we return the ssh command, and...

Convert two (simple) LUs to be concurrent

LUQueryClusterInfo and LUDumpClusterConfig can be made concurrent and
don't need to acquire any locks. In fact they don't interact with the
cluster at all, but just with its configuration, which is thread-safe by...

Put the poper RAPI baserlib

Reviewed-by: imsnah

Split RAPI resources to pieces

Reviewed-by: iustinp

Add signal handler class

This signal handler class abstracts some of the code previously
used in other places. It also uninstalls its handler when Reset()
is called or the class is destructed, thereby restoring the
previous behaviour.

Reviewed-by: iustinp

Breath life in to RAPI for trunk

Reviewed-by: imsnah

Rename BaseJO to BaseOpCode

Since we don't have for now a job definition object anymore, we rename
this class to BaseOpCode. It's still useful (and not merged with OpCode)
since it holds all the 'pure' logic (no custom field handling, etc.)
whereas OpCode holds opcode specific data (OP_ID handling, etc)....

Further fixes to enable RAPI startup

Note that since RAPI itself doesn't use luxi.Client yet, nothing works,
but at least it can startup now.

Reviewed-by: imsnah

Improve cli.SubmitOpCode

Currently, the feedback_fn argument to SubmitOpCode is no longer used.
We still need it in burnin, so we re-enable it by making the code call
that function with the msg argument in case feedback_fn is callable. The
patch also modifies burnin to accept the new argument format (msg is not...

Cache some jobs in memory

This patch adds a caching mechanisms to the JobStorage. Note that is
does not make the memory cache authoritative.

The algorithm is:
- all jobs loaded from disks are entered in the cache
- all new jobs are entered in the cache...

Change JobStorage to work with ids not filenames

Currently some of the functions in JobStorage work with filenames (which
is an implementation detail and should only be used when dealing with
the storage) and not with job IDs. We need to change this in order to...

Convert backend.py to the logging module

The patch also switches some of the exception logs to use
logging.exception (and therefore the log message will have a diferent
format).

(Note that this might not be a good choice in all cases, though)

Reviewed-by: imsnah

Fix backend.NodeVolumes handling of LVM output

This is the same fix as for GetVolumeList.

I've checked manually and all other places that call lvm commands are
already checking the output validity in terms of correct number of
fields.

Reviewed-by: ultrotter

Add generic HTTP server classes

Some of the code is adopted from the 1.2 branch
(lib/rapi/RESTHTTPServer.py). This code can be used as a base for the
various HTTP servers in Ganeti.

Reviewed-by: iustinp

Initial copy of RAPI filebase to the trunk

Reviewed-by: iustinp

Switch _QueuedOpCode to have their own lock

Right now, the queued opcode doesn't have a lock, and instead relies on
the parent QueuedJob's lock.

This is not good for logging feedback, so it's better to have a lock for
each queuedopcode.

Reviewed-by: ultrotter

jqueue: Log more information when running opcodes

Reviewed-by: iustinp

Move the master socket in the ganeti run dir

... as it was intended from the beggining, but by mistake left in the
top run dir.

Reviewed-by: ultrotter

Convert bdev.py to the logging module

This does not enhance in any way the messages; it just switches to the
new module.

Reviewed-by: imsnah

Remove the old locking functions

This removes (hopefully) all traces of the old locking functions and
uses.

Reviewed-by: imsnah

Change masterd/client RPC protocol

- Introduce abstraction class on client side
- Use constants for method names
- Adopt legacy function SubmitOpCode to use it

Reviewed-by: iustinp

Add very simple job queue

Reviewed-by: iustinp

Processor: Acquire locks before executing an LU

If we're running in a "new style" LU we may need some locks, as required
by the ExpandNames function, to be able to run. We'll walk up the lock
levels present in the needed_locks dictionary and acquire them, then run...

Processor: Move LU execution to its own method

This makes the try...finally code simplier, and helps adding a more
complex locking structure before the actual execution. It also fixes a
concurrency bug caused by the fact that write_count was read before
acquiring the BGL, and thus spurious config update hooks run could have...

workerpool: Don't notify if there was no task

Workers have to notify their pool if they finished a task to make
the WorkerPool.Quiesce function work. This is done in the finally:
clause to notify even in case of an exception. However, before
we notified on each run, even if there was no task, thereby creating...

symlinks: Add DISK_LINKS_DIR constant

The DISK_LINKS_DIR points to the RUN_DIR/ganeti/instance-disks
directory, which will contain symlinks to the instances' disks. These
provide a stable name accross all nodes for them, and permit
live-migration to happen....

serializer.DumpJson: Control indentation by parameter

If the simplejson module supports indentation, it's always used. There
are cases where we might not want to use it or enable it only for
debugging purposes, such as in RPC.

Reviewed-by: iustinp

Fix an error accessing the cfg

Since the context is passed to LogicalUnit, rather than the cfg, we can
only access the cfg as self.cfg, self.context.cfg, or context.cfg (in
the constructor). cfg is not valid anymore.

Reviewed-by: iustinp

Pass context to LUs

Rather than passing a ConfigWriter to the LUs we'll pass the whole
context, from which a ConfigWriter can be extracted, but we can also
access the GanetiLockManager. This also fixes the places where a FakeLU
is created.

Reviewed-by: iustinp

Locking: remove LEVEL_CONFIG lockset

Since the ConfigWriter now handles its own locking it's not necessary to
have a specific level for the config in the Locking Manager anymore.
This patch thus removes it, and all the unittest calls that used it, or
depended on it being present....

Locking: add ssynchronized decorator

This patch creates a new decorator function ssynchronized in the locking
library, which takes as input a SharedLock, and synchronizes access to
the decorated functions using it. The usual SharedLock semantics apply,
so it's possible to call more than one synchronized function at the same...

Add generic worker pool implementation

Reviewed-by: ultrotter

Add custom logging setup for daemons

It's better for daemons if:
- they log only to one log file
- the log level is included
- for debug runs, the filename/line number is included

This patch moves the custom formatter from the watcher to the logging...

Add a FirstFree function to utils.py

This function will return the first unused integer based on a list of
used integers (e.g. [0, 1, 3] will return 2).

Reviewed-by: imsnah