Synnefo » snf-ganeti

Rename BaseJO to BaseOpCode

Since we don't have for now a job definition object anymore, we rename
this class to BaseOpCode. It's still useful (and not merged with OpCode)
since it holds all the 'pure' logic (no custom field handling, etc.)
whereas OpCode holds opcode specific data (OP_ID handling, etc)....

Fix previous patch using workerpool in masterd

The function to stop a worker pool is TerminateWorkers(), not Shutdown().

Reviewed-by: iustinp

Use workerpool in master daemon

Reusing threads instead of starting one for each request is more efficient.

Reviewed-by: iustinp

Add forgotten RAPI constant

This was forgot on the forward-porting of RAPI.

Reviewed-by: imsnah

First version of user feedback fixes

This patch contains a raw version for fixing feedback_fn.

The new mechanism works as follows:
- instead of a per-Processor feedback_fn, there's one for each
ExecOpCode, so that feedback for different opcodes go via possibly...

Fix JobStorage._GetJobIDsUnlocked

The job ID returned must be an integer (and the regex enforces that),
but we didn't convert it manually.

Reviewed-by: imsnah

Add experimental persistency to job queue

It's not perfect and it's not finished, but it's a start.

- Serial number is read only once, but written on each update
- Jobs are kept only on disk (caching will be implemented)

Reviewed-by: iustinp

Add PID to all logs

This patch (for trunk) adds the PID to all daemon logs.

Reviewed-by: imsnah

Fix backend.NodeVolumes handling of LVM output

This is the same fix as for GetVolumeList.

I've checked manually and all other places that call lvm commands are
already checking the output validity in terms of correct number of
fields.

Reviewed-by: ultrotter

Fix backend.GetVolumeList handling of LVM output

Sometimes ‘lvs’ can spit error messages on stdout, even when one wants
to parse the output:
...
Inconsistent metadata copies found - updating to use version 2776
...

So we need to validate the output to guard against such cases....

Add generic HTTP server classes

Some of the code is adopted from the 1.2 branch
(lib/rapi/RESTHTTPServer.py). This code can be used as a base for the
various HTTP servers in Ganeti.

Reviewed-by: iustinp

Initial copy of RAPI filebase to the trunk

Reviewed-by: iustinp

Move watcher's LockFile function to utils

Reviewed-by: iustinp

Add a simple decorator for instance methods

This is just a simple, hardcoded decorator for object methods needing
synchronization on the _lock instance attribute.

Reviewed-by: ultrotter

Fix double-logging in daemons

Currently, in debug mode, both the logfile handler and the stderr
handler will log debug messages. Since the stderr is redirected to the
same logfile (to catch non-logged errors), it means log entries are
doubled.

The patch adds an extra parameter to the logger.SetupDaemon() function...

Fix utils.py unittest

Reviewed-by: imsnah

Reduce duplicate Attach() calls in bdev

Currently, the 'public' functions of bdev (FindDevice and
AttachOrAssemble) will call the Attach() method right after class
instantiation.

But the constructor itself calls this function, and therefore we have
duplicate Attach() calls (which are not cheap at all)....

Convert utils.py to the logging module

The patch also logs all commands executed from RunCmd when we are at
debug level.

Reviewed-by: imsnah

Remove old job queue code

Reviewed-by: iustinp

Make luxi RPC more flexible

- Use constants for dict entries
- Handle exceptions on server side
- Rename client function to CallMethod to match server side naming

Reviewed-by: iustinp

Add very simple job queue

Reviewed-by: iustinp

Convert LUTestDelay to concurrent usage

In order to do so:
- We set REQ_BGL to False
- We implement ExpandNames

That's it, really.

Reviewed-by: iustinp

LogicalUnit: add ExpandNames function

New concurrent LUs will need to call ExpandNames so that any names
passed in by the user are canonicalized, and can be used by hooks,
locking and other parts of the code. This was done in CheckPrereq
before, but it's now splitted out, as it's needed for locking, which in...

Add a new LockSet unittest

This test checks the LockSet behaviour when an empty list is passed.
The current behaviour is expected, but since this is a corner case,
we're safer to keep it under a check, and if we need a different one
monitor that everything is as we expect it to be....

workerpool: Don't notify if there was no task

Workers have to notify their pool if they finished a task to make
the WorkerPool.Quiesce function work. This is done in the finally:
clause to notify even in case of an exception. However, before
we notified on each run, even if there was no task, thereby creating...

Add a top level RUN_GANETI_DIR constant

This patch creates a base RUN_GANETI_DIR and then moves the other run
dir constants to use that (even if just setting BDEV_CACHE_DIR as equal
to it, rather than putting it deeper, for now).

Also we create a constant list of all the subdirs we need in RUN_DIR to...

luxi: Use serializer module instead of simplejson

Reviewed-by: iustinp

Add a missing import to cmdlib

cmdlib uses some constants from locking (ie. locking levels) but doesn't
import it. This patch fixes the issue.

Reviewed-by: iustinp

Add and remove instance/node locks

Whenever we add an instance or node to the cluster (i.e. to the config
and whenever we remove them we should add/remove locks as well). In the
future we may want to optimize this so that the configwriter does it, or
it's handled at the context level, but till we're adding/removing...

mocks: create a FakeContext object

This will be passed to FakeLUs

Reviewed-by: iustinp

Locking: remove LEVEL_CONFIG lockset

Since the ConfigWriter now handles its own locking it's not necessary to
have a specific level for the config in the Locking Manager anymore.
This patch thus removes it, and all the unittest calls that used it, or
depended on it being present....

Locking: add ssynchronized decorator

This patch creates a new decorator function ssynchronized in the locking
library, which takes as input a SharedLock, and synchronizes access to
the decorated functions using it. The usual SharedLock semantics apply,
so it's possible to call more than one synchronized function at the same...

Fix some issues with the watcher

This patch fixes two bugs:
- the state file is not saved because we use the method for checking
for udpated data
- in two places 'Error' was used instead of 'Exception', which breaks
error handling

Additionally:...

Reuse the luxi client in cli.SubmitOpCode

By a mistake, we don't reuse the luxi client. As such, we open and close
the connection at each poll cycle and spam the server logs.

Reviewed-by: ultrotter

Remove custom locking code from gnt-instance

The gnt-instance script doesn't run in the same process anymore, so we
can't and don't have to unlock.

Reviewed-by: ultrotter

ganeti-masterd: Use logging module

Reviewed-by: ultrotter, iustinp

Set locale when using docbook programs

At least docbook2man inserts a date formatted using the current
locale into its output.

Reviewed-by: iustinp

Add a FirstFree function to utils.py

This function will return the first unused integer based on a list of
used integers (e.g. [0, 1, 3] will return 2).

Reviewed-by: imsnah

Processor: acquire the BGL for LUs requiring it

If a LU required the BGL (all LUs do, right now, by default) we'll
acquire it in the Processor before starting them. For LUs that don't
we'll still acquire it, but in a shared fashion, so that they cannot run...

Add REQ_BGL LogicalUnit run requirement

When logical units have REQ_BGL set (it is currently the default) they
need to be the only ganeti operation run on the cluster, and we'll
guarantee it at the master daemon level. Currently only one thread is
running at a time, so this requirement is never broken....

Implement “gnt-job list -o +...”

This adds the same “-o +...” functionality in gnt-job as in the node and
instance scripts.

Reviewed-by: imsnah

Remove duplicate code in hooks unittests

All the tests there used to creare a cfg, a sstore, an opcode and a LU.
Put all the duplicate code in the setUp function.

Reviewed-by: iustinp

AddNode: move the initial setup to boostrap

From the master node we can't start ssh and connect to the remote node,
nor we can do it from ganeti-noded as this ssh section will possibly ask
for key confirmation and password. So the code to copy the ganeti-noded...

LUAddNode: use node-verify to check node hostname

As we can't use ssh.VerifyNodeHostname directly, we'll set up a mini
node-verify to do checking between the master and the new node. In the
future networking checks, or more nodes, can be added as well.

Reviewed-by: iustinp

LUAddNode: upload files via rpc, not scp

We used to scp all the ssconf files, and the vnc password file to the
new node. With this patch we use the upload_file rpc, specifying just
the new node as a destination. All the files previously copied by scp
are already allowed by the backend....

Change fping to TcpPing in two LUs

Two LUs are using RunCmd to call fping, in order to check for an IP
presence on the network. Substituting it with TcpPing will get rid of
it, which makes it not break in the new world order, where the master
cannot fork....

ganeti-noded: Fix handling of QuitGanetiException

- s/GanetiQuitException/QuitGanetiException/
- Look for the arguments in err.args, not err itself

Reviewed-by: iustinp

logger: Set formatter for stderr

Having a timestamp on log messages is very useful. The default
format string doesn't include a timestamp.

Reviewed-by: ultrotter

ganeti-noded: quit on QuitGanetiException

Accoring to the usage documented in the QuitGanetiException docstring,
if we receive such an exception we'll set the global _EXIT_GANETI_NODED
variable to True, and then return either a valid value or an error
message to the user. This will be the last request we serve, though,...

ganeti-noded: serve not quite forever

Rather than calling httpd.serve_forever() in ganeti-noded we'll call
httpd.handle_request() but just while a global variable, which we'll
call _EXIT_GANETI_NODED, remains false.

Reviewed-by: iustinp