Synnefo » snf-ganeti

Fix a misuse of exc_info in logging.info

This is my fault, sorry.

Reviewed-by: imsnah

Fix some errors detected by pylint

Reviewed-by: imsnah

Simplify the log constants and add another one

The patch changes the log constants by moving the slash to the end of
the log dir instead of at the beginning of each log file name.

It also adds a new LOG_COMMANDS constant (to be used in a next patch).
...

Parallelize {Startup,Shutdown,Reboot}Instance

Reviewed-by: iustinp

LogicalUnit._LockInstancesNodes helper function

This function is used to lock instances' primary and secondary nodes
after locking instances themselves.

Reviewed-by: iustinp

Add LogicalUnit.DeclareLocks

This additional LogicalUnit function is optional to implement, but lets
you change your locking needs for one level just before locking it, but
after the previous levels have been already locked. It is useful for
example to calculate what nodes to lock after locking an instance....

Rewrite job queue

We found several issues in the old job queue implementation. It had race
conditions, deadlocks and other deficiencies.

Short summary:
- _QueuedOpCode and _QueuedJob are now more or less data structures with a few
utility functions. __Setup is gone....

Rework master startup/shutdown/failover

This (big) patch reworks the master startup/shutdown and the fixes the
master failover.

What does the patch do?

For master start/stop:
- remove the old ganeti-master script and its associated man page
- moves the ip start/stop directly into the backend.(Start|Stop)Master...

Implement checking for the master role in rapi

This patch moves the CheckMaster function from ganeti-masterd to ssconf
(most logical place, it cannot go in utils since we would have recursive
imports between ssconf and utils) and changes ganeti-rapi to also call...

Log thread name when debug output is enabled

Reviewed-by: iustinp

Fix constants typo

Reviewed-by: imsnah

Add a KillProcess function

We cannot depend on all environments to have a start-stop-daemon or
similar tool. We instead implement a KillProcess function that behaves
similar to “start-stop-daemon --retry”.

Note that the attached unittest can hang in foreground if the child...

Make the rapi daemon create a pidfile

This is needed for controlling it cleanly with start-stop daemon.

Reviewed-by: ultrotter

Implement signal handling in ganeti-rapi

Reviewed-by: iustinp

Replace httperror module with ganeti.http

The generic HTTP server doesn't know about httperror based exceptions
and would treat them as unknown exceptions, thereby not doing the right
thing with HTTP errors.

Reviewed-by: iustinp

Implement job canceling on server side

Locking is not completeley right due to a deadlock when the job calls
UpdateJob after changing its status.

Reviewed-by: ultrotter

Add “canceled” status for opcodes

Reviewed-by: ultrotter

Move code extracting job ID into function

It might come in handy at some point and makes the code a bit easier
to read.

Reviewed-by: iustinp

Switch RAPI to ganeti.http module

Reviewed-by: imsnah

Implement job archiving on the server side

So far no error reporting to the client is done. Clients don't get
noticed if a job doesn't exist or couldn't be archived because of
its current status.

The internal cache is always cleaned when the preconditions didn't...

Fix RPC parameters for {Cancel,Archive}Job

They aren't be tuples on the client side.

Reviewed-by: iustinp

Move code formatting job ID into a base class

A later patch will add a memory based job storage class, hence this
code is going into a separate class. It also changes the number format
to always use at least 10 digits, allowing up to 9'999'999'999 jobs to...

ganeti-masterd: write and remove pidfile

Reviewed-by: iustinp

Add utils.{Write,Remove}PidFile

WritePidFile is a helper function that writes the current pid in a
pidfile within the ganeti run directory. RemovePidFile tries to delete
it.

Reviewed-by: iustinp

Invert nodes/instances locking order

An implementation mistake from the original design caused nodes to be
locked before instances, rather than after. This patch inverts the level
numbering, changing also the relevant unittests and the recursive
locking function starting point....

Rename JobStorage to DiskJobStorage

Reviewed-by: iustinp

Fix logging with string job IDs

The job ID is now a string, hence logging must use %s instead of %d.

Reviewed-by: iustinp

Make job ID a string

The docstring says that _NewSerialUnlocked returns “a string
representing the job identifier”. Until now it returned an
integer and this patch changes it.

Reviewed-by: iustinp

Make the job storage init reuse a serial file

This will be needed for master failover. If we don't have a valid queue
directory, we need to reinitialize it, but we should keep the existing
serial number.

As such, we abstract the reading of the serial and if we find a valid...

Convert SetInstanceParams to concurrency

Grab a lock for the instance we're working on, and update its params.

Reviewed-by: iustinp

Convert LUConnectConsole to concurrency

For ConnectConsole we just need to lock the instance we're connecting
to. We make a few rpcs to its primary node, but node daemons can now
handle multiple queries and nodes cannot be removed till they have
instances on them anyway. Note that since we return the ssh command, and...

Convert two (simple) LUs to be concurrent

LUQueryClusterInfo and LUDumpClusterConfig can be made concurrent and
don't need to acquire any locks. In fact they don't interact with the
cluster at all, but just with its configuration, which is thread-safe by...

Put the poper RAPI baserlib

Reviewed-by: imsnah

Split RAPI resources to pieces

Reviewed-by: iustinp

Handle signals in node daemon

This also fixes a TODO added by ultrotter by killing the parent
process when QuitGanetiException is raised.

Reviewed-by: ultrotter

Add signal handler class

This signal handler class abstracts some of the code previously
used in other places. It also uninstalls its handler when Reset()
is called or the class is destructed, thereby restoring the
previous behaviour.

Reviewed-by: iustinp

Breath life in to RAPI for trunk

Reviewed-by: imsnah

Documentation updates

Reviewed-by: imsnah

Add apidoc makefile target

The patch adds the apidoc target and the epydoc config file for it. Note
that this is for epydoc 3.0 and that it will put the docs into
./doc/api/.

The patch also adds a new .gitignore rule for the auto-generated rapi
fragment....

Sort the job list in _GetJobIDsUnlocked

Since the IDs are integers, we can simply sort them.

Reviewed-by: imsnah

Fix a syntax error in gnt-backup

I broke gnt-backup in rev 1035, sorry :(

Reviewed-by: imsnah

Further fixes to enable RAPI startup

Note that since RAPI itself doesn't use luxi.Client yet, nothing works,
but at least it can startup now.

Reviewed-by: imsnah

Improve cli.SubmitOpCode

Currently, the feedback_fn argument to SubmitOpCode is no longer used.
We still need it in burnin, so we re-enable it by making the code call
that function with the msg argument in case feedback_fn is callable. The
patch also modifies burnin to accept the new argument format (msg is not...

Cache some jobs in memory

This patch adds a caching mechanisms to the JobStorage. Note that is
does not make the memory cache authoritative.

The algorithm is:
- all jobs loaded from disks are entered in the cache
- all new jobs are entered in the cache...

Change JobStorage to work with ids not filenames

Currently some of the functions in JobStorage work with filenames (which
is an implementation detail and should only be used when dealing with
the storage) and not with job IDs. We need to change this in order to...

Convert backend.py to the logging module

The patch also switches some of the exception logs to use
logging.exception (and therefore the log message will have a diferent
format).

(Note that this might not be a good choice in all cases, though)

Reviewed-by: imsnah

Fix rapi documentation

As a nice side-effect this also fixes "make distcheck".

The way used to include the generated rapi-resources.sgml file only
works if it's built in the same directory. That's not the case during
"make distcheck". The patch changes the sed script used for variables...

Copy the rest of the Restful-API files to trunk

Reviewed-by: imsnah

Use new HTTP server classes in ganeti-noded

Reviewed-by: iustinp

Make "gnt-job list" work again

"gnt-job list" was broken after my recent changes in the RPC
between clients and the master. This patch makes it work again.

Reviewed-by: iustinp