Synnefo » snf-ganeti

Convert rpc module to RpcRunner

This big patch changes the call model used in internode-rpc from
standalong function calls in the rpc module to via a RpcRunner class,
that holds all the methods. This can be used in the future to enable
smarter processing in the RPC layer itself (some quick examples are not...

Implement job auto-archiving

This patch adds a new luxi call that implements auto-archiving of jobs
older than a certain age (or -1 for all completed jobs), and the gnt-job
command that makes use of this (with 'all' for -1).

Reviewed-by: imsnah

Enhance the job-related timestamps

This patch adds start, stop, and received timestamp for jobs (and allows
querying of them), and allows querying of the opcode timestamps.

Reviewed-by: imsnah

Implement job summary in gnt-job list

It is not currently possibly to show a summary of the job in the output
of “gnt-job list”. The closes is listing the whole opcode(s), but that
is too verbose. Also, the default output (id, status) is not very
useful, unless one looks for (and knows about) an exact job ID....

jqueue: Add common RPC error handling function

We didn't decide yet what exactly it should do with failed nodes.

Reviewed-by: ultrotter

jqueue: Replace normal cache dict with weakref dict

A job should only exist once in memory. After the cache is cleaned,
there can still be references to a job somewhere else. If there
are multiple instances, one can get updated while a function is
waiting for changes on another instance. By using...

jqueue: Reset run_op_idx after job is done

It can be confusing otherwise.

Reviewed-by: ultrotter

Add RPC call to wait for job changes

This way clients can react faster to status or message changes and
don't have to poll anymore.

Reviewed-by: ultrotter

jqueue: Move archived jobs on all nodes

Otherwise one might have archived jobs back in the list after a master
failover.

Reviewed-by: iustinp

jqueue: Use new job queue RPC functions

Reviewed-by: iustinp

jqueue: Don't pass the list of nodes to SubmitJob anymore

The job queue now maintains its own list and is updated when
nodes are added or removed from the cluster.

Reviewed-by: iustinp

jqueue: Replicate jobs to all nodes

Newly added nodes are not yet taken care of. Queue locking on
non-master nodes is not yet correct.

Reviewed-by: iustinp

jqueue: Move assert into decorator

This reduces code duplication. A later patch will modify the job queue
a bit more and will need a change of this assert. The assertion is
also removed from all class-internal functions.

Reviewed-by: iustinp

Fix pylint-detected issues

This is mostly:
- whitespace fix (space at EOL in some files, not all, broken
indentation, etc)
- variable names overriding others (one is a real bug in there)
- too-long-lines
- cleanup of most unused imports (not all)...

jqueue: Fix error logging

The passed parameters were not correct.

Reviewed-by: iustinp, ultrotter

Add “canceled” status for opcodes

Reviewed-by: ultrotter

Implement job archiving on the server side

So far no error reporting to the client is done. Clients don't get
noticed if a job doesn't exist or couldn't be archived because of
its current status.

The internal cache is always cleaned when the preconditions didn't...

Move code formatting job ID into a base class

A later patch will add a memory based job storage class, hence this
code is going into a separate class. It also changes the number format
to always use at least 10 digits, allowing up to 9'999'999'999 jobs to...

Fix logging with string job IDs

The job ID is now a string, hence logging must use %s instead of %d.

Reviewed-by: iustinp

Distribute the queue serial file after each update

This patch adds distribution of the queue serial file after each write
to it (but before a new job is created and written with that ID, and
before a response is returned, so we should be safe from crashes in...

Make argument to CleanCacheUnlocked mandatory

Not passing the argument means it has the value None. Iterating None
doesn't work:
>>> "123" in None
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: iterable argument required...

Sort the job list in _GetJobIDsUnlocked

Since the IDs are integers, we can simply sort them.

Reviewed-by: imsnah

Cache some jobs in memory

This patch adds a caching mechanisms to the JobStorage. Note that is
does not make the memory cache authoritative.

The algorithm is:
- all jobs loaded from disks are entered in the cache
- all new jobs are entered in the cache...

Change JobStorage to work with ids not filenames

Currently some of the functions in JobStorage work with filenames (which
is an implementation detail and should only be used when dealing with
the storage) and not with job IDs. We need to change this in order to...

Make "gnt-job list" work again

"gnt-job list" was broken after my recent changes in the RPC
between clients and the master. This patch makes it work again.

Reviewed-by: iustinp

Add a simple decorator for instance methods

This is just a simple, hardcoded decorator for object methods needing
synchronization on the _lock instance attribute.

Reviewed-by: ultrotter

Remove old job queue code

Reviewed-by: iustinp

Fix a typo in jqueue.py

s/result/op_result/ (this code was never used, so this wasn't caught)

Reviewed-by: iustinp

Implement selective job query

This patch implements query-ing of only selected jobs instead of all.

Reviewed-by: ultrotter

A dumb queue implementation

This patch adds a very dumb in-memory only queue implementation.

Reviewed-by: imsnah