Synnefo » snf-ganeti

LockSet: allow lists with duplicate values

If a list with a duplicate value is passed to a lockset what the code
now does is to try to acquire the lock twice, generating a
double-acquire exception in the SharedLock code. This is definitely an
issue. In order to solve it we can either forbit double values in a list...

LURebootInstance: move arg check in ExpandNames

The check for the reboot type can be done without any locks held, so
we'll move it to ExpandNames. Plus, we note in a FIXME that if the
reboot type is not full, we can probably just lock the primary node, and...

Use Linux-specific way to name master socket

By using this Linux-specific way we don't have to care about removing the
socket file when quitting or starting (after an unclean shutdown). For a
more detailed description, see the comment in the patch.

Reviewed-by: schreiberal

SshRunner: Add parameter to always accept peer's SSH key

This will be used to add nodes without user interaction, specifically
in QA tests.

Reviewed-by: ultrotter

SshRunner.Run: Pass all arguments to BuildCmd

This patch changes SshRunner.Run to pass all arguments to
SshRunner.BuildCmd. They had the same arguments before
and should stay that way. This change makes it easier
to add new or change existing arguments.
...

RunCmd: add optional environment overriding

If the user passes an env dict to RunCmd we'll override the environment
passed to the to-be-executed command with the values in the dict. This
allows us to pass arbitrary environment values to commands we run.
...

Add KVM hypervisor code

ht_kvm.py contains the code for ganeti to work under kvm.
This patch also modifies Makefile.am to ship that file, and
lib/hypervisor/__init__.py to import it, and add kvm to the
hypervisors map.

Reviewed-by: imsnah

Add --with-kvm-path configure option

This allows to configure a different path to the kvm binary. By default
/usr/bin/kvm is used, which is the one found in debian and ubuntu.

Reviewed-by: imsnah

Convert RunCmd to an epydoc docstring

Reviewed-by: imsnah

Fix race locking issue in noded

Noded didn't release the job queue lock after initialising it. This
patch makes sure to unlock once the work is done.

Reviewed-by: ultrotter

Add RPC call to wait for job changes

This way clients can react faster to status or message changes and
don't have to poll anymore.

Reviewed-by: ultrotter

Add functions to split time into tuple and merge it back

These will be used for job logs.

Reviewed-by: ultrotter

Don't always remove queue lock when queue is purged

The lock should only be removed if ganeti-noded is going to quit.
Otherwise it needs to be kept to prevent another process from creating
it again while we're still holding the (removed) lock. This is due to...

jqueue: Move archived jobs on all nodes

Otherwise one might have archived jobs back in the list after a master
failover.

Reviewed-by: iustinp

backend: Add function to check whether file is in queue dir

Another function will need to check whether its parameters
are job queue files.

Reviewed-by: iustinp

jstore: Change to not always require a lock

This way we can do locking when both noded and masterd are running
on the same machine, the latter holding an exclusive lock on the
queue.

Reviewed-by: iustinp

Disallow uploading job queue files through upload_file

The job queue is now updated through its own RPC functions.

Reviewed-by: iustinp

Add job queue RPC functions

jobqueue_update: Uploads a job queue file's content to a node. The
most common operation is to upload something that we already have
in a string. Unlike in the upload_file function, the file is not
read again when distributing changes, but content has to be passed...

Fix cli.PollJob

feedback_fn wasn't passed to it.

Reviewed-by: iustinp

jqueue: Implement {Add,Remove}Node

These functions will be used to notify the queue about newly added
or removed nodes.

Reviewed-by: iustinp

Maintain node list in job queue

The code makes sure not to include the master in the list.

Reviewed-by: iustinp

Implement query for nodes

Reviewed-by: iustinp

jqueue: Replicate jobs to all nodes

Newly added nodes are not yet taken care of. Queue locking on
non-master nodes is not yet correct.

Reviewed-by: iustinp

jstore: Add queue helper functions

This will be used to move common code out of jqueue.

Reviewed-by: iustinp

jqueue: Move assert into decorator

This reduces code duplication. A later patch will modify the job queue
a bit more and will need a change of this assert. The assertion is
also removed from all class-internal functions.

Reviewed-by: iustinp

Allow job queue files to be uploaded through ganeti-noded

This is needed for job queue replication.

Reviewed-by: iustinp

jqueue: Store context in job queue instead of worker pool

The job queue will need to access to configuration, which is provided
through the context object, to get a list of nodes.

Reviewed-by: iustinp

First write operation (add tag) for Ganeti RAPI

Add instance tag handling, improved error logging.
...oh, yes adopt instance listing for RAPI2!

Reviewed-by: iustinp

Xen: remove two end-of-line semicolons

It's python, isn't it?

Reviewed-by: iustinp

Make gnt-* commands fail nicely on non-masters

This patch adds a check that we are on the master after failing to
connect to the socket, and log nicely the master name.

Reviewed-by: ultrotter

ChainOpCode is still BGL-only

Prevent mistakes with an assert.

Reviewed-by: iustinp

Fix pylint-detected issues

This is mostly:
- whitespace fix (space at EOL in some files, not all, broken
indentation, etc)
- variable names overriding others (one is a real bug in there)
- too-long-lines
- cleanup of most unused imports (not all)...

Unify SetupDaemon/SetupLogging

The 'old-style' info, error, debug logs do not make much sense. This
patch unifies the SetupLogging and SetupDaemon functions. As a result,
all the commands logs to a 'commands.log' file.

The patch also changes the log setup to keep going if there's an error...

Parallelize {Startup,Shutdown,Reboot}Instance

Reviewed-by: iustinp

LogicalUnit._LockInstancesNodes helper function

This function is used to lock instances' primary and secondary nodes
after locking instances themselves.

Reviewed-by: iustinp

Add LogicalUnit.DeclareLocks

This additional LogicalUnit function is optional to implement, but lets
you change your locking needs for one level just before locking it, but
after the previous levels have been already locked. It is useful for
example to calculate what nodes to lock after locking an instance....

Rewrite job queue

We found several issues in the old job queue implementation. It had race
conditions, deadlocks and other deficiencies.

Short summary:
- _QueuedOpCode and _QueuedJob are now more or less data structures with a few
utility functions. __Setup is gone....

Rework master startup/shutdown/failover

This (big) patch reworks the master startup/shutdown and the fixes the
master failover.

What does the patch do?

For master start/stop:
- remove the old ganeti-master script and its associated man page
- moves the ip start/stop directly into the backend.(Start|Stop)Master...

Implement checking for the master role in rapi

This patch moves the CheckMaster function from ganeti-masterd to ssconf
(most logical place, it cannot go in utils since we would have recursive
imports between ssconf and utils) and changes ganeti-rapi to also call...

Log thread name when debug output is enabled

Reviewed-by: iustinp

Fix constants typo

Reviewed-by: imsnah

Add a KillProcess function

We cannot depend on all environments to have a start-stop-daemon or
similar tool. We instead implement a KillProcess function that behaves
similar to “start-stop-daemon --retry”.

Note that the attached unittest can hang in foreground if the child...

Move ganeti-rapi core code to daemon

All other daemons have their main code in themselves and not in a module.
This patch does the same to ganeti-rapi by moving the code from
lib/rapi/RESTHTTPServer.py to daemons/ganeti-rapi.

Reviewed-by: iustinp

Implement job canceling on server side

Locking is not completeley right due to a deadlock when the job calls
UpdateJob after changing its status.

Reviewed-by: ultrotter

Add “canceled” status for opcodes

Reviewed-by: ultrotter

Convert set to a list in LUGetTags

The set triggers exception on a list-tags command and RAPI calls for tags
since it is not serializable by JSON.

Reviewed-by: iustinp

Implement job archiving on the server side

So far no error reporting to the client is done. Clients don't get
noticed if a job doesn't exist or couldn't be archived because of
its current status.

The internal cache is always cleaned when the preconditions didn't...

Move code formatting job ID into a base class

A later patch will add a memory based job storage class, hence this
code is going into a separate class. It also changes the number format
to always use at least 10 digits, allowing up to 9'999'999'999 jobs to...

Add utils.IsPidFileAlive function

This helper function reads a pid from a file containing it and checks
whether it refers to a live process.

Reviewed-by: iustinp

Generalization of bulk output mapping

Reviewed-by: iustinp