Synnefo » snf-ganeti » ganeti-local

Export backend.GetMasterInfo over the rpc layer

We create a multi-node call so that querying all nodes for agreement
will be fast.

Reviewed-by: imsnah

noded: Get job queue lock while purging queue content

Only one process should modify the queue at the same time.

Reviewed-by: iustinp

Make sure that client programs get all messages

This is a large patch, but I can't figure out how to split it without
breaking stuff. The old way of getting messages by always getting the
last one didn't bring all messages to the client if they were added...

Add RPC call to wait for job changes

This way clients can react faster to status or message changes and
don't have to poll anymore.

Reviewed-by: ultrotter

noded: Add RPC function to rename job queue files

This will be used to archive jobs.

Reviewed-by: iustinp

Implement queue locking in node daemon

Reviewed-by: iustinp

Add job queue RPC functions

jobqueue_update: Uploads a job queue file's content to a node. The
most common operation is to upload something that we already have
in a string. Unlike in the upload_file function, the file is not
read again when distributing changes, but content has to be passed...

Notify job queue about added/removed nodes

The job queue maintains its own node list and must be notified
when nodes are added/removed.

Reviewed-by: iustinp

jqueue: Don't pass the list of nodes to SubmitJob anymore

The job queue now maintains its own list and is updated when
nodes are added or removed from the cluster.

Reviewed-by: iustinp

Implement query for nodes

Reviewed-by: iustinp

First write operation (add tag) for Ganeti RAPI

Add instance tag handling, improved error logging.
...oh, yes adopt instance listing for RAPI2!

Reviewed-by: iustinp

Rework master startup/shutdown/failover

This (big) patch reworks the master startup/shutdown and the fixes the
master failover.

What does the patch do?

For master start/stop:
- remove the old ganeti-master script and its associated man page
- moves the ip start/stop directly into the backend.(Start|Stop)Master...

Add a new parameter to backend.(Start|Stop)Master

This patch adds a new, unused for now, parameter to the start and stop
master operations in backend. The idea behind it is that we need to be
able to control whether the IP (de)activation is coupled with daemon...

Make the rapi daemon create a pidfile

This is needed for controlling it cleanly with start-stop daemon.

Reviewed-by: ultrotter

Move ganeti-rapi core code to daemon

All other daemons have their main code in themselves and not in a module.
This patch does the same to ganeti-rapi by moving the code from
lib/rapi/RESTHTTPServer.py to daemons/ganeti-rapi.

Reviewed-by: iustinp

ganeti-masterd: write and remove pidfile

Reviewed-by: iustinp

Distribute the queue serial file after each update

This patch adds distribution of the queue serial file after each write
to it (but before a new job is created and written with that ID, and
before a response is returned, so we should be safe from crashes in...

Use new signal handler class in master daemon

Reviewed-by: ultrotter

Fork ganeti-noded

Create a new ForkingHTTPServer in ganeti-noded by deriving both from
NodeDaemonHttpServer and ForkingMixin. This will allow us to process
concurrent requests.

Reviewed-by: imsnah

Use workerpool in master daemon

Reusing threads instead of starting one for each request is more efficient.

Reviewed-by: iustinp

Initial copy of RAPI filebase to the trunk

Reviewed-by: iustinp

Move watcher's LockFile function to utils

Reviewed-by: iustinp

ganeti-noded logging improvements

The patch adds some more logging to the node daemon:

- log methods at beggining not only at the end
- log method parameters (they are very verbose, but useful)

A separate change is to initialize the global variable in the global...

Remove old job queue code

Reviewed-by: iustinp

Make luxi RPC more flexible

- Use constants for dict entries
- Handle exceptions on server side
- Rename client function to CallMethod to match server side naming

Reviewed-by: iustinp

Create all SUB_RUN_DIRS in ganeti-noded

Rather than just creating BDEV_CACHE_DIR we loop through the
SUB_RUN_DIRS list and create all its childs.

Reviewed-by: iustinp

Add custom logging setup for daemons

It's better for daemons if:
- they log only to one log file
- the log level is included
- for debug runs, the filename/line number is included

This patch moves the custom formatter from the watcher to the logging...

ganeti-masterd: Use logging module

Reviewed-by: ultrotter, iustinp

Increase the thread size to 5

Now that we use the locking library to make sure running opcodes cannot
step on each other toes we can have a bigger thread size, and
potentially process many opcodes in a parallel manner.

Reviewed-by: iustinp

ganeti-masterd: init and distribute common context

This patch creates a new GanetiContext class, which is used to hold
context common to all ganeti worker threads. As for the
GanetiLockingManager class it is paramount that there is only one such
class throughout the execution of Ganeti, so the class checks for that,...

ganeti-noded: quit on QuitGanetiException

Accoring to the usage documented in the QuitGanetiException docstring,
if we receive such an exception we'll set the global _EXIT_GANETI_NODED
variable to True, and then return either a valid value or an error
message to the user. This will be the last request we serve, though,...

Handle any exception in ganeti-masterd

If an uncaught exception is thrown currently it destroys the calling
thread. This patch changes the behaviour to failing the current job,
logging a message, but trying to keep the daemon up.

Reviewed-by: imsnah

Use a single Makefile.am instead of many

This change allows us to use cleaner dependencies between
directories. The build system is basically rewritten in large parts
and may contain bugs.

Reviewed-by: iustinp

ganeti-watcher: Don't write file if data didn't change

This is the safest way to detect changes and the amount of data
is small, so keeping a copy around is cheap enough.

Reviewed-by: iustinp

Don't log SystemExit exception in ganeti-watcher

Reviewed-by: iustinp

Write ganeti-watcher status file even if something failed

Reviewed-by: iustinp

Replace custom logging code in watcher with logging module

- Log timestamp for all messages
- Write everything to logfile and optionally to stderr
- Log messages are no longer buffered, allowing a user to see progress

Reviewed-by: ultrotter

Add migration support at the rpc layer

This patch adds the migration rpc call and its implementation in the
backend. The patch does not deal with the correct activation of disks.

Because of the new RPC, the protocol version is increased.

Reviewed-by: imsnah

Fail job on ganeti exceptions

When a Job raises a ganeti exception a message is printed but nothing is
reported in the job itself. It's better to update the job status, thus
notifying the client, possibly polling for the job result, of what went
wrong.

Reviewed-by: iustinp

Watcher: do not activate disks for admin_down

Currently the watcher does activate disks (via bootid mechanisms) even
for admin_down instances. This patch logs and skips over these
instances.

Reviewed-by: ultrotter

Disable forking in the master daemon

This patch adds a mechanism to disable utils.RunCmd in selected
programs. This is needed in the master daemon unless we confirm
threading doesn't pose any problems.

This makes cluster init fail, but creating new trunk clusters is anyway...

Convert cli.SubmitOpCode to use the master

This patch converts the cli.py SubmitOpCode method to use the unix
protocol and thus execute the opcodes via the master.

The patch allows a partial burnin to work with the master. Currently the
query opcodes, since they are executed via the SubmitOpCode, are...

Add per-opcode results to job processing

This patch changes the definition of a job and introduces per-opcode
results.

First, the result and status fields of a job are condensed into a single
'status' attribute. Then, we introduce an opcode status and one result...

ganeti-noded directory functions for file backend

Reviewed-by: ultrotter

Move the daemonize function to utils.py

Currently, in ganeti-noded we have the createDaemon function. Since
we'll need the same in other daemons, we move this function to utils.py

With the move, a few changes were also done:
- change the name to Daemonize()...

Reduce log noise for the new http-based rpc

This patch just removes an extraneous \n from the log message making it
nicer to view.

Reviewed-by: schreiberal

Modify utils.TcpPing to make source address optional

This patch modifies TcpPing and its callers to make the source address
selection optional. Usually, the kernel will know better what
source address to use, just in some cases we want to enforce a given...

Add a test opcode that sleeps for a given duration

This can be used for testing purposes.

Reviewed-by: ultrotter,imsnah

Various code style fixes for strings.

- When line wrapping is needed, move spaces to the next line.
- Remove embedded line breaks from error messages.

Reviewed-by: schreiberal

Small changes and fixes in ganeti-watcher.

- Use constants for keys.
- Fix bug through which automatic instance restarts wouldn't be limited

Reviewed-by: iustinp