Synnefo » snf-ganeti » ganeti-local

Pass ssconf values from master to node

Instead of parsing the configuration on the node, we pass the ssconf
values from the master.

Reviewed-by: iustinp

Get rid of node daemon password

With the new SSL client certificate stuff it's no longer needed.

Reviewed-by: iustinp

Reuse HTTP client pool for RPC

ganeti-masterd: Add initialization and shutdown of RPC pool. It needs
to be shutdown before forking.

ganeti.cli: Add decorator function to initialize and shutdown RPC pool.

ganeti.rpc: Add functions to initialize and shutdown RPC pool. Throw...

Abstract runtime creation of dirs into a function

Currently the dir creation in ganeti-noded is in the main function. This
is not nice: we move it into a separate function and also add creation
of the OS_LOG_DIR (with different permissions, but in the same way)....

Export the disk index in the import/export scripts

We want to export the disk index as some OSes will only want to export
the first disk (or the second one, etc.), even if we have multiple
disks.

The patch also updates the backend.ExportSnapshot docstring....

Pass request headers in to RAPI handlers.

Reviewed-by: iustinp

Remove the logger.py module

Since now we use only one function from the logger module
(SetupLogging), we move it to utils.py (which is already imported by all
users of this function), and we remove the module.

Reviewed-by: imsnah

ETag passing support.

Reviewed-by: imsnah

Improvements to the master startup checks

In order to account for future improvements to master failover, we move
the actual data gathering capabilities from ganeti-masterd into
bootstrap.py, and we leave only the verification into masterd.

The verification procedure is then changed to retry multiple times (up...

Add a rpc call for changing the drain flag

A new multi-node call is added that sets/resets the drain flag.

Reviewed-by: imsnah

rapi: Whitespace fixes

Reviewed-by: ultrotter

Fix a few rpc-related errors

This fixes:
- whitespace change, double lines between methods
- duplication of call_upload_file, introduced by mistake in rev 1795
and which went undetected because of the many changes in that ref
(only diff -b shows it clearly)...

Convert ganeti-noded to new HTTP server class

Reviewed-by: iustinp

Move the hypervisor attribute to the instances

This (big) patch moves the hypervisor type from the cluster to the
instance level; the cluster attribute remains as the default hypervisor,
and will be renamed accordingly in a next patch. The cluster also gains...

Implement job 'waiting' status

Background: when we have multiple jobs in the queue (more than just a
few), many of the jobs (up to the number of threads) will be in state
'running', although many of them could be actually blocked, waiting for
some locks. This is not good, as one cannot easily see what is...

backend.py change to get cluster name from master

Currently there are three function in backend that need the cluster name
in order to instantiate an SshRunner. The patch changes these to get the
cluster name from the master in the rpc call; once the multi-hypervisor...

Convert ganeti-watcher

Use RPC calls instead of ssconf.

Reviewed-by: iustinp

Add new query to get cluster config values

This can be used to retrieve certain cluster config values from
within clients.

OpDumpClusterConfig was not used anywhere, hence I'm just reusing
it. The way ConfigWriter.DumpConfig returned the configuration
was not thread-safe, anyway (no deepcopy)....

Fix the watcher not restarting instance bug

The watcher was using conflicting attributes of the instance:
- it queried the admin_/oper_state, which are booleans
- but it compared those to the status (which is a text field)

The code was changed to query the aggregated 'status' field, as that...

ganeti-noded: Add constant for queue lock timeout

Reviewed-by: iustinp

Export backend.GetMasterInfo over the rpc layer

We create a multi-node call so that querying all nodes for agreement
will be fast.

Reviewed-by: imsnah

noded: Get job queue lock while purging queue content

Only one process should modify the queue at the same time.

Reviewed-by: iustinp

Make sure that client programs get all messages

This is a large patch, but I can't figure out how to split it without
breaking stuff. The old way of getting messages by always getting the
last one didn't bring all messages to the client if they were added...

Add RPC call to wait for job changes

This way clients can react faster to status or message changes and
don't have to poll anymore.

Reviewed-by: ultrotter

noded: Add RPC function to rename job queue files

This will be used to archive jobs.

Reviewed-by: iustinp

Implement queue locking in node daemon

Reviewed-by: iustinp

Add job queue RPC functions

jobqueue_update: Uploads a job queue file's content to a node. The
most common operation is to upload something that we already have
in a string. Unlike in the upload_file function, the file is not
read again when distributing changes, but content has to be passed...

Notify job queue about added/removed nodes

The job queue maintains its own node list and must be notified
when nodes are added/removed.

Reviewed-by: iustinp

jqueue: Don't pass the list of nodes to SubmitJob anymore

The job queue now maintains its own list and is updated when
nodes are added or removed from the cluster.

Reviewed-by: iustinp

Implement query for nodes

Reviewed-by: iustinp

First write operation (add tag) for Ganeti RAPI

Add instance tag handling, improved error logging.
...oh, yes adopt instance listing for RAPI2!

Reviewed-by: iustinp

Rework master startup/shutdown/failover

This (big) patch reworks the master startup/shutdown and the fixes the
master failover.

What does the patch do?

For master start/stop:
- remove the old ganeti-master script and its associated man page
- moves the ip start/stop directly into the backend.(Start|Stop)Master...

Add a new parameter to backend.(Start|Stop)Master

This patch adds a new, unused for now, parameter to the start and stop
master operations in backend. The idea behind it is that we need to be
able to control whether the IP (de)activation is coupled with daemon...

Make the rapi daemon create a pidfile

This is needed for controlling it cleanly with start-stop daemon.

Reviewed-by: ultrotter

Move ganeti-rapi core code to daemon

All other daemons have their main code in themselves and not in a module.
This patch does the same to ganeti-rapi by moving the code from
lib/rapi/RESTHTTPServer.py to daemons/ganeti-rapi.

Reviewed-by: iustinp

ganeti-masterd: write and remove pidfile

Reviewed-by: iustinp

Distribute the queue serial file after each update

This patch adds distribution of the queue serial file after each write
to it (but before a new job is created and written with that ID, and
before a response is returned, so we should be safe from crashes in...

Use new signal handler class in master daemon

Reviewed-by: ultrotter

Fork ganeti-noded

Create a new ForkingHTTPServer in ganeti-noded by deriving both from
NodeDaemonHttpServer and ForkingMixin. This will allow us to process
concurrent requests.

Reviewed-by: imsnah

Use workerpool in master daemon

Reusing threads instead of starting one for each request is more efficient.

Reviewed-by: iustinp

Initial copy of RAPI filebase to the trunk

Reviewed-by: iustinp

Move watcher's LockFile function to utils

Reviewed-by: iustinp

ganeti-noded logging improvements

The patch adds some more logging to the node daemon:

- log methods at beggining not only at the end
- log method parameters (they are very verbose, but useful)

A separate change is to initialize the global variable in the global...

Remove old job queue code

Reviewed-by: iustinp

Make luxi RPC more flexible

- Use constants for dict entries
- Handle exceptions on server side
- Rename client function to CallMethod to match server side naming

Reviewed-by: iustinp

Create all SUB_RUN_DIRS in ganeti-noded

Rather than just creating BDEV_CACHE_DIR we loop through the
SUB_RUN_DIRS list and create all its childs.

Reviewed-by: iustinp

Add custom logging setup for daemons

It's better for daemons if:
- they log only to one log file
- the log level is included
- for debug runs, the filename/line number is included

This patch moves the custom formatter from the watcher to the logging...

ganeti-masterd: Use logging module

Reviewed-by: ultrotter, iustinp

Increase the thread size to 5

Now that we use the locking library to make sure running opcodes cannot
step on each other toes we can have a bigger thread size, and
potentially process many opcodes in a parallel manner.

Reviewed-by: iustinp

ganeti-masterd: init and distribute common context

This patch creates a new GanetiContext class, which is used to hold
context common to all ganeti worker threads. As for the
GanetiLockingManager class it is paramount that there is only one such
class throughout the execution of Ganeti, so the class checks for that,...