Synnefo » snf-ganeti

While Ganeti 1.2 is usable, it severely limits the flexibility of the

- fixed number of NICs

create a more flexible code base for future developments.

Among these problems, the single-operation at a time restriction is

biggest issue with the current version of Ganeti. It is such a big

impediment in operating bigger clusters that many times one is tempted

to remove the lock just to do a simple operation like start instance

while an OS installation is running.

is the fact that currently there is no long-lived process in Ganeti

that can coordinate multiple operations. Each command tries to acquire

the so called *cmd* lock and when it succeeds, it takes complete

ownership of the cluster configuration and state.

Architecture issues

-------------------

The fact that each command is a separate process that reads the

cluster state, executes the command, and saves the new state is also

an issue on big clusters where the configuration data for the cluster

begins to be non-trivial in size.

clients that talk to this daemon (see `Master daemon`_). This will

allow us to get rid of the global cluster lock for most operations,

having instead a per-object lock (see `Granular locking`_). Also, the

daemon will be able to queue jobs, and this will allow the individual

clients to submit jobs without waiting for them to finish, and also

see the result of old requests (see `Job Queue`_).

storage, and separate that into name spaces (such that an Xen PVM

flexibility in defining additional parameters. For more details see

`Object parameters`_.

away from Twisted and use HTTP(s) for intra- and extra-cluster

The master-daemon related interaction paths are:

- (CLI tools/RAPI daemon) and the master daemon, via the so called *LUXI* API

There are also some additional interaction paths for exceptional cases:

- CLI tools might access via SSH the nodes (for ``gnt-cluster copyfile``

  and ``gnt-cluster command``)

- master failover is a special case when a non-master node will SSH

  and do node-RPC calls to the current master

changed from (Ganeti 1.2) Twisted PB (perspective broker) to HTTP(S),

using a simple PUT/GET of JSON-encoded messages. This is done due to

difficulties in working with the Twisted framework and its protocols

in a multithreaded environment, which we can overcome by using a

simpler stack (see the caveats section).

The protocol between the CLI/RAPI and the master daemon will be a

custom one (called *LUXI*): on a UNIX socket on the master node, with

rights restricted by filesystem permissions, the CLI/RAPI will talk to

the master daemon using JSON-encoded messages.

For more details about the RAPI daemon see `Remote API changes`_, and

for the node daemon see `Node daemon changes`_.

As described above, the protocol for making requests or queries to the

master daemon will be a UNIX-socket based simple RPC of JSON-encoded

messages.

The choice of UNIX was in order to get rid of the need of

authentication and authorisation inside Ganeti; for 2.0, the

permissions on the Unix socket itself will determine the access

rights.

We will have two main classes of operations over this API:

equivalent of the ``OP_QUERY_*`` opcodes in Ganeti 1.2 (and they are

For more details of the actual operation list, see the `Job Queue`_.

Both requests and responses will consist of a JSON-encoded message

followed by the ``ETX`` character (ASCII decimal 3), which is not a

valid character in JSON messages and thus can serve as a message

delimiter. The contents of the messages will be a dictionary with two

fields:

:method:

  the name of the method called

:args:

  the arguments to the method, as a list (no keyword arguments allowed)

Responses will follow the same format, with the two fields being:

:success:

  a boolean denoting the success of the operation

:result:

  the actual result, or error message in case of failure

There are two special value for the result field:

- in the case that the operation failed, and this field is a list of

  length two, the client library will try to interpret is as an exception,

  the first element being the exception type and the second one the

  actual exception arguments; this will allow a simple method of passing

  Ganeti-related exception across the interface

- for the *WaitForChange* call (that waits on the server for a job to

  change status), if the result is equal to ``nochange`` instead of the

  usual result for this call (a list of changes), then the library will

  internally retry the call; this is done in order to differentiate

  internally between master daemon hung and job simply not changed

Users of the API that don't use the provided python library should

take care of the above two cases.

Master daemon implementation

++++++++++++++++++++++++++++

  (LUXI) to the clients, and are short-lived

    - if any of the above is false, we prevent the current operation

      (i.e. we don't become the master)

Since due to exceptional conditions we could have a situation in which

no node can become the master due to inconsistent data, we will have

an override switch for the master daemon startup that will assume the

current node has the right data and will replicate all the

configuration files to the other nodes.

**Note**: the above algorithm is by no means an election algorithm; it

is a *confirmation* of the master role currently held by a node.

The logging system will be switched completely to the standard python

logging module; currently it's logging-based, but exposes a different

API, which is just overhead. As such, the code will be switched over

to standard logging calls, and only the setup will be custom.

  in the standard HTTP log format for possible parsing by other tools

Since the `watcher`_ will only submit jobs to the master for startup

of the instances, its log file will contain less information than

before, mainly that it will start the instance, but not the results.

Node daemon changes

+++++++++++++++++++

The only change to the node daemon is that, since we need better

concurrency, we don't process the inter-node RPC calls in the node

daemon itself, but we fork and process each request in a separate

child.

Since we don't have many calls, and we only fork (not exec), the

overhead should be minimal.

implementation. While Twisted has its advantages, there are also many

disadvantages to using it:

  you use twisted, all the code needs to be 'twiste-ized' and written

  in an asynchronous manner, using deferreds; while this method works,

  it's not a common way to code and it requires that the entire process

  workflow is based around a single *reactor* (Twisted name for a main

  loop)

- the more advanced granular locking that we want to implement would

  require, if written in the async-manner, deep integration with the

  Twisted stack, to such an extend that business-logic is inseparable

  from the protocol coding; we felt that this is an unreasonable request,

  and that a good protocol library should allow complete separation of

  low-level protocol calls and business logic; by comparison, the threaded

  approach combined with HTTPs protocol required (for the first iteration)

  absolutely no changes from the 1.2 code, and later changes for optimizing

  the inter-node RPC calls required just syntactic changes (e.g.

  ``rpc.call_...`` to ``self.rpc.call_...``)

Another issue is with the Twisted API stability - during the Ganeti

1.x lifetime, we had to to implement many times workarounds to changes

in the Twisted version, so that for example 1.2 is able to use both

Twisted 2.x and 8.x.

In the end, since we already had an HTTP server library for the RAPI,

we just reused that for inter-node communication.

- we preserve data coherency

- we prevent deadlocks

- we prevent job starvation

This section only talks about parallelising Ganeti level operations, aka

Logical Units, and the locking needed for that. Any other synchronization lock

Library details

+++++++++++++++

- internally managing all the locks, making the implementation transparent

- automatically grabbing multiple locks in the right order (avoid deadlock)

- ability to transparently handle conversion to more granularity

- support asynchronous operation (future goal)

Locking will be valid only on the master node and will not be a

distributed operation. Therefore, in case of master failure, the

operations currently running will be aborted and the locks will be

lost; it remains to the administrator to cleanup (if needed) the

operation result (e.g. make sure an instance is either installed

correctly or removed).

A corollary of this is that a master-failover operation with both

masters alive needs to happen while no operations are running, and

therefore no locks are held.

All the locks will be represented by objects (like

``lockings.SharedLock``), and the individual locks for each object

will be created at initialisation time, from the config file.

The API will have a way to grab one or more than one locks at the same time.

Any attempt to grab a lock while already holding one in the wrong order will be

checked for, and fail.

Each lock has the following three possible statuses:

- unlocked (anyone can grab the lock)

- shared (anyone can grab/have the lock but only in shared mode)

- exclusive (no one else can grab/have the lock)

sub-locks and live for the time necessary for all the code to convert (or

any other concurrent Ganeti operations.

When instances or nodes disappear from the cluster the relevant locks

must be removed. This is easier than adding new elements, as the code

which removes them must own them exclusively already, and thus deals

with metalocks exactly as normal code acquiring those locks. Any

operation queuing on a removed lock will fail after its removal.

- try to acquire this lock set and fail if not possible

- try to acquire one of these lock sets and return the first one you were

Code examples

+++++++++++++

This makes sure we release all locks, and avoid possible deadlocks. Of

course extra care must be used not to leave, if possible locked

structures in an unusable state. Note that with Python 2.5 a simpler

syntax will be possible, but we want to keep compatibility with Python

2.4 so the new constructs should not be used.

A Ganeti job will consist of multiple ``OpCodes`` which are the basic

   be canceled.

``/var/lib/ganeti/queue/archive/``.  This is done in order to speed up

the queue handling: by default, the jobs in the archive are not

touched by any functions. Only the current (unarchived) jobs are

parsed, loaded, and verified (if implemented) by the master daemon.

  kernel path for `PVM`_ which makes no sense for `HVM`_).

  values for hypervisor parameters that are not defined/overridden by

e.g. ``instance.hvparams.vnc_console_port`` instead of using both

``instance.hvparams.hvm_vnc_console_port`` and

``instance.hvparams.kvm_vnc_console_port``.

a disk has both Ganeti-related parameters (e.g. the name of the LV)

  comparing ``hvparams.keys()`` and ``cls.PARAMETERS``; this is a class

  method that can be called from within master code (i.e. cmdlib) and

  should be safe to do so

- ``OpCreateInstance``, where the new hv and be parameters will be sent as

- ``OpQueryInstances``, where we have to be able to query these new

- ``OpModifyInstance``, where the the modified parameters are sent as

One main assumption made was that disk failures should be treated as 'rare'

- disk failures can be a common occurrence, based on usage patterns or cluster

that device type; if there's a user-created DRBD minor, it will be

Removal of obsolete device types (MD, DRBD7)

DRBD7 has bad failure modes in case of dual failures (both network and

just panics. Second, due to the asymmetry between primary and

secondary in MD+DRBD mode, we cannot do live failover (not even if we

had MD+DRBD8).

Using files instead of logical volumes for instance storage would

allow us to get rid of the hard requirement for volume groups for

testing clusters and it would also allow usage of SAN storage to do

live failover taking advantage of this storage solution.

    #. if only one node is inconsistent:

        #. log this occurrence in the Ganeti log in a form that

- tear down the current DRBD association and setup a DRBD pairing between

  start re-syncing from S2

data loss, since we don't have a good copy of the data anymore, so in

will not reuse by mistake the DRBD device of another instance, since

it always looks for either our own or a free one.

syncing between two instance's disks, causing data loss.

only the instance creation has hard coded number of disks, not the disk

read-write a disk afterward.

of length one), the OS API which currently can export/import only one

- ease of porting from the old API

  operations will be forbidden for instances of that OS. This makes it easier

  The version of the OS API that the following parameters comply with;

  The hypervisor the instance should run on (e.g. 'xen-pvm', 'xen-hvm', 'kvm')

  The number of NICs this instance will have

  Type of the Nth NIC as seen by the instance. For example 'virtio',

  'rtl8139', etc.

These are only the basic variables we are thinking of now, but more

may come during the implementation and they will be documented in the

``ganeti-os-api`` man page. All these variables will be available to

all scripts.

script. The data will be compressed by Ganeti, so no compression should be

'ganeti_api_version' containing list of numbers matching the

version(s) of the API they implement. Ganeti itself will always be

compatible with one version of the API and may maintain backwards

compatibility if it's feasible to do so. The numbers are one-per-line,

so an OS supporting both version 5 and version 20 will have a file

containing two lines. This is different from Ganeti 1.2, which only

supported one version number.

subset of the Ganeti hypervisors, by declaring them in the 'hypervisors' file.

The first Ganeti remote API (RAPI) was designed and deployed with the

Ganeti 1.2.5 release.  That version provide read-only access to the

cluster state. Fully functional read-write API demands significant

internal changes which will be implemented in version 2.0.

We decided to go with implementing the Ganeti RAPI in a RESTful way,

which is aligned with key features we looking. It is simple,

stateless, scalable and extensible paradigm of API implementation. As

transport it uses HTTP over SSL, and we are implementing it with JSON

encoding, but in a way it possible to extend and provide any other

one.

The Ganeti RAPI is implemented as independent daemon, running on the

same node with the same permission level as Ganeti master

daemon. Communication is done through the LUXI library to the master

daemon. In order to keep communication asynchronous RAPI processes two

types of client requests:

- queries: server is able to answer immediately

- job submission: some time is required for a useful response

In the query case requested data send back to client in the HTTP

response body. Typical examples of queries would be: list of nodes,

instances, cluster info, etc.

In the case of job submission, the client receive a job ID, the

identifier which allows to query the job progress in the job queue

(see `Job Queue`_).

Internally, each exported object has an version identifier, which is

used as a state identifier in the HTTP header E-Tag field for

requests/responses to avoid race conditions.

The key difference of using REST instead of others API is that REST

requires separation of services via resources with unique URIs. Each

of them should have limited amount of state and support standard HTTP

For example in Ganeti's case we can have a set of URI:

 - ``/{clustername}/instances``

 - ``/{clustername}/instances/{instancename}``

 - ``/{clustername}/instances/{instancename}/tag``

 - ``/{clustername}/tag``

A GET request to ``/{clustername}/instances`` will return the list of

instances, a POST to ``/{clustername}/instances`` should create a new

instance, a DELETE ``/{clustername}/instances/{instancename}`` should

delete the instance, a GET ``/{clustername}/tag`` should return get

cluster tags.

Each resource URI will have a version prefix. The resource IDs are to

be determined.

Internal encoding might be JSON, XML, or any other. The JSON encoding

fits nicely in Ganeti RAPI needs. The client can request a specific

representation via the Accept field in the HTTP header.

REST uses HTTP as its transport and application protocol for resource

access. The set of possible responses is a subset of standard HTTP

responses.

The statelessness model provides additional reliability and

transparency to operations (e.g. only one request needs to be analyzed

to understand the in-progress operation, not a sequence of multiple

requests/responses).

With the write functionality security becomes a much bigger an issue.

The Ganeti RAPI uses basic HTTP authentication on top of an

SSL-secured connection to grant access to an exported resource. The

password is stored locally in an Apache-style ``.htpasswd`` file. Only

one level of privileges is supported.

Caveats

+++++++

The model detailed above for job submission requires the client to

poll periodically for updates to the job; an alternative would be to

allow the client to request a callback, or a 'wait for updates' call.

The callback model was not considered due to the following two issues:

- callbacks would require a new model of allowed callback URLs,

  together with a method of managing these

- callbacks only work when the client and the master are in the same

  security domain, and they fail in the other cases (e.g. when there is

  a firewall between the client and the RAPI daemon that only allows

  client-to-RAPI calls, which is usual in DMZ cases)

The 'wait for updates' method is not suited to the HTTP protocol,

where requests are supposed to be short-lived.

requires some noticeable changes in the way command line arguments are

- extend and modify command line syntax to support new features

- ensure consistent patterns in command line arguments to reduce

  cognitive load

As such, there are several areas of Ganeti where the command line

There are several areas of Ganeti where the command line arguments

will change:

The syntax to the device specific options is similar to the generic

The syntax to the device specific options is similar to the generic

Example::

  gnt-instance modify --disk 2:access=r

via the ``--hypervisor`` option. The generic syntax of the hypervisor

option is as follows::

time via the ``gnt-instance add`` command. If the hypervisor for an

using ``--hypervisor`` option of the ``gnt-instance modify``

command. However, the hypervisor type of an existing instance can not

be changed, only the particular hypervisor specific option can be

changed. Therefore, the format of the option parameters has been

simplified to omit the hypervisor name and only contain the comma

separated list of option-value pairs.

Example::

  gnt-instance modify --hypervisor cdrom=/srv/boot.iso,boot_order=cdrom:network test-instance

The generic format of the hypervisor cluster wide default setting

option is::

Glossary

========

Since this document is only a delta from the Ganeti 1.2, there are

some unexplained terms. Here is a non-exhaustive list.

.. _HVM:

HVM

  hardware virtualization mode, where the virtual machine is oblivious

  to the fact that's being virtualized and all the hardware is emulated

.. _LU:

LogicalUnit

  the code associated with an OpCode, i.e. the code that implements the

  startup of an instance

.. _opcode:

OpCode

  a data structure encapsulating a basic cluster operation; for example,

  start instance, add instance, etc.;

.. _PVM:

PVM

  para-virtualization mode, where the virtual machine knows it's being

  virtualized and as such there is no need for hardware emulation

.. _watcher:

watcher

  ``ganeti-watcher`` is a tool that should be run regularly from cron

  and takes care of restarting failed instances, restarting secondary

  DRBD devices, etc. For more details, see the man page

  ``ganeti-watcher(8)``.