Synnefo » snf-ganeti

Ganeti 2.0 Master daemon

========================

Objective

---------

Many of the important features of Ganeti 2.0 — job queue, granular

locking, external API, etc. — will be integrated via a master

daemon. While not absolutely necessary, it is the best way to

integrate all these components.

Background

----------

Currently there is no "master" daemon in Ganeti (1.2). Each command

tries to acquire the so called *cmd* lock and when it succeeds, it

takes complete ownership of the cluster configuration and state. The

scheduled improvements to Ganeti require or can use a daemon that

coordinates the activities/jobs scheduled/etc.

Overview

--------

The master daemon will be the central point of the cluster; command

line tools and the external API will interact with the cluster via

this daemon; it will be one coordinating the node daemons.

This design doc is best read in the context of the accompanying design

docs for Ganeti 2.0: Granular locking design and Job queue design.

Detailed Design

---------------

In Ganeti 2.0, we will have the following *entities*:

- the master daemon (on master node)

- the node daemon (all nodes)

- the command line tools (master node)

- the RAPI daemon (master node)

Interaction paths are between:

- (CLI tools/RAPI daemon) and the master daemon, via the so called *luxi* API

- the master daemon and the node daemons, via the node RPC

The protocol between the master daemon and the node daemons will be

changed to HTTP(S), using a simple PUT/GET of JSON-encoded

messages. This is done due to difficulties in working with the Twisted

framework and its protocols in a multithreaded environment, which we can

overcome by using a simpler stack (see the caveats section). The protocol

between the CLI/RAPI and the master daemon will be a custom one: on a UNIX

socket on the master node, with rights restricted by filesystem

permissions, the CLI/RAPI will talk to the master daemon using JSON-encoded

messages.

The operations supported over this internal protocol will be encoded

via a python library that will expose a simple API for its

users. Internally, the protocol will simply encode all objects in JSON

format and decode them on the receiver side.

The LUXI protocol

~~~~~~~~~~~~~~~~~

We will have two main classes of operations over the master daemon API:

- cluster query functions

- job related functions

The cluster query functions are usually short-duration, and are the

equivalent of the OP_QUERY_* opcodes in ganeti 1.2 (and they are

internally implemented still with these opcodes). The clients are

guaranteed to receive the response in a reasonable time via a timeout.

The job-related functions will be:

- submit job

- query job (which could also be categorized in the query-functions)

- archive job (see the job queue design doc)

- wait for job change, which allows a client to wait without polling

For more details, see the job queue design document.

Daemon implementation

~~~~~~~~~~~~~~~~~~~~~

The daemon will be based around a main I/O thread that will wait for

new requests from the clients, and that does the setup/shutdown of the

other thread (pools).

There will two other classes of threads in the daemon:

- job processing threads, part of a thread pool, and which are

  long-lived, started at daemon startup and terminated only at shutdown

  time

- client I/O threads, which are the ones that talk the local protocol

  to the clients

Master startup/failover

~~~~~~~~~~~~~~~~~~~~~~~

In Ganeti 1.x there is no protection against failing over the master

to a node with stale configuration. In effect, the responsibility of

correct failovers falls on the admin. This is true both for the new

master and for when an old, offline master startup.

Since in 2.x we are extending the cluster state to cover the job queue

and have a daemon that will execute by itself the job queue, we want

to have more resilience for the master role.

The following algorithm will happen whenever a node is ready to

transition to the master role, either at startup time or at node

failover:

#. read the configuration file and parse the node list

   contained within

#. query all the nodes and make sure we obtain an agreement via

   a quorum of at least half plus one nodes for the following:

    - we have the latest configuration and job list (as

      determined by the serial number on the configuration and

      highest job ID on the job queue)

    - there is not even a single node having a newer

      configuration file

    - if we are not failing over (but just starting), the

      quorum agrees that we are the designated master

#. at this point, the node transitions to the master role

#. for all the in-progress jobs, mark them as failed, with

   reason unknown or something similar (master failed, etc.)

Logging

~~~~~~~

The logging system will be switched completely to the logging module;

currently it's logging-based, but exposes a different API, which is

just overhead. As such, the code will be switched over to standard

logging calls, and only the setup will be custom.

With this change, we will remove the separate debug/info/error logs,

and instead have always one logfile per daemon model:

- master-daemon.log for the master daemon

- node-daemon.log for the node daemon (this is the same as in 1.2)

- rapi-daemon.log for the RAPI daemon logs

- rapi-access.log, an additional log file for the RAPI that will be

  in the standard http log format for possible parsing by other tools

Since the watcher will only submit jobs to the master for startup of

the instances, its log file will contain less information than before,

mainly that it will start the instance, but not the results.

Caveats

-------

A discussed alternative is to keep the current individual processes

touching the cluster configuration model. The reasons we have not

chosen this approach is:

- the speed of reading and unserializing the cluster state

  today is not small enough that we can ignore it; the addition of

  the job queue will make the startup cost even higher. While this

  runtime cost is low, it can be on the order of a few seconds on

  bigger clusters, which for very quick commands is comparable to

  the actual duration of the computation itself

- individual commands would make it harder to implement a

  fire-and-forget job request, along the lines "start this

  instance but do not wait for it to finish"; it would require a

  model of backgrounding the operation and other things that are

  much better served by a daemon-based model

Another area of discussion is moving away from Twisted in this new

implementation. While Twisted hase its advantages, there are also many

disatvantanges to using it:

- first and foremost, it's not a library, but a framework; thus, if

  you use twisted, all the code needs to be 'twiste-ized'; we were able

  to keep the 1.x code clean by hacking around twisted in an

  unsupported, unrecommended way, and the only alternative would have

  been to make all the code be written for twisted

- it has some weaknesses in working with multiple threads, since its base

  model is designed to replace thread usage by using deferred calls, so while

  it can use threads, it's not less flexible in doing so

And, since we already have an HTTP server library for the RAPI, we

can just reuse that for inter-node communication.