Synnefo » snf-ganeti » ganeti-local

.. contents:: :depth: 4

As for 2.1 we divide the 2.2 design into three areas:

- core changes, which affect the master daemon/job queue/locking or

  all/most logical units

- logical unit/feature changes

Core changes

Master Daemon Scaling improvements

Current state and shortcomings

Currently the Ganeti master daemon is based on four sets of threads:

scalability issues:

Core daemon connection handling

Since the 16 client worker threads handle one connection each, it's very

easy to exhaust them, by just connecting to masterd 16 times and not

informed that everything is proceeding, and doesn't need to time out.

Wait for job change

The REQ_WAIT_FOR_JOB_CHANGE luxi operation makes the relevant client

thread block on its job for a relative long time. This is another easy

contention (see below).

Job Queue lock

The job queue lock is quite heavily contended, and certain easily

reproducible workloads show that's it's very easy to put masterd in

    remote rpcs to complete (starting, finishing, and submitting jobs)

Proposed changes

In order to be able to interact with the master daemon even when it's

under heavy load, and  to make it simpler to add core functionality

understand, debug, and scale.

Connection handling

We'll move the main thread of ganeti-masterd to asyncore, so that it can

share the mainloop code with all other Ganeti daemons. Then all luxi

thread on the socket.

Wait for job change

The REQ_WAIT_FOR_JOB_CHANGE luxi request is changed to be

subscription-based, so that the executing thread doesn't have to be

    them at a maximum rate (lower priority).

Job Queue lock

In order to decrease the job queue lock contention, we will change the

code paths in the following ways, initially:

Remote procedure call timeouts

Current state and shortcomings

The current RPC protocol used by Ganeti is based on HTTP. Every request

consists of an HTTP PUT request (e.g. ``PUT /hooks_runner HTTP/1.0``)

unresponsive node daemon cases.

Proposed changes

RPC glossary

Function call ID

  Unique identifier returned by ``ganeti-noded`` after invoking a

  Process started by ``ganeti-noded`` to call actual (backend) function.

Protocol

Initially we chose HTTP as our RPC protocol because there were existing

libraries, which, unfortunately, turned out to miss important features

would be an implicit ping-mechanism.

Request handling

To support the protocol changes described above, the way the node daemon

handles request will have to change. Instead of forking and handling

Inter-cluster instance moves

Current state and shortcomings

With the current design of Ganeti, moving whole instances between

different clusters involves a lot of manual work. There are several ways

this process in Ganeti 2.2.

Proposed changes

Authorization, Authentication and Security

Until now, each Ganeti cluster was a self-contained entity and wouldn't

talk to other Ganeti clusters. Nodes within clusters only had to trust

certificate while providing a client certificate to the server.

Copying data

To simplify the implementation, we decided to operate at a block-device

level only, allowing us to easily support non-DRBD instance moves.

directly, where it'll be written to the new block device directly again.

Workflow

#. Third party tells source cluster to shut down instance, asks for the

   instance specification and for the public part of an encryption key

#. Source cluster removes the instance if requested

Instance move in pseudo code

.. highlight:: python

.. highlight:: text

Miscellaneous notes

- A very similar system could also be used for instance exports within

  the same cluster. Currently OpenSSH is being used, but could be

Privilege separation

Current state and shortcomings

All Ganeti daemons are run under the user root. This is not ideal from a

security perspective as for possible exploitation of any daemon the user

is in the same group.

Implementation

For Ganeti 2.2 the implementation will be focused on a the RAPI daemon

only. This involves changes to ``daemons.py`` so it's possible to drop

Feature changes

KVM Security

Current state and shortcomings

Currently all kvm processes run as root. Taking ownership of the

hypervisor process, from inside a virtual machine, would mean a full

option of subverting other basic services on the cluster (eg: ssh).

Proposed changes

We would like to decrease the surface of attack available if an

hypervisor is compromised. We can do so adding different features to

subvert the node.

Dropping privileges in kvm to a single user (easy)

By passing the ``-runas`` option to kvm, we can make it drop privileges.

The user can be chosen by an hypervisor parameter, so that each instance

- read unprotected data on the node filesystem

Running kvm in a chroot (slightly harder)

By passing the ``-chroot`` option to kvm, we can restrict the kvm

process in its own (possibly empty) root directory. We need to set this

Running kvm with a pool of users (slightly harder)

If rather than passing a single user as an hypervisor parameter, we have

a pool of useable ones, we can dynamically choose a free one to use and

can still be combined with the chroot benefits.

Running iptables rules to limit network interaction (easy)

These don't need to be handled by Ganeti, but we can ship examples. If

the users used to run VMs would be blocked from sending some or all

Running kvm inside a container (even harder)

Recent linux kernels support different process namespaces through

control groups. PIDs, users, filesystems and even network interfaces can

just rely on iptables.

Implementation plan

We will first implement dropping privileges for kvm processes as a

single user, and most probably backport it to 2.1. Then we'll ship

External interface changes

OS API

The OS variants implementation in Ganeti 2.1 didn't prove to be useful

enough to alleviate the need to hack around the Ganeti API in order to

OS version

A new ``os_version`` file will be supported by Ganeti. This file is not

required, but if existing, its contents will be checked for consistency

intra-cluster migration.

Parameters

The interface between Ganeti and the OS scripts will be based on

environment variables, and as such the parameters and their values will

need to be valid in this context.

Names

The parameter names will be declared in a new file, ``parameters.list``,

together with a one-line documentation (whitespace-separated). Example::

parameters which differ in case only.

Values

The values of the parameters are, from Ganeti's point of view,

completely freeform. If a given parameter has, from the OS' point of

Environment variables

The parameters will be exposed in the environment upper-case and

prefixed with the string ``OSP_``. For example, a parameter declared in