Synnefo » snf-ganeti

Design for parallelized instance creations and opportunistic locking

====================================================================

.. contents:: :depth: 3

Current state and shortcomings

------------------------------

As of Ganeti 2.6, instance creations acquire all node locks when an

:doc:`instance allocator <iallocator>` (henceforth "iallocator") is

used. In situations where many instance should be created in a short

timeframe, there is a lot of congestion on node locks. Effectively all

instance creations are serialized, even on big clusters with multiple

groups.

The situation gets worse when disk wiping is enabled (see

:manpage:`gnt-cluster(8)`) as that can take, depending on disk size and

hardware performance, from minutes to hours. Not waiting for DRBD disks

to synchronize (``wait_for_sync=false``) makes instance creations

slightly faster, but there's a risk of impacting I/O of other instances.

Proposed changes

----------------

The target is to speed up instance creations in combination with an

iallocator even when the cluster's balance is sacrificed in the process.

The cluster can later be re-balanced using ``hbal``. The main objective

is to reduce the number of node locks acquired for creation and to

release un-used locks as fast as possible (the latter is already being

done). To do this safely, several changes are necessary.

Locking library

~~~~~~~~~~~~~~~

Instead of forcibly acquiring all node locks for creating an instance

using an iallocator, only those currently available will be acquired.

To this end, the locking library must be extended to implement

opportunistic locking. Lock sets must be able to only acquire all locks

available at the time, ignoring and not waiting for those held by

another thread.

Locks (``SharedLock``) already support a timeout of zero. The latter is

different from a blocking acquisition, in which case the timeout would

be ``None``.

Lock sets can essentially be acquired in two different modes. One is to

acquire the whole set, which in turn will also block adding new locks

from other threads, and the other is to acquire specific locks by name.

The function to acquire locks in a set accepts a timeout which, if not

``None`` for blocking acquisitions, counts for the whole duration of

acquiring, if necessary, the lock set's internal lock, as well as the

member locks. For opportunistic acquisitions the timeout is only

meaningful when acquiring the whole set, in which case it is only used

for acquiring the set's internal lock (used to block lock additions).

For acquiring member locks the timeout is effectively zero to make them

opportunistic.

A new and optional boolean parameter named ``opportunistic`` is added to

``LockSet.acquire`` and re-exported through

``GanetiLockManager.acquire`` for use by ``mcpu``. Internally, lock sets

do the lock acquisition using a helper function, ``__acquire_inner``. It

will be extended to support opportunistic acquisitions. The algorithm is

very similar to acquiring the whole set with the difference that

acquisitions timing out will be ignored (the timeout in this case is

zero).

New lock level

~~~~~~~~~~~~~~

With opportunistic locking used for instance creations (controlled by a

parameter), multiple such requests can start at (essentially) the same

time and compete for node locks. Some logical units, such as

``LUClusterVerifyGroup``, need to acquire all node locks. In the latter

case all instance allocations would fail to get their locks. This also

applies when multiple instance creations are started at roughly the same

time.

To avoid situations where an opcode holding all or many node locks

causes allocations to fail, a new lock level must be added to control

allocations. The logical units for instance failover and migration can

only safely determine whether they need all node locks after the

instance lock has been acquired. Therefore the new lock level, named

"node-alloc" (shorthand for "node-allocation") will be inserted after

instances (``LEVEL_INSTANCE``) and before node groups

(``LEVEL_NODEGROUP``). Similar to the "big cluster lock" ("BGL") there

is only a single lock at this level whose name is "node allocation lock"

("NAL").

As a rule-of-thumb, the node allocation lock must be acquired in the

same mode as nodes and/or node resources. If all or a large number of

node locks are acquired, the node allocation lock should be acquired as

well. Special attention should be given to logical units started for all

node groups, such as ``LUGroupVerifyDisks``, as they also block many

nodes over a short amount of time.

iallocator

~~~~~~~~~~

The :doc:`iallocator interface <iallocator>` does not need any

modification. When an instance is created, the information for all nodes

is passed to the iallocator plugin. Nodes for which the lock couldn't be

acquired and therefore shouldn't be used for the instance in question,

will be shown as offline.

Opcodes

~~~~~~~

The opcodes ``OpInstanceCreate`` and ``OpInstanceMultiAlloc`` will gain

a new parameter to enable opportunistic locking. By default this mode is

disabled as to not break backwards compatibility.

A new error type is added to describe a temporary lack of resources. Its

name will be ``ECODE_TEMP_NORES``. With opportunistic locks the opcodes

mentioned before only have a partial view of the cluster and can no

longer decide if an instance could not be allocated due to the locks it

has been given or whether the whole cluster is lacking resources.

Therefore it is required, upon encountering the error code for a

temporary lack of resources, for the job submitter to make this decision

by re-submitting the job or by re-directing it to another cluster.

.. vim: set textwidth=72 :

.. Local Variables:

.. mode: rst

.. fill-column: 72

.. End: