The 2.0 version will constitute a rewrite of the 'core' architecture,
paving the way for additional features in future 2.x versions.
-.. contents::
+.. contents:: :depth: 3
Objective
=========
Background
==========
-While Ganeti 1.2 is usable, it severly limits the flexibility of the
+While Ganeti 1.2 is usable, it severely limits the flexibility of the
cluster administration and imposes a very rigid model. It has the
following main scalability issues:
- poor handling of node failures in the cluster
- mixing hypervisors in a cluster not allowed
-It also has a number of artificial restrictions, due to historical design:
+It also has a number of artificial restrictions, due to historical
+design:
- fixed number of disks (two) per instance
-- fixed number of nics
+- fixed number of NICs
.. [#] Replace disks will release the lock, but this is an exception
and not a recommended way to operate
The 2.0 version is intended to address some of these problems, and
-create a more flexible codebase for future developments.
+create a more flexible code base for future developments.
+
+Among these problems, the single-operation at a time restriction is
+biggest issue with the current version of Ganeti. It is such a big
+impediment in operating bigger clusters that many times one is tempted
+to remove the lock just to do a simple operation like start instance
+while an OS installation is running.
Scalability problems
--------------------
- It is impossible for two people to efficiently interact with a cluster
(for example for debugging) at the same time.
-- When batch jobs are running it's impossible to do other work (for example
- failovers/fixes) on a cluster.
+- When batch jobs are running it's impossible to do other work (for
+ example failovers/fixes) on a cluster.
This poses scalability problems: as clusters grow in node and instance
size it's a lot more likely that operations which one could conceive
One of the main causes of this global lock (beside the higher
difficulty of ensuring data consistency in a more granular lock model)
-is the fact that currently there is no "master" daemon in Ganeti. Each
-command tries to acquire the so called *cmd* lock and when it
-succeeds, it takes complete ownership of the cluster configuration and
-state.
+is the fact that currently there is no long-lived process in Ganeti
+that can coordinate multiple operations. Each command tries to acquire
+the so called *cmd* lock and when it succeeds, it takes complete
+ownership of the cluster configuration and state.
Other scalability problems are due the design of the DRBD device
model, which assumed at its creation a low (one to four) number of
touches multiple areas (configuration, import/export, command line)
that it's more fitted to a major release than a minor one.
+Architecture issues
+-------------------
+
+The fact that each command is a separate process that reads the
+cluster state, executes the command, and saves the new state is also
+an issue on big clusters where the configuration data for the cluster
+begins to be non-trivial in size.
+
Overview
========
won't change (e.g. start instance will do the same things, the way
these operations are scheduled internally will change radically.
+The new design will change the cluster architecture to:
+
+.. digraph:: "ganeti-2.0-architecture"
+
+ compound=false
+ concentrate=true
+ mclimit=100.0
+ nslimit=100.0
+ edge[fontsize="8" fontname="Helvetica-Oblique"]
+ node[width="0" height="0" fontsize="12" fontcolor="black" shape=rect]
+
+ subgraph outside {
+ rclient[label="external clients"]
+ label="Outside the cluster"
+ }
+
+ subgraph cluster_inside {
+ label="ganeti cluster"
+ labeljust=l
+ subgraph cluster_master_node {
+ label="master node"
+ rapi[label="RAPI daemon"]
+ cli[label="CLI"]
+ watcher[label="Watcher"]
+ burnin[label="Burnin"]
+ masterd[shape=record style=filled label="{ <luxi> luxi endpoint | master I/O thread | job queue | {<w1> worker| <w2> worker | <w3> worker }}"]
+ {rapi;cli;watcher;burnin} -> masterd:luxi [label="LUXI" labelpos=100]
+ }
+
+ subgraph cluster_nodes {
+ label="nodes"
+ noded1 [shape=record label="{ RPC listener | Disk management | Network management | Hypervisor } "]
+ noded2 [shape=record label="{ RPC listener | Disk management | Network management | Hypervisor } "]
+ noded3 [shape=record label="{ RPC listener | Disk management | Network management | Hypervisor } "]
+ }
+ masterd:w2 -> {noded1;noded2;noded3} [label="node RPC"]
+ cli -> {noded1;noded2;noded3} [label="SSH"]
+ }
+
+ rclient -> rapi [label="RAPI protocol"]
+
+This differs from the 1.2 architecture by the addition of the master
+daemon, which will be the only entity to talk to the node daemons.
+
+
Detailed design
===============
The main changes will be switching from a per-process model to a
daemon based model, where the individual gnt-* commands will be
-clients that talk to this daemon (see the design-2.0-master-daemon
-document). This will allow us to get rid of the global cluster lock
-for most operations, having instead a per-object lock (see
-design-2.0-granular-locking). Also, the daemon will be able to queue
-jobs, and this will allow the invidual clients to submit jobs without
-waiting for them to finish, and also see the result of old requests
-(see design-2.0-job-queue).
+clients that talk to this daemon (see `Master daemon`_). This will
+allow us to get rid of the global cluster lock for most operations,
+having instead a per-object lock (see `Granular locking`_). Also, the
+daemon will be able to queue jobs, and this will allow the individual
+clients to submit jobs without waiting for them to finish, and also
+see the result of old requests (see `Job Queue`_).
Beside these major changes, another 'core' change but that will not be
as visible to the users will be changing the model of object attribute
-storage, and separate that into namespaces (such that an Xen PVM
+storage, and separate that into name spaces (such that an Xen PVM
instance will not have the Xen HVM parameters). This will allow future
-flexibility in defining additional parameters. More details in the
-design-2.0-cluster-parameters document.
+flexibility in defining additional parameters. For more details see
+`Object parameters`_.
The various changes brought in by the master daemon model and the
read-write RAPI will require changes to the cluster security; we move
-away from Twisted and use http(s) for intra- and extra-cluster
+away from Twisted and use HTTP(s) for intra- and extra-cluster
communications. For more details, see the security document in the
doc/ directory.
- the command line tools (on the master node)
- the RAPI daemon (on the master node)
-Interaction paths are between:
+The master-daemon related interaction paths are:
-- (CLI tools/RAPI daemon) and the master daemon, via the so called *luxi* API
+- (CLI tools/RAPI daemon) and the master daemon, via the so called
+ *LUXI* API
- the master daemon and the node daemons, via the node RPC
+There are also some additional interaction paths for exceptional cases:
+
+- CLI tools might access via SSH the nodes (for ``gnt-cluster copyfile``
+ and ``gnt-cluster command``)
+- master failover is a special case when a non-master node will SSH
+ and do node-RPC calls to the current master
+
The protocol between the master daemon and the node daemons will be
-changed to HTTP(S), using a simple PUT/GET of JSON-encoded
-messages. This is done due to difficulties in working with the Twisted
-framework and its protocols in a multithreaded environment, which we
-can overcome by using a simpler stack (see the caveats section). The
-protocol between the CLI/RAPI and the master daemon will be a custom
-one (called *luxi*): on a UNIX socket on the master node, with rights
-restricted by filesystem permissions, the CLI/RAPI will talk to the
-master daemon using JSON-encoded messages.
+changed from (Ganeti 1.2) Twisted PB (perspective broker) to HTTP(S),
+using a simple PUT/GET of JSON-encoded messages. This is done due to
+difficulties in working with the Twisted framework and its protocols
+in a multithreaded environment, which we can overcome by using a
+simpler stack (see the caveats section).
+
+The protocol between the CLI/RAPI and the master daemon will be a
+custom one (called *LUXI*): on a UNIX socket on the master node, with
+rights restricted by filesystem permissions, the CLI/RAPI will talk to
+the master daemon using JSON-encoded messages.
The operations supported over this internal protocol will be encoded
via a python library that will expose a simple API for its
users. Internally, the protocol will simply encode all objects in JSON
format and decode them on the receiver side.
+For more details about the RAPI daemon see `Remote API changes`_, and
+for the node daemon see `Node daemon changes`_.
+
+.. _luxi:
+
The LUXI protocol
+++++++++++++++++
-We will have two main classes of operations over the master daemon API:
+As described above, the protocol for making requests or queries to the
+master daemon will be a UNIX-socket based simple RPC of JSON-encoded
+messages.
+
+The choice of UNIX was in order to get rid of the need of
+authentication and authorisation inside Ganeti; for 2.0, the
+permissions on the Unix socket itself will determine the access
+rights.
+
+We will have two main classes of operations over this API:
- cluster query functions
- job related functions
The cluster query functions are usually short-duration, and are the
-equivalent of the OP_QUERY_* opcodes in ganeti 1.2 (and they are
+equivalent of the ``OP_QUERY_*`` opcodes in Ganeti 1.2 (and they are
internally implemented still with these opcodes). The clients are
guaranteed to receive the response in a reasonable time via a timeout.
- archive job (see the job queue design doc)
- wait for job change, which allows a client to wait without polling
-For more details, see the job queue design document.
+For more details of the actual operation list, see the `Job Queue`_.
-Daemon implementation
-+++++++++++++++++++++
+Both requests and responses will consist of a JSON-encoded message
+followed by the ``ETX`` character (ASCII decimal 3), which is not a
+valid character in JSON messages and thus can serve as a message
+delimiter. The contents of the messages will be a dictionary with two
+fields:
+
+:method:
+ the name of the method called
+:args:
+ the arguments to the method, as a list (no keyword arguments allowed)
+
+Responses will follow the same format, with the two fields being:
+
+:success:
+ a boolean denoting the success of the operation
+:result:
+ the actual result, or error message in case of failure
+
+There are two special value for the result field:
+
+- in the case that the operation failed, and this field is a list of
+ length two, the client library will try to interpret is as an
+ exception, the first element being the exception type and the second
+ one the actual exception arguments; this will allow a simple method of
+ passing Ganeti-related exception across the interface
+- for the *WaitForChange* call (that waits on the server for a job to
+ change status), if the result is equal to ``nochange`` instead of the
+ usual result for this call (a list of changes), then the library will
+ internally retry the call; this is done in order to differentiate
+ internally between master daemon hung and job simply not changed
+
+Users of the API that don't use the provided python library should
+take care of the above two cases.
+
+
+Master daemon implementation
+++++++++++++++++++++++++++++
The daemon will be based around a main I/O thread that will wait for
new requests from the clients, and that does the setup/shutdown of the
long-lived, started at daemon startup and terminated only at shutdown
time
- client I/O threads, which are the ones that talk the local protocol
- to the clients
+ (LUXI) to the clients, and are short-lived
Master startup/failover
+++++++++++++++++++++++
determined by the serial number on the configuration and
highest job ID on the job queue)
- - there is not even a single node having a newer
- configuration file
-
- if we are not failing over (but just starting), the
quorum agrees that we are the designated master
+ - if any of the above is false, we prevent the current operation
+ (i.e. we don't become the master)
+
#. at this point, the node transitions to the master role
#. for all the in-progress jobs, mark them as failed, with
reason unknown or something similar (master failed, etc.)
+Since due to exceptional conditions we could have a situation in which
+no node can become the master due to inconsistent data, we will have
+an override switch for the master daemon startup that will assume the
+current node has the right data and will replicate all the
+configuration files to the other nodes.
+
+**Note**: the above algorithm is by no means an election algorithm; it
+is a *confirmation* of the master role currently held by a node.
Logging
+++++++
-The logging system will be switched completely to the logging module;
-currently it's logging-based, but exposes a different API, which is
-just overhead. As such, the code will be switched over to standard
-logging calls, and only the setup will be custom.
+The logging system will be switched completely to the standard python
+logging module; currently it's logging-based, but exposes a different
+API, which is just overhead. As such, the code will be switched over
+to standard logging calls, and only the setup will be custom.
With this change, we will remove the separate debug/info/error logs,
and instead have always one logfile per daemon model:
- node-daemon.log for the node daemon (this is the same as in 1.2)
- rapi-daemon.log for the RAPI daemon logs
- rapi-access.log, an additional log file for the RAPI that will be
- in the standard http log format for possible parsing by other tools
+ in the standard HTTP log format for possible parsing by other tools
-Since the watcher will only submit jobs to the master for startup of
-the instances, its log file will contain less information than before,
-mainly that it will start the instance, but not the results.
+Since the :term:`watcher` will only submit jobs to the master for
+startup of the instances, its log file will contain less information
+than before, mainly that it will start the instance, but not the
+results.
+
+Node daemon changes
++++++++++++++++++++
+
+The only change to the node daemon is that, since we need better
+concurrency, we don't process the inter-node RPC calls in the node
+daemon itself, but we fork and process each request in a separate
+child.
+
+Since we don't have many calls, and we only fork (not exec), the
+overhead should be minimal.
Caveats
+++++++
much better served by a daemon-based model
Another area of discussion is moving away from Twisted in this new
-implementation. While Twisted hase its advantages, there are also many
-disatvantanges to using it:
+implementation. While Twisted has its advantages, there are also many
+disadvantages to using it:
- first and foremost, it's not a library, but a framework; thus, if
- you use twisted, all the code needs to be 'twiste-ized'; we were able
- to keep the 1.x code clean by hacking around twisted in an
- unsupported, unrecommended way, and the only alternative would have
- been to make all the code be written for twisted
-- it has some weaknesses in working with multiple threads, since its base
- model is designed to replace thread usage by using deferred calls, so while
- it can use threads, it's not less flexible in doing so
-
-And, since we already have an HTTP server library for the RAPI, we
-can just reuse that for inter-node communication.
+ you use twisted, all the code needs to be 'twiste-ized' and written
+ in an asynchronous manner, using deferreds; while this method works,
+ it's not a common way to code and it requires that the entire process
+ workflow is based around a single *reactor* (Twisted name for a main
+ loop)
+- the more advanced granular locking that we want to implement would
+ require, if written in the async-manner, deep integration with the
+ Twisted stack, to such an extend that business-logic is inseparable
+ from the protocol coding; we felt that this is an unreasonable
+ request, and that a good protocol library should allow complete
+ separation of low-level protocol calls and business logic; by
+ comparison, the threaded approach combined with HTTPs protocol
+ required (for the first iteration) absolutely no changes from the 1.2
+ code, and later changes for optimizing the inter-node RPC calls
+ required just syntactic changes (e.g. ``rpc.call_...`` to
+ ``self.rpc.call_...``)
+
+Another issue is with the Twisted API stability - during the Ganeti
+1.x lifetime, we had to to implement many times workarounds to changes
+in the Twisted version, so that for example 1.2 is able to use both
+Twisted 2.x and 8.x.
+
+In the end, since we already had an HTTP server library for the RAPI,
+we just reused that for inter-node communication.
Granular locking
~~~~~~~~~~~~~~~~
-We want to make sure that multiple operations can run in parallel on a Ganeti
-Cluster. In order for this to happen we need to make sure concurrently run
-operations don't step on each other toes and break the cluster.
+We want to make sure that multiple operations can run in parallel on a
+Ganeti Cluster. In order for this to happen we need to make sure
+concurrently run operations don't step on each other toes and break the
+cluster.
This design addresses how we are going to deal with locking so that:
-- high urgency operations are not stopped by long length ones
-- long length operations can run in parallel
-- we preserve safety (data coherency) and liveness (no deadlock, no work
- postponed indefinitely) on the cluster
-
-Reaching the maximum possible parallelism is a Non-Goal. We have identified a
-set of operations that are currently bottlenecks and need to be parallelised
-and have worked on those. In the future it will be possible to address other
-needs, thus making the cluster more and more parallel one step at a time.
+- we preserve data coherency
+- we prevent deadlocks
+- we prevent job starvation
-This document only talks about parallelising Ganeti level operations, aka
-Logical Units, and the locking needed for that. Any other synchronisation lock
-needed internally by the code is outside its scope.
+Reaching the maximum possible parallelism is a Non-Goal. We have
+identified a set of operations that are currently bottlenecks and need
+to be parallelised and have worked on those. In the future it will be
+possible to address other needs, thus making the cluster more and more
+parallel one step at a time.
-Ganeti 1.2
-++++++++++
+This section only talks about parallelising Ganeti level operations, aka
+Logical Units, and the locking needed for that. Any other
+synchronization lock needed internally by the code is outside its scope.
-We intend to implement a Ganeti locking library, which can be used by the
-various ganeti code components in order to easily, efficiently and correctly
-grab the locks they need to perform their function.
+Library details
++++++++++++++++
The proposed library has these features:
-- Internally managing all the locks, making the implementation transparent
- from their usage
-- Automatically grabbing multiple locks in the right order (avoid deadlock)
-- Ability to transparently handle conversion to more granularity
-- Support asynchronous operation (future goal)
-
-Locking will be valid only on the master node and will not be a distributed
-operation. In case of master failure, though, if some locks were held it means
-some opcodes were in progress, so when recovery of the job queue is done it
-will be possible to determine by the interrupted opcodes which operations could
-have been left half way through and thus which locks could have been held. It
-is then the responsibility either of the master failover code, of the cluster
-verification code, or of the admin to do what's necessary to make sure that any
-leftover state is dealt with. This is not an issue from a locking point of view
-because the fact that the previous master has failed means that it cannot do
-any job.
-
-A corollary of this is that a master-failover operation with both masters alive
-needs to happen while no other locks are held.
+- internally managing all the locks, making the implementation
+ transparent from their usage
+- automatically grabbing multiple locks in the right order (avoid
+ deadlock)
+- ability to transparently handle conversion to more granularity
+- support asynchronous operation (future goal)
+
+Locking will be valid only on the master node and will not be a
+distributed operation. Therefore, in case of master failure, the
+operations currently running will be aborted and the locks will be
+lost; it remains to the administrator to cleanup (if needed) the
+operation result (e.g. make sure an instance is either installed
+correctly or removed).
+
+A corollary of this is that a master-failover operation with both
+masters alive needs to happen while no operations are running, and
+therefore no locks are held.
+
+All the locks will be represented by objects (like
+``lockings.SharedLock``), and the individual locks for each object
+will be created at initialisation time, from the config file.
+
+The API will have a way to grab one or more than one locks at the same
+time. Any attempt to grab a lock while already holding one in the wrong
+order will be checked for, and fail.
+
The Locks
+++++++++
- One lock per node in the cluster
- One lock per instance in the cluster
-All the instance locks will need to be taken before the node locks, and the
-node locks before the config lock. Locks will need to be acquired at the same
-time for multiple instances and nodes, and internal ordering will be dealt
-within the locking library, which, for simplicity, will just use alphabetical
-order.
-
-Handling conversion to more granularity
-+++++++++++++++++++++++++++++++++++++++
+All the instance locks will need to be taken before the node locks, and
+the node locks before the config lock. Locks will need to be acquired at
+the same time for multiple instances and nodes, and internal ordering
+will be dealt within the locking library, which, for simplicity, will
+just use alphabetical order.
-In order to convert to a more granular approach transparently each time we
-split a lock into more we'll create a "metalock", which will depend on those
-sublocks and live for the time necessary for all the code to convert (or
-forever, in some conditions). When a metalock exists all converted code must
-acquire it in shared mode, so it can run concurrently, but still be exclusive
-with old code, which acquires it exclusively.
+Each lock has the following three possible statuses:
-In the beginning the only such lock will be what replaces the current "command"
-lock, and will acquire all the locks in the system, before proceeding. This
-lock will be called the "Big Ganeti Lock" because holding that one will avoid
-any other concurrent ganeti operations.
+- unlocked (anyone can grab the lock)
+- shared (anyone can grab/have the lock but only in shared mode)
+- exclusive (no one else can grab/have the lock)
-We might also want to devise more metalocks (eg. all nodes, all nodes+config)
-in order to make it easier for some parts of the code to acquire what it needs
-without specifying it explicitly.
+Handling conversion to more granularity
++++++++++++++++++++++++++++++++++++++++
-In the future things like the node locks could become metalocks, should we
-decide to split them into an even more fine grained approach, but this will
-probably be only after the first 2.0 version has been released.
+In order to convert to a more granular approach transparently each time
+we split a lock into more we'll create a "metalock", which will depend
+on those sub-locks and live for the time necessary for all the code to
+convert (or forever, in some conditions). When a metalock exists all
+converted code must acquire it in shared mode, so it can run
+concurrently, but still be exclusive with old code, which acquires it
+exclusively.
-Library API
-+++++++++++
+In the beginning the only such lock will be what replaces the current
+"command" lock, and will acquire all the locks in the system, before
+proceeding. This lock will be called the "Big Ganeti Lock" because
+holding that one will avoid any other concurrent Ganeti operations.
-All the locking will be its own class, and the locks will be created at
-initialisation time, from the config file.
+We might also want to devise more metalocks (eg. all nodes, all
+nodes+config) in order to make it easier for some parts of the code to
+acquire what it needs without specifying it explicitly.
-The API will have a way to grab one or more than one locks at the same time.
-Any attempt to grab a lock while already holding one in the wrong order will be
-checked for, and fail.
+In the future things like the node locks could become metalocks, should
+we decide to split them into an even more fine grained approach, but
+this will probably be only after the first 2.0 version has been
+released.
Adding/Removing locks
+++++++++++++++++++++
-When a new instance or a new node is created an associated lock must be added
-to the list. The relevant code will need to inform the locking library of such
-a change.
+When a new instance or a new node is created an associated lock must be
+added to the list. The relevant code will need to inform the locking
+library of such a change.
-This needs to be compatible with every other lock in the system, especially
-metalocks that guarantee to grab sets of resources without specifying them
-explicitly. The implementation of this will be handled in the locking library
-itself.
+This needs to be compatible with every other lock in the system,
+especially metalocks that guarantee to grab sets of resources without
+specifying them explicitly. The implementation of this will be handled
+in the locking library itself.
-Of course when instances or nodes disappear from the cluster the relevant locks
-must be removed. This is easier than adding new elements, as the code which
-removes them must own them exclusively or can queue for their ownership, and
-thus deals with metalocks exactly as normal code acquiring those locks. Any
-operation queueing on a removed lock will fail after its removal.
+When instances or nodes disappear from the cluster the relevant locks
+must be removed. This is easier than adding new elements, as the code
+which removes them must own them exclusively already, and thus deals
+with metalocks exactly as normal code acquiring those locks. Any
+operation queuing on a removed lock will fail after its removal.
Asynchronous operations
+++++++++++++++++++++++
For the first version the locking library will only export synchronous
-operations, which will block till the needed lock are held, and only fail if
-the request is impossible or somehow erroneous.
+operations, which will block till the needed lock are held, and only
+fail if the request is impossible or somehow erroneous.
In the future we may want to implement different types of asynchronous
operations such as:
-- Try to acquire this lock set and fail if not possible
-- Try to acquire one of these lock sets and return the first one you were
- able to get (or after a timeout) (select/poll like)
+- try to acquire this lock set and fail if not possible
+- try to acquire one of these lock sets and return the first one you
+ were able to get (or after a timeout) (select/poll like)
-These operations can be used to prioritize operations based on available locks,
-rather than making them just blindly queue for acquiring them. The inherent
-risk, though, is that any code using the first operation, or setting a timeout
-for the second one, is susceptible to starvation and thus may never be able to
-get the required locks and complete certain tasks. Considering this
-providing/using these operations should not be among our first priorities.
+These operations can be used to prioritize operations based on available
+locks, rather than making them just blindly queue for acquiring them.
+The inherent risk, though, is that any code using the first operation,
+or setting a timeout for the second one, is susceptible to starvation
+and thus may never be able to get the required locks and complete
+certain tasks. Considering this providing/using these operations should
+not be among our first priorities.
Locking granularity
+++++++++++++++++++
For the first version of this code we'll convert each Logical Unit to
-acquire/release the locks it needs, so locking will be at the Logical Unit
-level. In the future we may want to split logical units in independent
-"tasklets" with their own locking requirements. A different design doc (or mini
-design doc) will cover the move from Logical Units to tasklets.
+acquire/release the locks it needs, so locking will be at the Logical
+Unit level. In the future we may want to split logical units in
+independent "tasklets" with their own locking requirements. A different
+design doc (or mini design doc) will cover the move from Logical Units
+to tasklets.
-Lock acquisition code path
-++++++++++++++++++++++++++
+Code examples
++++++++++++++
-In general when acquiring locks we should use a code path equivalent to::
+In general when acquiring locks we should use a code path equivalent
+to::
lock.acquire()
try:
finally:
lock.release()
-This makes sure we release all locks, and avoid possible deadlocks. Of course
-extra care must be used not to leave, if possible locked structures in an
-unusable state.
+This makes sure we release all locks, and avoid possible deadlocks. Of
+course extra care must be used not to leave, if possible locked
+structures in an unusable state. Note that with Python 2.5 a simpler
+syntax will be possible, but we want to keep compatibility with Python
+2.4 so the new constructs should not be used.
-In order to avoid this extra indentation and code changes everywhere in the
-Logical Units code, we decided to allow LUs to declare locks, and then execute
-their code with their locks acquired. In the new world LUs are called like
-this::
+In order to avoid this extra indentation and code changes everywhere in
+the Logical Units code, we decided to allow LUs to declare locks, and
+then execute their code with their locks acquired. In the new world LUs
+are called like this::
# user passed names are expanded to the internal lock/resource name,
# then known needed locks are declared
lu.Exec()
... locks declared for removal are removed, all acquired locks released ...
-The Processor and the LogicalUnit class will contain exact documentation on how
-locks are supposed to be declared.
+The Processor and the LogicalUnit class will contain exact documentation
+on how locks are supposed to be declared.
Caveats
+++++++
This library will provide an easy upgrade path to bring all the code to
granular locking without breaking everything, and it will also guarantee
-against a lot of common errors. Code switching from the old "lock everything"
-lock to the new system, though, needs to be carefully scrutinised to be sure it
-is really acquiring all the necessary locks, and none has been overlooked or
-forgotten.
+against a lot of common errors. Code switching from the old "lock
+everything" lock to the new system, though, needs to be carefully
+scrutinised to be sure it is really acquiring all the necessary locks,
+and none has been overlooked or forgotten.
-The code can contain other locks outside of this library, to synchronise other
-threaded code (eg for the job queue) but in general these should be leaf locks
-or carefully structured non-leaf ones, to avoid deadlock race conditions.
+The code can contain other locks outside of this library, to synchronise
+other threaded code (eg for the job queue) but in general these should
+be leaf locks or carefully structured non-leaf ones, to avoid deadlock
+race conditions.
+.. _jqueue-original-design:
+
Job Queue
~~~~~~~~~
queue to store these and to be able to process as many as possible in
parallel.
-A ganeti job will consist of multiple ``OpCodes`` which are the basic
+A Ganeti job will consist of multiple ``OpCodes`` which are the basic
element of operation in Ganeti 1.2 (and will remain as such). Most
command-level commands are equivalent to one OpCode, or in some cases
to a sequence of opcodes, all of the same type (e.g. evacuating a node
Job execution—“Life of a Ganeti job”
++++++++++++++++++++++++++++++++++++
-#. Job gets submitted by the client. A new job identifier is generated and
- assigned to the job. The job is then automatically replicated [#replic]_
- to all nodes in the cluster. The identifier is returned to the client.
-#. A pool of worker threads waits for new jobs. If all are busy, the job has
- to wait and the first worker finishing its work will grab it. Otherwise any
- of the waiting threads will pick up the new job.
-#. Client waits for job status updates by calling a waiting RPC function.
- Log message may be shown to the user. Until the job is started, it can also
- be cancelled.
-#. As soon as the job is finished, its final result and status can be retrieved
- from the server.
+#. Job gets submitted by the client. A new job identifier is generated
+ and assigned to the job. The job is then automatically replicated
+ [#replic]_ to all nodes in the cluster. The identifier is returned to
+ the client.
+#. A pool of worker threads waits for new jobs. If all are busy, the job
+ has to wait and the first worker finishing its work will grab it.
+ Otherwise any of the waiting threads will pick up the new job.
+#. Client waits for job status updates by calling a waiting RPC
+ function. Log message may be shown to the user. Until the job is
+ started, it can also be canceled.
+#. As soon as the job is finished, its final result and status can be
+ retrieved from the server.
#. If the client archives the job, it gets moved to a history directory.
There will be a method to archive all jobs older than a a given age.
-.. [#replic] We need replication in order to maintain the consistency across
- all nodes in the system; the master node only differs in the fact that
- now it is running the master daemon, but it if fails and we do a master
- failover, the jobs are still visible on the new master (though marked as
- failed).
+.. [#replic] We need replication in order to maintain the consistency
+ across all nodes in the system; the master node only differs in the
+ fact that now it is running the master daemon, but it if fails and we
+ do a master failover, the jobs are still visible on the new master
+ (though marked as failed).
Failures to replicate a job to other nodes will be only flagged as
errors in the master daemon log if more than half of the nodes failed,
- a file can be atomically replaced
- a file can easily be replicated to other nodes
-- checking consistency across nodes can be implemented very easily, since
- all job files should be (at a given moment in time) identical
+- checking consistency across nodes can be implemented very easily,
+ since all job files should be (at a given moment in time) identical
The other possible choices that were discussed and discounted were:
-- single big file with all job data: not feasible due to difficult updates
+- single big file with all job data: not feasible due to difficult
+ updates
- in-process databases: hard to replicate the entire database to the
- other nodes, and replicating individual operations does not mean wee keep
- consistency
+ other nodes, and replicating individual operations does not mean wee
+ keep consistency
Queue structure
+++++++++++++++
-All file operations have to be done atomically by writing to a temporary file
-and subsequent renaming. Except for log messages, every change in a job is
-stored and replicated to other nodes.
+All file operations have to be done atomically by writing to a temporary
+file and subsequent renaming. Except for log messages, every change in a
+job is stored and replicated to other nodes.
::
Locking
+++++++
-Locking in the job queue is a complicated topic. It is called from more than
-one thread and must be thread-safe. For simplicity, a single lock is used for
-the whole job queue.
+Locking in the job queue is a complicated topic. It is called from more
+than one thread and must be thread-safe. For simplicity, a single lock
+is used for the whole job queue.
-A more detailed description can be found in doc/locking.txt.
+A more detailed description can be found in doc/locking.rst.
Internal RPC
Client RPC
++++++++++
-RPC between Ganeti clients and the Ganeti master daemon supports the following
-operations:
+RPC between Ganeti clients and the Ganeti master daemon supports the
+following operations:
SubmitJob(ops)
- Submits a list of opcodes and returns the job identifier. The identifier is
- guaranteed to be unique during the lifetime of a cluster.
+ Submits a list of opcodes and returns the job identifier. The
+ identifier is guaranteed to be unique during the lifetime of a
+ cluster.
WaitForJobChange(job_id, fields, […], timeout)
- This function waits until a job changes or a timeout expires. The condition
- for when a job changed is defined by the fields passed and the last log
- message received.
+ This function waits until a job changes or a timeout expires. The
+ condition for when a job changed is defined by the fields passed and
+ the last log message received.
QueryJobs(job_ids, fields)
Returns field values for the job identifiers passed.
CancelJob(job_id)
- Cancels the job specified by identifier. This operation may fail if the job
- is already running, canceled or finished.
+ Cancels the job specified by identifier. This operation may fail if
+ the job is already running, canceled or finished.
ArchiveJob(job_id)
- Moves a job into the …/archive/ directory. This operation will fail if the
- job has not been canceled or finished.
+ Moves a job into the …/archive/ directory. This operation will fail if
+ the job has not been canceled or finished.
Job and opcode status
Error
The job/opcode was aborted with an error.
-If the master is aborted while a job is running, the job will be set to the
-Error status once the master started again.
+If the master is aborted while a job is running, the job will be set to
+the Error status once the master started again.
History
+++++++
Archived jobs are kept in a separate directory,
-/var/lib/ganeti/queue/archive/. This is done in order to speed up the
-queue handling: by default, the jobs in the archive are not touched by
-any functions. Only the current (unarchived) jobs are parsed, loaded,
-and verified (if implemented) by the master daemon.
+``/var/lib/ganeti/queue/archive/``. This is done in order to speed up
+the queue handling: by default, the jobs in the archive are not
+touched by any functions. Only the current (unarchived) jobs are
+parsed, loaded, and verified (if implemented) by the master daemon.
Ganeti updates
way to prevent new jobs entering the queue.
-
Object parameters
~~~~~~~~~~~~~~~~~
a hypervisor parameter (or hypervisor specific parameter) is defined
as a parameter that is interpreted by the hypervisor support code in
Ganeti and usually is specific to a particular hypervisor (like the
- kernel path for PVM which makes no sense for HVM).
+ kernel path for :term:`PVM` which makes no sense for :term:`HVM`).
:backend parameter:
a backend parameter is defined as an instance parameter that can be
For example: memory, vcpus, auto_balance
- All these parameters will be encoded into constants.py with the prefix "BE\_"
- and the whole list of parameters will exist in the set "BES_PARAMETERS"
+ All these parameters will be encoded into constants.py with the prefix
+ "BE\_" and the whole list of parameters will exist in the set
+ "BES_PARAMETERS"
:proper parameter:
- a parameter whose value is unique to the instance (e.g. the name of a LV,
- or the MAC of a NIC)
+ a parameter whose value is unique to the instance (e.g. the name of a
+ LV, or the MAC of a NIC)
As a general rule, for all kind of parameters, “None” (or in
JSON-speak, “nil”) will no longer be a valid value for a parameter. As
hold defaults for the instances:
- hvparams, a dictionary indexed by hypervisor type, holding default
- values for hypervisor parameters that are not defined/overrided by
+ values for hypervisor parameters that are not defined/overridden by
the instances of this hypervisor type
- beparams, a dictionary holding (for 2.0) a single element 'default',
Node-related parameters are very few, and we will continue using the
same model for these as previously (attributes on the Node object).
+There are three new node flags, described in a separate section "node
+flags" below.
+
Instance parameters
+++++++++++++++++++
The names for hypervisor parameters in the instance.hvparams subtree
should be choosen as generic as possible, especially if specific
parameters could conceivably be useful for more than one hypervisor,
-e.g. instance.hvparams.vnc_console_port instead of using both
-instance.hvparams.hvm_vnc_console_port and
-instance.hvparams.kvm_vnc_console_port.
+e.g. ``instance.hvparams.vnc_console_port`` instead of using both
+``instance.hvparams.hvm_vnc_console_port`` and
+``instance.hvparams.kvm_vnc_console_port``.
There are some special cases related to disks and NICs (for example):
-a disk has both ganeti-related parameters (e.g. the name of the LV)
+a disk has both Ganeti-related parameters (e.g. the name of the LV)
and hypervisor-related parameters (how the disk is presented to/named
in the instance). The former parameters remain as proper-instance
parameters, while the latter value are migrated to the hvparams
for this hypervisor
:CheckParamSyntax(hvparams): checks that the given parameters are
valid (as in the names are valid) for this hypervisor; usually just
- comparing hvparams.keys() and cls.PARAMETERS; this is a class method
- that can be called from within master code (i.e. cmdlib) and should
- be safe to do so
+ comparing ``hvparams.keys()`` and ``cls.PARAMETERS``; this is a class
+ method that can be called from within master code (i.e. cmdlib) and
+ should be safe to do so
:ValidateParameters(hvparams): verifies the values of the provided
parameters against this hypervisor; this is a method that will be
called on the target node, from backend.py code, and as such can
- ``Cluster.FillBE(instance, be_type="default")``, which returns the
beparams dict, based on the instance and cluster beparams
-The FillHV/BE transformations will be used, for example, in the RpcRunner
-when sending an instance for activation/stop, and the sent instance
-hvparams/beparams will have the final value (noded code doesn't know
-about defaults).
+The FillHV/BE transformations will be used, for example, in the
+RpcRunner when sending an instance for activation/stop, and the sent
+instance hvparams/beparams will have the final value (noded code doesn't
+know about defaults).
LU code will need to self-call the transformation, if needed.
The parameter changes will have impact on the OpCodes, especially on
the following ones:
-- OpCreateInstance, where the new hv and be parameters will be sent as
- dictionaries; note that all hv and be parameters are now optional, as
- the values can be instead taken from the cluster
-- OpQueryInstances, where we have to be able to query these new
+- ``OpInstanceCreate``, where the new hv and be parameters will be sent
+ as dictionaries; note that all hv and be parameters are now optional,
+ as the values can be instead taken from the cluster
+- ``OpInstanceQuery``, where we have to be able to query these new
parameters; the syntax for names will be ``hvparam/$NAME`` and
``beparam/$NAME`` for querying an individual parameter out of one
dictionary, and ``hvparams``, respectively ``beparams``, for the whole
dictionaries
-- OpModifyInstance, where the the modified parameters are sent as
+- ``OpModifyInstance``, where the the modified parameters are sent as
dictionaries
Additionally, we will need new OpCodes to modify the cluster-level
E.g. for the drbd shared secrets, we could export these with the
values replaced by an empty string.
+Node flags
+~~~~~~~~~~
+
+Ganeti 2.0 adds three node flags that change the way nodes are handled
+within Ganeti and the related infrastructure (iallocator interaction,
+RAPI data export).
+
+*master candidate* flag
++++++++++++++++++++++++
+
+Ganeti 2.0 allows more scalability in operation by introducing
+parallelization. However, a new bottleneck is reached that is the
+synchronization and replication of cluster configuration to all nodes
+in the cluster.
+
+This breaks scalability as the speed of the replication decreases
+roughly with the size of the nodes in the cluster. The goal of the
+master candidate flag is to change this O(n) into O(1) with respect to
+job and configuration data propagation.
+
+Only nodes having this flag set (let's call this set of nodes the
+*candidate pool*) will have jobs and configuration data replicated.
+
+The cluster will have a new parameter (runtime changeable) called
+``candidate_pool_size`` which represents the number of candidates the
+cluster tries to maintain (preferably automatically).
+
+This will impact the cluster operations as follows:
+
+- jobs and config data will be replicated only to a fixed set of nodes
+- master fail-over will only be possible to a node in the candidate pool
+- cluster verify needs changing to account for these two roles
+- external scripts will no longer have access to the configuration
+ file (this is not recommended anyway)
+
+
+The caveats of this change are:
+
+- if all candidates are lost (completely), cluster configuration is
+ lost (but it should be backed up external to the cluster anyway)
+
+- failed nodes which are candidate must be dealt with properly, so
+ that we don't lose too many candidates at the same time; this will be
+ reported in cluster verify
+
+- the 'all equal' concept of ganeti is no longer true
+
+- the partial distribution of config data means that all nodes will
+ have to revert to ssconf files for master info (as in 1.2)
+
+Advantages:
+
+- speed on a 100+ nodes simulated cluster is greatly enhanced, even
+ for a simple operation; ``gnt-instance remove`` on a diskless instance
+ remove goes from ~9seconds to ~2 seconds
+
+- node failure of non-candidates will be less impacting on the cluster
+
+The default value for the candidate pool size will be set to 10 but
+this can be changed at cluster creation and modified any time later.
+
+Testing on simulated big clusters with sequential and parallel jobs
+show that this value (10) is a sweet-spot from performance and load
+point of view.
+
+*offline* flag
+++++++++++++++
+
+In order to support better the situation in which nodes are offline
+(e.g. for repair) without altering the cluster configuration, Ganeti
+needs to be told and needs to properly handle this state for nodes.
+
+This will result in simpler procedures, and less mistakes, when the
+amount of node failures is high on an absolute scale (either due to
+high failure rate or simply big clusters).
+
+Nodes having this attribute set will not be contacted for inter-node
+RPC calls, will not be master candidates, and will not be able to host
+instances as primaries.
+
+Setting this attribute on a node:
+
+- will not be allowed if the node is the master
+- will not be allowed if the node has primary instances
+- will cause the node to be demoted from the master candidate role (if
+ it was), possibly causing another node to be promoted to that role
+
+This attribute will impact the cluster operations as follows:
+
+- querying these nodes for anything will fail instantly in the RPC
+ library, with a specific RPC error (RpcResult.offline == True)
+
+- they will be listed in the Other section of cluster verify
+
+The code is changed in the following ways:
+
+- RPC calls were be converted to skip such nodes:
+
+ - RpcRunner-instance-based RPC calls are easy to convert
+
+ - static/classmethod RPC calls are harder to convert, and were left
+ alone
+
+- the RPC results were unified so that this new result state (offline)
+ can be differentiated
+
+- master voting still queries in repair nodes, as we need to ensure
+ consistency in case the (wrong) masters have old data, and nodes have
+ come back from repairs
+
+Caveats:
+
+- some operation semantics are less clear (e.g. what to do on instance
+ start with offline secondary?); for now, these will just fail as if
+ the flag is not set (but faster)
+- 2-node cluster with one node offline needs manual startup of the
+ master with a special flag to skip voting (as the master can't get a
+ quorum there)
+
+One of the advantages of implementing this flag is that it will allow
+in the future automation tools to automatically put the node in
+repairs and recover from this state, and the code (should/will) handle
+this much better than just timing out. So, future possible
+improvements (for later versions):
+
+- watcher will detect nodes which fail RPC calls, will attempt to ssh
+ to them, if failure will put them offline
+- watcher will try to ssh and query the offline nodes, if successful
+ will take them off the repair list
+
+Alternatives considered: The RPC call model in 2.0 is, by default,
+much nicer - errors are logged in the background, and job/opcode
+execution is clearer, so we could simply not introduce this. However,
+having this state will make both the codepaths clearer (offline
+vs. temporary failure) and the operational model (it's not a node with
+errors, but an offline node).
+
+
+*drained* flag
+++++++++++++++
+
+Due to parallel execution of jobs in Ganeti 2.0, we could have the
+following situation:
+
+- gnt-node migrate + failover is run
+- gnt-node evacuate is run, which schedules a long-running 6-opcode
+ job for the node
+- partway through, a new job comes in that runs an iallocator script,
+ which finds the above node as empty and a very good candidate
+- gnt-node evacuate has finished, but now it has to be run again, to
+ clean the above instance(s)
+
+In order to prevent this situation, and to be able to get nodes into
+proper offline status easily, a new *drained* flag was added to the
+nodes.
+
+This flag (which actually means "is being, or was drained, and is
+expected to go offline"), will prevent allocations on the node, but
+otherwise all other operations (start/stop instance, query, etc.) are
+working without any restrictions.
+
+Interaction between flags
++++++++++++++++++++++++++
+
+While these flags are implemented as separate flags, they are
+mutually-exclusive and are acting together with the master node role
+as a single *node status* value. In other words, a flag is only in one
+of these roles at a given time. The lack of any of these flags denote
+a regular node.
+
+The current node status is visible in the ``gnt-cluster verify``
+output, and the individual flags can be examined via separate flags in
+the ``gnt-node list`` output.
+
+These new flags will be exported in both the iallocator input message
+and via RAPI, see the respective man pages for the exact names.
+
Feature changes
---------------
assumptions made initially are not true and that more flexibility is
needed.
-One main assupmtion made was that disk failures should be treated as 'rare'
-events, and that each of them needs to be manually handled in order to ensure
-data safety; however, both these assumptions are false:
+One main assumption made was that disk failures should be treated as
+'rare' events, and that each of them needs to be manually handled in
+order to ensure data safety; however, both these assumptions are false:
-- disk failures can be a common occurence, based on usage patterns or cluster
- size
-- our disk setup is robust enough (referring to DRBD8 + LVM) that we could
- automate more of the recovery
+- disk failures can be a common occurrence, based on usage patterns or
+ cluster size
+- our disk setup is robust enough (referring to DRBD8 + LVM) that we
+ could automate more of the recovery
-Note that we still don't have fully-automated disk recovery as a goal, but our
-goal is to reduce the manual work needed.
+Note that we still don't have fully-automated disk recovery as a goal,
+but our goal is to reduce the manual work needed.
As such, we plan the following main changes:
-- DRBD8 is much more flexible and stable than its previous version (0.7),
- such that removing the support for the ``remote_raid1`` template and
- focusing only on DRBD8 is easier
+- DRBD8 is much more flexible and stable than its previous version
+ (0.7), such that removing the support for the ``remote_raid1``
+ template and focusing only on DRBD8 is easier
-- dynamic discovery of DRBD devices is not actually needed in a cluster that
- where the DRBD namespace is controlled by Ganeti; switching to a static
- assignment (done at either instance creation time or change secondary time)
- will change the disk activation time from O(n) to O(1), which on big
- clusters is a significant gain
+- dynamic discovery of DRBD devices is not actually needed in a cluster
+ that where the DRBD namespace is controlled by Ganeti; switching to a
+ static assignment (done at either instance creation time or change
+ secondary time) will change the disk activation time from O(n) to
+ O(1), which on big clusters is a significant gain
-- remove the hard dependency on LVM (currently all available storage types are
- ultimately backed by LVM volumes) by introducing file-based storage
+- remove the hard dependency on LVM (currently all available storage
+ types are ultimately backed by LVM volumes) by introducing file-based
+ storage
Additionally, a number of smaller enhancements are also planned:
- support variable number of disks
parameters.
This means that we in effect take ownership of the minor space for
-that device type; if there's a user-created drbd minor, it will be
+that device type; if there's a user-created DRBD minor, it will be
automatically removed.
The change will have the effect of reducing the number of external
commands run per device from a constant number times the index of the
first free DRBD minor to just a constant number.
-Removal of obsolete device types (md, drbd7)
+Removal of obsolete device types (MD, DRBD7)
++++++++++++++++++++++++++++++++++++++++++++
We need to remove these device types because of two issues. First,
-drbd7 has bad failure modes in case of dual failures (both network and
+DRBD7 has bad failure modes in case of dual failures (both network and
disk - it cannot propagate the error up the device stack and instead
-just panics. Second, due to the assymetry between primary and
-secondary in md+drbd mode, we cannot do live failover (not even if we
-had md+drbd8).
+just panics. Second, due to the asymmetry between primary and
+secondary in MD+DRBD mode, we cannot do live failover (not even if we
+had MD+DRBD8).
File-based storage support
++++++++++++++++++++++++++
-This is covered by a separate design doc (<em>Vinales</em>) and
-would allow us to get rid of the hard requirement for testing
-clusters; it would also allow people who have SAN storage to do live
-failover taking advantage of their storage solution.
+Using files instead of logical volumes for instance storage would
+allow us to get rid of the hard requirement for volume groups for
+testing clusters and it would also allow usage of SAN storage to do
+live failover taking advantage of this storage solution.
Better LVM allocation
+++++++++++++++++++++
#. if no, and previous status was no, do nothing
#. if no, and previous status was yes:
#. if more than one node is inconsistent, do nothing
- #. if only one node is incosistent:
+ #. if only one node is inconsistent:
#. run ``vgreduce --removemissing``
- #. log this occurence in the ganeti log in a form that
+ #. log this occurrence in the Ganeti log in a form that
can be used for monitoring
#. [FUTURE] run ``replace-disks`` for all
instances affected
*failover to any* functionality, removing many of the layout
restrictions of a cluster:
-- the need to reserve memory on the current secondary: this gets reduced to
- a must to reserve memory anywhere on the cluster
+- the need to reserve memory on the current secondary: this gets reduced
+ to a must to reserve memory anywhere on the cluster
- the need to first failover and then replace secondary for an
instance: with failover-to-any, we can directly failover to
made between P1 and S1. This choice can be constrained, depending on
which of P1 and S1 has failed.
-- if P1 has failed, then S1 must become S2, and live migration is not possible
+- if P1 has failed, then S1 must become S2, and live migration is not
+ possible
- if S1 has failed, then P1 must become S2, and live migration could be
possible (in theory, but this is not a design goal for 2.0)
- verify that S2 (the node the user has chosen to keep as secondary) has
valid data (is consistent)
-- tear down the current DRBD association and setup a drbd pairing between
- P2 (P2 is indicated by the user) and S2; since P2 has no data, it will
- start resyncing from S2
+- tear down the current DRBD association and setup a DRBD pairing
+ between P2 (P2 is indicated by the user) and S2; since P2 has no data,
+ it will start re-syncing from S2
-- as soon as P2 is in state SyncTarget (i.e. after the resync has started
- but before it has finished), we can promote it to primary role (r/w)
- and start the instance on P2
+- as soon as P2 is in state SyncTarget (i.e. after the resync has
+ started but before it has finished), we can promote it to primary role
+ (r/w) and start the instance on P2
- as soon as the P2?S2 sync has finished, we can remove
the old data on the old node that has not been chosen for
will cause I/O errors on the instance, so (if a longer instance
downtime is acceptable) we can postpone the restart of the instance
until the resync is done. However, disk I/O errors on S2 will cause
-dataloss, since we don't have a good copy of the data anymore, so in
+data loss, since we don't have a good copy of the data anymore, so in
this case waiting for the sync to complete is not an option. As such,
it is recommended that this feature is used only in conjunction with
proper disk monitoring.
+++++++
The dynamic device model, while more complex, has an advantage: it
-will not reuse by mistake another's instance DRBD device, since it
-always looks for either our own or a free one.
+will not reuse by mistake the DRBD device of another instance, since
+it always looks for either our own or a free one.
The static one, in contrast, will assume that given a minor number N,
it's ours and we can take over. This needs careful implementation such
that if the minor is in use, either we are able to cleanly shut it
down, or we abort the startup. Otherwise, it could be that we start
-syncing between two instance's disks, causing dataloss.
+syncing between two instance's disks, causing data loss.
Variable number of disk/NICs per instance
In order to support high-security scenarios (for example read-only sda
and read-write sdb), we need to make a fully flexibly disk
definition. This has less impact that it might look at first sight:
-only the instance creation has hardcoded number of disks, not the disk
+only the instance creation has hard coded number of disks, not the disk
handling code. The block device handling and most of the instance
handling code is already working with "the instance's disks" as
opposed to "the two disks of the instance", but some pieces are not
The objective is to be able to specify the number of disks at
instance creation, and to be able to toggle from read-only to
-read-write a disk afterwards.
+read-write a disk afterward.
Variable number of NICs
+++++++++++++++++++++++
Similar to the disk change, we need to allow multiple network
interfaces per instance. This will affect the internal code (some
function will have to stop assuming that ``instance.nics`` is a list
-of length one), the OS api which currently can export/import only one
+of length one), the OS API which currently can export/import only one
instance, and the command line interface.
Interface changes
OS interface
~~~~~~~~~~~~
-The current Ganeti OS interface, version 5, is tailored for Ganeti 1.2. The
-interface is composed by a series of scripts which get called with certain
-parameters to perform OS-dependent operations on the cluster. The current
-scripts are:
+The current Ganeti OS interface, version 5, is tailored for Ganeti 1.2.
+The interface is composed by a series of scripts which get called with
+certain parameters to perform OS-dependent operations on the cluster.
+The current scripts are:
create
called when a new instance is added to the cluster
called to perform the os-specific operations necessary for renaming an
instance
-Currently these scripts suffer from the limitations of Ganeti 1.2: for example
-they accept exactly one block and one swap devices to operate on, rather than
-any amount of generic block devices, they blindly assume that an instance will
-have just one network interface to operate, they can not be configured to
-optimise the instance for a particular hypervisor.
+Currently these scripts suffer from the limitations of Ganeti 1.2: for
+example they accept exactly one block and one swap devices to operate
+on, rather than any amount of generic block devices, they blindly assume
+that an instance will have just one network interface to operate, they
+can not be configured to optimise the instance for a particular
+hypervisor.
-Since in Ganeti 2.0 we want to support multiple hypervisors, and a non-fixed
-number of network and disks the OS interface need to change to transmit the
-appropriate amount of information about an instance to its managing operating
-system, when operating on it. Moreover since some old assumptions usually used
-in OS scripts are no longer valid we need to re-establish a common knowledge on
-what can be assumed and what cannot be regarding Ganeti environment.
+Since in Ganeti 2.0 we want to support multiple hypervisors, and a
+non-fixed number of network and disks the OS interface need to change to
+transmit the appropriate amount of information about an instance to its
+managing operating system, when operating on it. Moreover since some old
+assumptions usually used in OS scripts are no longer valid we need to
+re-establish a common knowledge on what can be assumed and what cannot
+be regarding Ganeti environment.
When designing the new OS API our priorities are:
- ease of use
- future extensibility
-- ease of porting from the old api
+- ease of porting from the old API
- modularity
-As such we want to limit the number of scripts that must be written to support
-an OS, and make it easy to share code between them by uniforming their input.
-We also will leave the current script structure unchanged, as far as we can,
-and make a few of the scripts (import, export and rename) optional. Most
-information will be passed to the script through environment variables, for
-ease of access and at the same time ease of using only the information a script
-needs.
+As such we want to limit the number of scripts that must be written to
+support an OS, and make it easy to share code between them by uniforming
+their input. We also will leave the current script structure unchanged,
+as far as we can, and make a few of the scripts (import, export and
+rename) optional. Most information will be passed to the script through
+environment variables, for ease of access and at the same time ease of
+using only the information a script needs.
The Scripts
+++++++++++
-As in Ganeti 1.2, every OS which wants to be installed in Ganeti needs to
-support the following functionality, through scripts:
+As in Ganeti 1.2, every OS which wants to be installed in Ganeti needs
+to support the following functionality, through scripts:
create:
- used to create a new instance running that OS. This script should prepare the
- block devices, and install them so that the new OS can boot under the
- specified hypervisor.
+ used to create a new instance running that OS. This script should
+ prepare the block devices, and install them so that the new OS can
+ boot under the specified hypervisor.
export (optional):
- used to export an installed instance using the given OS to a format which can
- be used to import it back into a new instance.
+ used to export an installed instance using the given OS to a format
+ which can be used to import it back into a new instance.
import (optional):
- used to import an exported instance into a new one. This script is similar to
- create, but the new instance should have the content of the export, rather
- than contain a pristine installation.
+ used to import an exported instance into a new one. This script is
+ similar to create, but the new instance should have the content of the
+ export, rather than contain a pristine installation.
rename (optional):
- used to perform the internal OS-specific operations needed to rename an
- instance.
+ used to perform the internal OS-specific operations needed to rename
+ an instance.
-If any optional script is not implemented Ganeti will refuse to perform the
-given operation on instances using the non-implementing OS. Of course the
-create script is mandatory, and it doesn't make sense to support the either the
-export or the import operation but not both.
+If any optional script is not implemented Ganeti will refuse to perform
+the given operation on instances using the non-implementing OS. Of
+course the create script is mandatory, and it doesn't make sense to
+support the either the export or the import operation but not both.
Incompatibilities with 1.2
__________________________
-We expect the following incompatibilities between the OS scripts for 1.2 and
-the ones for 2.0:
+We expect the following incompatibilities between the OS scripts for 1.2
+and the ones for 2.0:
-- Input parameters: in 1.2 those were passed on the command line, in 2.0 we'll
- use environment variables, as there will be a lot more information and not
- all OSes may care about all of it.
-- Number of calls: export scripts will be called once for each device the
- instance has, and import scripts once for every exported disk. Imported
- instances will be forced to have a number of disks greater or equal to the
- one of the export.
-- Some scripts are not compulsory: if such a script is missing the relevant
- operations will be forbidden for instances of that os. This makes it easier
- to distinguish between unsupported operations and no-op ones (if any).
+- Input parameters: in 1.2 those were passed on the command line, in 2.0
+ we'll use environment variables, as there will be a lot more
+ information and not all OSes may care about all of it.
+- Number of calls: export scripts will be called once for each device
+ the instance has, and import scripts once for every exported disk.
+ Imported instances will be forced to have a number of disks greater or
+ equal to the one of the export.
+- Some scripts are not compulsory: if such a script is missing the
+ relevant operations will be forbidden for instances of that OS. This
+ makes it easier to distinguish between unsupported operations and
+ no-op ones (if any).
Input
_____
-Rather than using command line flags, as they do now, scripts will accept
-inputs from environment variables. We expect the following input values:
+Rather than using command line flags, as they do now, scripts will
+accept inputs from environment variables. We expect the following input
+values:
OS_API_VERSION
- The version of the OS api that the following parameters comply with;
+ The version of the OS API that the following parameters comply with;
this is used so that in the future we could have OSes supporting
multiple versions and thus Ganeti send the proper version in this
parameter
INSTANCE_NAME
Name of the instance acted on
HYPERVISOR
- The hypervisor the instance should run on (eg. 'xen-pvm', 'xen-hvm', 'kvm')
+ The hypervisor the instance should run on (e.g. 'xen-pvm', 'xen-hvm',
+ 'kvm')
DISK_COUNT
The number of disks this instance will have
NIC_COUNT
- The number of nics this instance will have
+ The number of NICs this instance will have
DISK_<N>_PATH
Path to the Nth disk.
DISK_<N>_ACCESS
W if read/write, R if read only. OS scripts are not supposed to touch
read-only disks, but will be passed them to know.
DISK_<N>_FRONTEND_TYPE
- Type of the disk as seen by the instance. Can be 'scsi', 'ide', 'virtio'
+ Type of the disk as seen by the instance. Can be 'scsi', 'ide',
+ 'virtio'
DISK_<N>_BACKEND_TYPE
Type of the disk as seen from the node. Can be 'block', 'file:loop' or
'file:blktap'
NIC_<N>_BRIDGE
Node bridge the Nth network interface will be connected to
NIC_<N>_FRONTEND_TYPE
- Type of the Nth nic as seen by the instance. For example 'virtio', 'rtl8139', etc.
+ Type of the Nth NIC as seen by the instance. For example 'virtio',
+ 'rtl8139', etc.
DEBUG_LEVEL
- Whether more out should be produced, for debugging purposes. Currently the
- only valid values are 0 and 1.
+ Whether more out should be produced, for debugging purposes. Currently
+ the only valid values are 0 and 1.
-These are only the basic variables we are thinking of now, but more may come
-during the implementation and they will be documented in the ganeti-os-api man
-page. All these variables will be available to all scripts.
+These are only the basic variables we are thinking of now, but more
+may come during the implementation and they will be documented in the
+:manpage:`ganeti-os-interface(7)` man page. All these variables will be
+available to all scripts.
Some scripts will need a few more information to work. These will have
per-script variables, such as for example:
OLD_INSTANCE_NAME
rename: the name the instance should be renamed from.
EXPORT_DEVICE
- export: device to be exported, a snapshot of the actual device. The data must be exported to stdout.
+ export: device to be exported, a snapshot of the actual device. The
+ data must be exported to stdout.
EXPORT_INDEX
export: sequential number of the instance device targeted.
IMPORT_DEVICE
- import: device to send the data to, part of the new instance. The data must be imported from stdin.
+ import: device to send the data to, part of the new instance. The data
+ must be imported from stdin.
IMPORT_INDEX
import: sequential number of the instance device targeted.
-(Rationale for INSTANCE_NAME as an environment variable: the instance name is
-always needed and we could pass it on the command line. On the other hand,
-though, this would force scripts to both access the environment and parse the
-command line, so we'll move it for uniformity.)
+(Rationale for INSTANCE_NAME as an environment variable: the instance
+name is always needed and we could pass it on the command line. On the
+other hand, though, this would force scripts to both access the
+environment and parse the command line, so we'll move it for
+uniformity.)
Output/Behaviour
________________
-As discussed scripts should only send user-targeted information to stderr. The
-create and import scripts are supposed to format/initialise the given block
-devices and install the correct instance data. The export script is supposed to
-export instance data to stdout in a format understandable by the the import
-script. The data will be compressed by ganeti, so no compression should be
-done. The rename script should only modify the instance's knowledge of what
-its name is.
+As discussed scripts should only send user-targeted information to
+stderr. The create and import scripts are supposed to format/initialise
+the given block devices and install the correct instance data. The
+export script is supposed to export instance data to stdout in a format
+understandable by the the import script. The data will be compressed by
+Ganeti, so no compression should be done. The rename script should only
+modify the instance's knowledge of what its name is.
Other declarative style features
++++++++++++++++++++++++++++++++
Similar to Ganeti 1.2, OS specifications will need to provide a
-'ganeti_api_version' containing list of numbers matching the version(s) of the
-api they implement. Ganeti itself will always be compatible with one version of
-the API and may maintain retrocompatibility if it's feasible to do so. The
-numbers are one-per-line, so an OS supporting both version 5 and version 20
-will have a file containing two lines. This is different from Ganeti 1.2, which
-only supported one version number.
+'ganeti_api_version' containing list of numbers matching the
+version(s) of the API they implement. Ganeti itself will always be
+compatible with one version of the API and may maintain backwards
+compatibility if it's feasible to do so. The numbers are one-per-line,
+so an OS supporting both version 5 and version 20 will have a file
+containing two lines. This is different from Ganeti 1.2, which only
+supported one version number.
-In addition to that an OS will be able to declare that it does support only a
-subset of the ganeti hypervisors, by declaring them in the 'hypervisors' file.
+In addition to that an OS will be able to declare that it does support
+only a subset of the Ganeti hypervisors, by declaring them in the
+'hypervisors' file.
Caveats/Notes
+++++++++++++
-We might want to have a "default" import/export behaviour that just dumps all
-disks and restores them. This can save work as most systems will just do this,
-while allowing flexibility for different systems.
+We might want to have a "default" import/export behaviour that just
+dumps all disks and restores them. This can save work as most systems
+will just do this, while allowing flexibility for different systems.
-Environment variables are limited in size, but we expect that there will be
-enough space to store the information we need. If we discover that this is not
-the case we may want to go to a more complex API such as storing those
-information on the filesystem and providing the OS script with the path to a
-file where they are encoded in some format.
+Environment variables are limited in size, but we expect that there will
+be enough space to store the information we need. If we discover that
+this is not the case we may want to go to a more complex API such as
+storing those information on the filesystem and providing the OS script
+with the path to a file where they are encoded in some format.
Remote API changes
~~~~~~~~~~~~~~~~~~
-The first Ganeti RAPI was designed and deployed with the Ganeti 1.2.5 release.
-That version provide Read-Only access to a cluster state. Fully functional
-read-write API demand significant internal changes which are in a pipeline for
-Ganeti 2.0 release.
+The first Ganeti remote API (RAPI) was designed and deployed with the
+Ganeti 1.2.5 release. That version provide read-only access to the
+cluster state. Fully functional read-write API demands significant
+internal changes which will be implemented in version 2.0.
-We decided to go with implementing the Ganeti RAPI in a RESTful way, which is
-aligned with key features we looking. It is simple, stateless, scalable and
-extensible paradigm of API implementation. As transport it uses HTTP over SSL,
-and we are implementing it in JSON encoding, but in a way it possible to extend
-and provide any other one.
+We decided to go with implementing the Ganeti RAPI in a RESTful way,
+which is aligned with key features we looking. It is simple,
+stateless, scalable and extensible paradigm of API implementation. As
+transport it uses HTTP over SSL, and we are implementing it with JSON
+encoding, but in a way it possible to extend and provide any other
+one.
Design
++++++
-The Ganeti API implemented as independent daemon, running on the same node
-with the same permission level as Ganeti master daemon. Communication done
-through unix socket protocol provided by Ganeti luxi library.
-In order to keep communication asynchronous RAPI process two types of client
-requests:
+The Ganeti RAPI is implemented as independent daemon, running on the
+same node with the same permission level as Ganeti master
+daemon. Communication is done through the LUXI library to the master
+daemon. In order to keep communication asynchronous RAPI processes two
+types of client requests:
-- queries: sever able to answer immediately
-- jobs: some time needed.
+- queries: server is able to answer immediately
+- job submission: some time is required for a useful response
-In the query case requested data send back to client in http body. Typical
-examples of queries would be list of nodes, instances, cluster info, etc.
-Dealing with jobs client instead of waiting until job completes receive a job
-id, the identifier which allows to query the job progress in the job queue.
-(See job queue design doc for details)
+In the query case requested data send back to client in the HTTP
+response body. Typical examples of queries would be: list of nodes,
+instances, cluster info, etc.
-Internally, each exported object has an version identifier, which is used as a
-state stamp in the http header E-Tag field for request/response to avoid a race
-condition.
+In the case of job submission, the client receive a job ID, the
+identifier which allows one to query the job progress in the job queue
+(see `Job Queue`_).
+
+Internally, each exported object has an version identifier, which is
+used as a state identifier in the HTTP header E-Tag field for
+requests/responses to avoid race conditions.
Resource representation
+++++++++++++++++++++++
-The key difference of REST approach from others API is instead having one URI
-for all our requests, REST demand separate service by resources with unique
-URI. Each of them should have limited amount of stateless and standard HTTP
+The key difference of using REST instead of others API is that REST
+requires separation of services via resources with unique URIs. Each
+of them should have limited amount of state and support standard HTTP
methods: GET, POST, DELETE, PUT.
-For example in Ganeti case we can have a set of URI:
- - /{clustername}/instances
- - /{clustername}/instances/{instancename}
- - /{clustername}/instances/{instancename}/tag
- - /{clustername}/tag
+For example in Ganeti's case we can have a set of URI:
+
+ - ``/{clustername}/instances``
+ - ``/{clustername}/instances/{instancename}``
+ - ``/{clustername}/instances/{instancename}/tag``
+ - ``/{clustername}/tag``
-A GET request to /{clustername}/instances will return list of instances, a POST
-to /{clustername}/instances should create new instance, a DELETE
-/{clustername}/instances/{instancename} should delete instance, a GET
-/{clustername}/tag get cluster tag
+A GET request to ``/{clustername}/instances`` will return the list of
+instances, a POST to ``/{clustername}/instances`` should create a new
+instance, a DELETE ``/{clustername}/instances/{instancename}`` should
+delete the instance, a GET ``/{clustername}/tag`` should return get
+cluster tags.
-Each resource URI has a version prefix. The complete list of resources id TBD.
+Each resource URI will have a version prefix. The resource IDs are to
+be determined.
-Internal encoding might be JSON, XML, or any other. The JSON encoding fits
-nicely in Ganeti RAPI needs. Specific representation client can request with
-Accept field in the HTTP header.
+Internal encoding might be JSON, XML, or any other. The JSON encoding
+fits nicely in Ganeti RAPI needs. The client can request a specific
+representation via the Accept field in the HTTP header.
-The REST uses standard HTTP as application protocol (not just as a transport)
-for resources access. Set of possible result codes is a subset of standard HTTP
-results. The stateless provide additional reliability and transparency to
-operations.
+REST uses HTTP as its transport and application protocol for resource
+access. The set of possible responses is a subset of standard HTTP
+responses.
+
+The statelessness model provides additional reliability and
+transparency to operations (e.g. only one request needs to be analyzed
+to understand the in-progress operation, not a sequence of multiple
+requests/responses).
Security
++++++++
-With the write functionality security becomes much bigger an issue. The Ganeti
-RAPI uses basic HTTP authentication on top of SSL connection to grant access to
-an exported resource. The password stores locally in Apache-style .htpasswd
-file. Only one level of privileges is supported.
+With the write functionality security becomes a much bigger an issue.
+The Ganeti RAPI uses basic HTTP authentication on top of an
+SSL-secured connection to grant access to an exported resource. The
+password is stored locally in an Apache-style ``.htpasswd`` file. Only
+one level of privileges is supported.
+
+Caveats
++++++++
+The model detailed above for job submission requires the client to
+poll periodically for updates to the job; an alternative would be to
+allow the client to request a callback, or a 'wait for updates' call.
+
+The callback model was not considered due to the following two issues:
+
+- callbacks would require a new model of allowed callback URLs,
+ together with a method of managing these
+- callbacks only work when the client and the master are in the same
+ security domain, and they fail in the other cases (e.g. when there is
+ a firewall between the client and the RAPI daemon that only allows
+ client-to-RAPI calls, which is usual in DMZ cases)
+
+The 'wait for updates' method is not suited to the HTTP protocol,
+where requests are supposed to be short-lived.
Command line changes
~~~~~~~~~~~~~~~~~~~~
Ganeti 2.0 introduces several new features as well as new ways to
handle instance resources like disks or network interfaces. This
-requires some noticable changes in the way commandline arguments are
+requires some noticeable changes in the way command line arguments are
handled.
-- extend and modify commandline syntax to support new features
-- ensure consistent patterns in commandline arguments to reduce cognitive load
+- extend and modify command line syntax to support new features
+- ensure consistent patterns in command line arguments to reduce
+ cognitive load
The design changes that require these changes are, in no particular
order:
cluster, each supporting different parameters,
- support for device type CDROM (via ISO image)
-As such, there are several areas of Ganeti where the commandline
+As such, there are several areas of Ganeti where the command line
arguments will change:
- Cluster configuration
- handling of CDROM devices and
- handling of hypervisor specific options.
-There are several areas of Ganeti where the commandline arguments will change:
+There are several areas of Ganeti where the command line arguments
+will change:
- Cluster configuration
:--net: for network interface cards
:--disk: for disk devices
-The syntax to the device specific options is similiar to the generic
+The syntax to the device specific options is similar to the generic
device options, but instead of specifying a device number like for
gnt-instance add, you specify the magic string add. The new device
will always be appended at the end of the list of devices of this type
:--net: for network interface cards
:--disk: for disk devices
-The syntax to the device specific options is similiar to the generic
+The syntax to the device specific options is similar to the generic
device options. The device number you specify identifies the device to
be modified.
-Example: gnt-instance modify --disk 2:access=r
+Example::
+
+ gnt-instance modify --disk 2:access=r
Hypervisor Options
++++++++++++++++++
Ganeti 2.0 will support more than one hypervisor. Different
hypervisors have various options that only apply to a specific
hypervisor. Those hypervisor specific options are treated specially
-via the --hypervisor option. The generic syntax of the hypervisor
-option is as follows:
+via the ``--hypervisor`` option. The generic syntax of the hypervisor
+option is as follows::
--hypervisor $HYPERVISOR:$OPTION=$VALUE[,$OPTION=$VALUE]
:$VALUE: hypervisor option value, string
The hypervisor option for an instance can be set on instance creation
-time via the gnt-instance add command. If the hypervisor for an
+time via the ``gnt-instance add`` command. If the hypervisor for an
instance is not specified upon instance creation, the default
hypervisor will be used.
+++++++++++++++++++++++++++++++
The hypervisor parameters of an existing instance can be modified
-using --hypervisor option of the gnt-instance modify command. However,
-the hypervisor type of an existing instance can not be changed, only
-the particular hypervisor specific option can be changed. Therefore,
-the format of the option parameters has been simplified to omit the
-hypervisor name and only contain the comma separated list of
-option-value pairs.
+using ``--hypervisor`` option of the ``gnt-instance modify``
+command. However, the hypervisor type of an existing instance can not
+be changed, only the particular hypervisor specific option can be
+changed. Therefore, the format of the option parameters has been
+simplified to omit the hypervisor name and only contain the comma
+separated list of option-value pairs.
-Example: gnt-instance modify --hypervisor
-cdrom=/srv/boot.iso,boot_order=cdrom:network test-instance
+Example::
+
+ gnt-instance modify --hypervisor cdrom=/srv/boot.iso,boot_order=cdrom:network test-instance
gnt-cluster commands
++++++++++++++++++++
Hypervisor cluster defaults
+++++++++++++++++++++++++++
-The generic format of the hypervisor clusterwide default setting option is:
+The generic format of the hypervisor cluster wide default setting
+option is::
--hypervisor-defaults $HYPERVISOR:$OPTION=$VALUE[,$OPTION=$VALUE]
:$OPTION: cluster default option, string,
:$VALUE: cluster default option value, string.
-
-Functionality changes
----------------------
-
-The disk storage will receive some changes, and will also remove
-support for the drbd7 and md disk types. See the
-design-2.0-disk-changes document.
-
-The configuration storage will be changed, with the effect that more
-data will be available on the nodes for access from outside ganeti
-(e.g. from shell scripts) and that nodes will get slightly more
-awareness of the cluster configuration.
-
-The RAPI will enable modify operations (beside the read-only queries
-that are available today), so in effect almost all the operations
-available today via the ``gnt-*`` commands will be available via the
-remote API.
-
-A change in the hypervisor support area will be that we will support
-multiple hypervisors in parallel in the same cluster, so one could run
-Xen HVM side-by-side with Xen PVM on the same cluster.
-
-New features
-------------
-
-There will be a number of minor feature enhancements targeted to
-either 2.0 or subsequent 2.x releases:
-
-- multiple disks, with custom properties (read-only/read-write, exportable,
- etc.)
-- multiple NICs
-
-These changes will require OS API changes, details are in the
-design-2.0-os-interface document. And they will also require many
-command line changes, see the design-2.0-commandline-parameters
-document.
+.. vim: set textwidth=72 :
projects / ganeti-local / blobdiff