=========
Ganeti 2.1 will add features to help further automatization of cluster
-operations, further improbe scalability to even bigger clusters, and
+operations, further improve scalability to even bigger clusters, and
make it easier to debug the Ganeti core.
-Background
-==========
-
-Overview
-========
-
Detailed design
===============
acquires and no current holders. The caller can have the lock
immediately.
-.. image:: design-2.1-lock-acquire.png
+.. digraph:: "design-2.1-lock-acquire"
+
+ graph[fontsize=8, fontname="Helvetica"]
+ node[fontsize=8, fontname="Helvetica", width="0", height="0"]
+ edge[fontsize=8, fontname="Helvetica"]
+
+ /* Actions */
+ abort[label="Abort\n(couldn't acquire)"]
+ acquire[label="Acquire lock"]
+ add_to_queue[label="Add condition to queue"]
+ wait[label="Wait for notification"]
+ remove_from_queue[label="Remove from queue"]
+
+ /* Conditions */
+ alone[label="Empty queue\nand can acquire?", shape=diamond]
+ have_timeout[label="Do I have\ntimeout?", shape=diamond]
+ top_of_queue_and_can_acquire[
+ label="On top of queue and\ncan acquire lock?",
+ shape=diamond,
+ ]
+
+ /* Lines */
+ alone->acquire[label="Yes"]
+ alone->add_to_queue[label="No"]
+
+ have_timeout->abort[label="Yes"]
+ have_timeout->wait[label="No"]
+
+ top_of_queue_and_can_acquire->acquire[label="Yes"]
+ top_of_queue_and_can_acquire->have_timeout[label="No"]
+ add_to_queue->wait
+ wait->top_of_queue_and_can_acquire
+ acquire->remove_from_queue
Release
*******
exclusive locks by forcing consecutive shared acquires to wait in the
queue.
-.. image:: design-2.1-lock-release.png
+.. digraph:: "design-2.1-lock-release"
+
+ graph[fontsize=8, fontname="Helvetica"]
+ node[fontsize=8, fontname="Helvetica", width="0", height="0"]
+ edge[fontsize=8, fontname="Helvetica"]
+
+ /* Actions */
+ remove_from_owners[label="Remove from owner list"]
+ notify[label="Notify topmost"]
+ swap_shared[label="Swap shared conditions"]
+ success[label="Success"]
+
+ /* Conditions */
+ have_pending[label="Any pending\nacquires?", shape=diamond]
+ was_active_queue[
+ label="Was active condition\nfor shared acquires?",
+ shape=diamond,
+ ]
+
+ /* Lines */
+ remove_from_owners->have_pending
+
+ have_pending->notify[label="Yes"]
+ have_pending->success[label="No"]
+
+ notify->was_active_queue
+
+ was_active_queue->swap_shared[label="Yes"]
+ was_active_queue->success[label="No"]
+
+ swap_shared->success
Delete
caller nonetheless.
+Node daemon availability
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Current State and shortcomings
+++++++++++++++++++++++++++++++
+
+Currently, when a Ganeti node suffers serious system disk damage, the
+migration/failover of an instance may not correctly shutdown the virtual
+machine on the broken node causing instances duplication. The ``gnt-node
+powercycle`` command can be used to force a node reboot and thus to
+avoid duplicated instances. This command relies on node daemon
+availability, though, and thus can fail if the node daemon has some
+pages swapped out of ram, for example.
+
+
+Proposed changes
+++++++++++++++++
+
+The proposed solution forces node daemon to run exclusively in RAM. It
+uses python ctypes to to call ``mlockall(MCL_CURRENT | MCL_FUTURE)`` on
+the node daemon process and all its children. In addition another log
+handler has been implemented for node daemon to redirect to
+``/dev/console`` messages that cannot be written on the logfile.
+
+With these changes node daemon can successfully run basic tasks such as
+a powercycle request even when the system disk is heavily damaged and
+reading/writing to disk fails constantly.
+
+
+New Features
+------------
+
+Automated Ganeti Cluster Merger
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Current situation
++++++++++++++++++
+
+Currently there's no easy way to merge two or more clusters together.
+But in order to optimize resources this is a needed missing piece. The
+goal of this design doc is to come up with a easy to use solution which
+allows you to merge two or more cluster together.
+
+Initial contact
++++++++++++++++
+
+As the design of Ganeti is based on an autonomous system, Ganeti by
+itself has no way to reach nodes outside of its cluster. To overcome
+this situation we're required to prepare the cluster before we can go
+ahead with the actual merge: We've to replace at least the ssh keys on
+the affected nodes before we can do any operation within ``gnt-``
+commands.
+
+To make this a automated process we'll ask the user to provide us with
+the root password of every cluster we've to merge. We use the password
+to grab the current ``id_dsa`` key and then rely on that ssh key for any
+further communication to be made until the cluster is fully merged.
+
+Cluster merge
++++++++++++++
+
+After initial contact we do the cluster merge:
+
+1. Grab the list of nodes
+2. On all nodes add our own ``id_dsa.pub`` key to ``authorized_keys``
+3. Stop all instances running on the merging cluster
+4. Disable ``ganeti-watcher`` as it tries to restart Ganeti daemons
+5. Stop all Ganeti daemons on all merging nodes
+6. Grab the ``config.data`` from the master of the merging cluster
+7. Stop local ``ganeti-masterd``
+8. Merge the config:
+
+ 1. Open our own cluster ``config.data``
+ 2. Open cluster ``config.data`` of the merging cluster
+ 3. Grab all nodes of the merging cluster
+ 4. Set ``master_candidate`` to false on all merging nodes
+ 5. Add the nodes to our own cluster ``config.data``
+ 6. Grab all the instances on the merging cluster
+ 7. Adjust the port if the instance has drbd layout:
+
+ 1. In ``logical_id`` (index 2)
+ 2. In ``physical_id`` (index 1 and 3)
+
+ 8. Add the instances to our own cluster ``config.data``
+
+9. Start ``ganeti-masterd`` with ``--no-voting`` ``--yes-do-it``
+10. ``gnt-node add --readd`` on all merging nodes
+11. ``gnt-cluster redist-conf``
+12. Restart ``ganeti-masterd`` normally
+13. Enable ``ganeti-watcher`` again
+14. Start all merging instances again
+
+Rollback
+++++++++
+
+Until we actually (re)add any nodes we can abort and rollback the merge
+at any point. After merging the config, though, we've to get the backup
+copy of ``config.data`` (from another master candidate node). And for
+security reasons it's a good idea to undo ``id_dsa.pub`` distribution by
+going on every affected node and remove the ``id_dsa.pub`` key again.
+Also we've to keep in mind, that we've to start the Ganeti daemons and
+starting up the instances again.
+
+Verification
+++++++++++++
+
+Last but not least we should verify that the merge was successful.
+Therefore we run ``gnt-cluster verify``, which ensures that the cluster
+overall is in a healthy state. Additional it's also possible to compare
+the list of instances/nodes with a list made prior to the upgrade to
+make sure we didn't lose any data/instance/node.
+
+Appendix
+++++++++
+
+cluster-merge.py
+^^^^^^^^^^^^^^^^
+
+Used to merge the cluster config. This is a POC and might differ from
+actual production code.
+
+::
+
+ #!/usr/bin/python
+
+ import sys
+ from ganeti import config
+ from ganeti import constants
+
+ c_mine = config.ConfigWriter(offline=True)
+ c_other = config.ConfigWriter(sys.argv[1])
+
+ fake_id = 0
+ for node in c_other.GetNodeList():
+ node_info = c_other.GetNodeInfo(node)
+ node_info.master_candidate = False
+ c_mine.AddNode(node_info, str(fake_id))
+ fake_id += 1
+
+ for instance in c_other.GetInstanceList():
+ instance_info = c_other.GetInstanceInfo(instance)
+ for dsk in instance_info.disks:
+ if dsk.dev_type in constants.LDS_DRBD:
+ port = c_mine.AllocatePort()
+ logical_id = list(dsk.logical_id)
+ logical_id[2] = port
+ dsk.logical_id = tuple(logical_id)
+ physical_id = list(dsk.physical_id)
+ physical_id[1] = physical_id[3] = port
+ dsk.physical_id = tuple(physical_id)
+ c_mine.AddInstance(instance_info, str(fake_id))
+ fake_id += 1
+
+
Feature changes
---------------
A confd query will look like this, on the wire::
- {
+ plj0{
"msg": "{\"type\": 1,
\"rsalt\": \"9aa6ce92-8336-11de-af38-001d093e835f\",
\"protocol\": 1,
"hmac": "4a4139b2c3c5921f7e439469a0a45ad200aead0f"
}
-Detailed explanation of the various fields:
+``plj0`` is a fourcc that details the message content. It stands for plain
+json 0, and can be changed as we move on to different type of protocols
+(for example protocol buffers, or encrypted json). What follows is a
+json encoded string, with the following fields:
+
+- ``msg`` contains a JSON-encoded query, its fields are:
+
+ - ``protocol``, integer, is the confd protocol version (initially
+ just ``constants.CONFD_PROTOCOL_VERSION``, with a value of 1)
+ - ``type``, integer, is the query type. For example "node role by
+ name" or "node primary ip by instance ip". Constants will be
+ provided for the actual available query types
+ - ``query`` is a multi-type field (depending on the ``type`` field):
-- 'msg' contains a JSON-encoded query, its fields are:
+ - it can be missing, when the request is fully determined by the
+ ``type`` field
+ - it can contain a string which denotes the search key: for
+ example an IP, or a node name
+ - it can contain a dictionary, in which case the actual details
+ vary further per request type
- - 'protocol', integer, is the confd protocol version (initially just
- constants.CONFD_PROTOCOL_VERSION, with a value of 1)
- - 'type', integer, is the query type. For example "node role by name"
- or "node primary ip by instance ip". Constants will be provided for
- the actual available query types.
- - 'query', string, is the search key. For example an ip, or a node
- name.
- - 'rsalt', string, is the required response salt. The client must use
- it to recognize which answer it's getting.
+ - ``rsalt``, string, is the required response salt; the client must
+ use it to recognize which answer it's getting.
-- 'salt' must be the current unix timestamp, according to the client.
- Servers can refuse messages which have a wrong timing, according to
- their configuration and clock.
-- 'hmac' is an hmac signature of salt+msg, with the cluster hmac key
+- ``salt`` must be the current unix timestamp, according to the
+ client; servers should refuse messages which have a wrong timing,
+ according to their configuration and clock
+- ``hmac`` is an hmac signature of salt+msg, with the cluster hmac key
If an answer comes back (which is optional, since confd works over UDP)
it will be in this format::
- {
+ plj0{
"msg": "{\"status\": 0,
\"answer\": 0,
\"serial\": 42,
Where:
-- 'msg' contains a JSON-encoded answer, its fields are:
-
- - 'protocol', integer, is the confd protocol version (initially just
- constants.CONFD_PROTOCOL_VERSION, with a value of 1)
- - 'status', integer, is the error code. Initially just 0 for 'ok' or
- '1' for 'error' (in which case answer contains an error detail,
- rather than an answer), but in the future it may be expanded to have
- more meanings (eg: 2, the answer is compressed)
- - 'answer', is the actual answer. Its type and meaning is query
- specific. For example for "node primary ip by instance ip" queries
+- ``plj0`` the message type magic fourcc, as discussed above
+- ``msg`` contains a JSON-encoded answer, its fields are:
+
+ - ``protocol``, integer, is the confd protocol version (initially
+ just constants.CONFD_PROTOCOL_VERSION, with a value of 1)
+ - ``status``, integer, is the error code; initially just ``0`` for
+ 'ok' or ``1`` for 'error' (in which case answer contains an error
+ detail, rather than an answer), but in the future it may be
+ expanded to have more meanings (e.g. ``2`` if the answer is
+ compressed)
+ - ``answer``, is the actual answer; its type and meaning is query
+ specific: for example for "node primary ip by instance ip" queries
it will be a string containing an IP address, for "node role by
- name" queries it will be an integer which encodes the role (master,
- candidate, drained, offline) according to constants.
+ name" queries it will be an integer which encodes the role
+ (master, candidate, drained, offline) according to constants
-- 'salt' is the requested salt from the query. A client can use it to
- recognize what query the answer is answering.
-- 'hmac' is an hmac signature of salt+msg, with the cluster hmac key
+- ``salt`` is the requested salt from the query; a client can use it
+ to recognize what query the answer is answering.
+- ``hmac`` is an hmac signature of salt+msg, with the cluster hmac key
Redistribute Config
Current State and shortcomings
++++++++++++++++++++++++++++++
-Currently LURedistributeConfig triggers a copy of the updated
+Currently LUClusterRedistConf triggers a copy of the updated
configuration file to all master candidates and of the ssconf files to
all nodes. There are other files which are maintained manually but which
are important to keep in sync. These are:
"classes" in the future. Instance classes is not a feature we plan
implementing in 2.1, though.
+
+Global hypervisor parameters
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Current State and shortcomings
+++++++++++++++++++++++++++++++
+
+Currently all hypervisor parameters are modifiable both globally
+(cluster level) and at instance level. However, there is no other
+framework to held hypervisor-specific parameters, so if we want to add
+a new class of hypervisor parameters that only makes sense on a global
+level, we have to change the hvparams framework.
+
+Proposed changes
+++++++++++++++++
+
+We add a new (global, not per-hypervisor) list of parameters which are
+not changeable on a per-instance level. The create, modify and query
+instance operations are changed to not allow/show these parameters.
+
+Furthermore, to allow transition of parameters to the global list, and
+to allow cleanup of inadverdently-customised parameters, the
+``UpgradeConfig()`` method of instances will drop any such parameters
+from their list of hvparams, such that a restart of the master daemon
+is all that is needed for cleaning these up.
+
+Also, the framework is simple enough that if we need to replicate it
+at beparams level we can do so easily.
+
+
Non bridged instances support
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
staggered way. In 2.1, we will simply add an “uuid” attribute to each
of the instances, nodes and cluster itself. This will be reported on
instance creation for nodes, and on node adds for the nodes. It will
-be of course avaiblable for querying via the OpQueryNodes/Instance and
+be of course avaiblable for querying via the OpNodeQuery/Instance and
cluster information, and via RAPI as well.
Note that Ganeti will not provide any way to change this attribute.
all instances on the node.
+User-id pool
+~~~~~~~~~~~~
+
+In order to allow running different processes under unique user-ids
+on a node, we introduce the user-id pool concept.
+
+The user-id pool is a cluster-wide configuration parameter.
+It is a list of user-ids and/or user-id ranges that are reserved
+for running Ganeti processes (including KVM instances).
+The code guarantees that on a given node a given user-id is only
+handed out if there is no other process running with that user-id.
+
+Please note, that this can only be guaranteed if all processes in
+the system - that run under a user-id belonging to the pool - are
+started by reserving a user-id first. That can be accomplished
+either by using the RequestUnusedUid() function to get an unused
+user-id or by implementing the same locking mechanism.
+
+Implementation
+++++++++++++++
+
+The functions that are specific to the user-id pool feature are located
+in a separate module: ``lib/uidpool.py``.
+
+Storage
+^^^^^^^
+
+The user-id pool is a single cluster parameter. It is stored in the
+*Cluster* object under the ``uid_pool`` name as a list of integer
+tuples. These tuples represent the boundaries of user-id ranges.
+For single user-ids, the boundaries are equal.
+
+The internal user-id pool representation is converted into a
+string: a newline separated list of user-ids or user-id ranges.
+This string representation is distributed to all the nodes via the
+*ssconf* mechanism. This means that the user-id pool can be
+accessed in a read-only way on any node without consulting the master
+node or master candidate nodes.
+
+Initial value
+^^^^^^^^^^^^^
+
+The value of the user-id pool cluster parameter can be initialized
+at cluster initialization time using the
+
+``gnt-cluster init --uid-pool <uid-pool definition> ...``
+
+command.
+
+As there is no sensible default value for the user-id pool parameter,
+it is initialized to an empty list if no ``--uid-pool`` option is
+supplied at cluster init time.
+
+If the user-id pool is empty, the user-id pool feature is considered
+to be disabled.
+
+Manipulation
+^^^^^^^^^^^^
+
+The user-id pool cluster parameter can be modified from the
+command-line with the following commands:
+
+- ``gnt-cluster modify --uid-pool <uid-pool definition>``
+- ``gnt-cluster modify --add-uids <uid-pool definition>``
+- ``gnt-cluster modify --remove-uids <uid-pool definition>``
+
+The ``--uid-pool`` option overwrites the current setting with the
+supplied ``<uid-pool definition>``, while
+``--add-uids``/``--remove-uids`` adds/removes the listed uids
+or uid-ranges from the pool.
+
+The ``<uid-pool definition>`` should be a comma-separated list of
+user-ids or user-id ranges. A range should be defined by a lower and
+a higher boundary. The boundaries should be separated with a dash.
+The boundaries are inclusive.
+
+The ``<uid-pool definition>`` is parsed into the internal
+representation, sanity-checked and stored in the ``uid_pool``
+attribute of the *Cluster* object.
+
+It is also immediately converted into a string (formatted in the
+input format) and distributed to all nodes via the *ssconf* mechanism.
+
+Inspection
+^^^^^^^^^^
+
+The current value of the user-id pool cluster parameter is printed
+by the ``gnt-cluster info`` command.
+
+The output format is accepted by the ``gnt-cluster modify --uid-pool``
+command.
+
+Locking
+^^^^^^^
+
+The ``uidpool.py`` module provides a function (``RequestUnusedUid``)
+for requesting an unused user-id from the pool.
+
+This will try to find a random user-id that is not currently in use.
+The algorithm is the following:
+
+1) Randomize the list of user-ids in the user-id pool
+2) Iterate over this randomized UID list
+3) Create a lock file (it doesn't matter if it already exists)
+4) Acquire an exclusive POSIX lock on the file, to provide mutual
+ exclusion for the following non-atomic operations
+5) Check if there is a process in the system with the given UID
+6) If there isn't, return the UID, otherwise unlock the file and
+ continue the iteration over the user-ids
+
+The user can than start a new process with this user-id.
+Once a process is successfully started, the exclusive POSIX lock can
+be released, but the lock file will remain in the filesystem.
+The presence of such a lock file means that the given user-id is most
+probably in use. The lack of a uid lock file does not guarantee that
+there are no processes with that user-id.
+
+After acquiring the exclusive POSIX lock, ``RequestUnusedUid``
+always performs a check to see if there is a process running with the
+given uid.
+
+A user-id can be returned to the pool, by calling the
+``ReleaseUid`` function. This will remove the corresponding lock file.
+Note, that it doesn't check if there is any process still running
+with that user-id. The removal of the lock file only means that there
+are most probably no processes with the given user-id. This helps
+in speeding up the process of finding a user-id that is guaranteed to
+be unused.
+
+There is a convenience function, called ``ExecWithUnusedUid`` that
+wraps the execution of a function (or any callable) that requires a
+unique user-id. ``ExecWithUnusedUid`` takes care of requesting an
+unused user-id and unlocking the lock file. It also automatically
+returns the user-id to the pool if the callable raises an exception.
+
+Code examples
++++++++++++++
+
+Requesting a user-id from the pool:
+
+::
+
+ from ganeti import ssconf
+ from ganeti import uidpool
+
+ # Get list of all user-ids in the uid-pool from ssconf
+ ss = ssconf.SimpleStore()
+ uid_pool = uidpool.ParseUidPool(ss.GetUidPool(), separator="\n")
+ all_uids = set(uidpool.ExpandUidPool(uid_pool))
+
+ uid = uidpool.RequestUnusedUid(all_uids)
+ try:
+ <start a process with the UID>
+ # Once the process is started, we can release the file lock
+ uid.Unlock()
+ except ..., err:
+ # Return the UID to the pool
+ uidpool.ReleaseUid(uid)
+
+
+Releasing a user-id:
+
+::
+
+ from ganeti import uidpool
+
+ uid = <get the UID the process is running under>
+ <stop the process>
+ uidpool.ReleaseUid(uid)
+
+
External interface changes
--------------------------
projects / ganeti-local / blobdiff