- core changes, which affect the master daemon/job queue/locking or
all/most logical units
- logical unit/feature changes
-- external interface changes (e.g. command line, os api, hooks, ...)
+- external interface changes (e.g. command line, OS API, hooks, ...)
Core changes
============
between machines connected to the same switch might be bigger than the
bandwidth for inter-switch connections.
-Moreover some operations inside a cluster require all nodes to be locked
+Moreover, some operations inside a cluster require all nodes to be locked
together for inter-node consistency, and won't scale if we increase the
number of nodes to a few hundreds.
~~~~~~~~~~~~~~~~
With this change we'll divide Ganeti nodes into groups. Nothing will
-change for clusters with only one node group, the default one. Bigger
-cluster instead will be able to have more than one group, and each node
-will belong to exactly one.
+change for clusters with only one node group. Bigger clusters will be
+able to have more than one group, and each node will belong to exactly
+one.
Node group management
+++++++++++++++++++++
To manage node groups and the nodes belonging to them, the following new
-commands/flags will be introduced::
+commands and flags will be introduced::
- gnt-node group-add <group> # add a new node group
- gnt-node group-del <group> # delete an empty group
- gnt-node group-list # list node groups
- gnt-node group-rename <oldname> <newname> # rename a group
- gnt-node list/info -g <group> # list only nodes belongin to a group
- gnt-node add -g <group> # add a node to a certain group
- gnt-node modify -g <group> # move a node to a new group
+ gnt-group add <group> # add a new node group
+ gnt-group remove <group> # delete an empty node group
+ gnt-group list # list node groups
+ gnt-group rename <oldname> <newname> # rename a node group
+ gnt-node {list,info} -g <group> # list only nodes belonging to a node group
+ gnt-node modify -g <group> # assign a node to a node group
+
+Node group attributes
++++++++++++++++++++++
+
+In clusters with more than one node group, it may be desirable to
+establish local policies regarding which groups should be preferred when
+performing allocation of new instances, or inter-group instance migrations.
+
+To help with this, we will provide an ``alloc_policy`` attribute for
+node groups. Such attribute will be honored by iallocator plugins when
+making automatic decisions regarding instance placement.
+
+The ``alloc_policy`` attribute can have the following values:
+
+- unallocable: the node group should not be a candidate for instance
+ allocations, and the operation should fail if only groups in this
+ state could be found that would satisfy the requirements.
+
+- last_resort: the node group should not be used for instance
+ allocations, unless this would be the only way to have the operation
+ succeed. Prioritization among groups in this state will be deferred to
+ the iallocator plugin that's being used.
+
+- preferred: the node group can be used freely for allocation of
+ instances (this is the default state for newly created node
+ groups). Note that prioritization among groups in this state will be
+ deferred to the iallocator plugin that's being used.
+
+Node group operations
++++++++++++++++++++++
+
+One operation at the node group level will be initially provided::
+
+ gnt-group drain <group>
+
+The purpose of this operation is to migrate all instances in a given
+node group to other groups in the cluster, e.g. to reclaim capacity if
+there are enough free resources in other node groups that share a
+storage pool with the evacuated group.
Instance level changes
++++++++++++++++++++++
-Instances will be able to live in only one group at a time. This is
-mostly important for DRBD instances, in which case both their primary
-and secondary nodes will need to be in the same group. To support this
-we envision the following changes:
-
- - The cluster will have a default group, which will initially be
- - Instance allocation will happen to the cluster's default group
- (which will be changable via gnt-cluster modify or RAPI) unless a
- group is explicitely specified in the creation job (with -g or via
- RAPI). Iallocator will be only passed the nodes belonging to that
+With the introduction of node groups, instances will be required to live
+in only one group at a time; this is mostly important for DRBD
+instances, which will not be allowed to have their primary and secondary
+nodes in different node groups. To support this, we envision the
+following changes:
+
+ - The iallocator interface will be augmented, and node groups exposed,
+ so that plugins will be able to make a decision regarding the group
+ in which to place a new instance. By default, all node groups will
+ be considered, but it will be possible to include a list of groups
+ in the creation job, in which case the plugin will limit itself to
+ considering those; in both cases, the ``alloc_policy`` attribute
+ will be honored.
+ - If, on the other hand, a primary and secondary nodes are specified
+ for a new instance, they will be required to be on the same node
group.
- Moving an instance between groups can only happen via an explicit
operation, which for example in the case of DRBD will work by
performing internally a replace-disks, a migration, and a second
- replace-disks. It will be possible to cleanup an interrupted
+ replace-disks. It will be possible to clean up an interrupted
group-move operation.
- - Cluster verify will signal an error if an instance has been left
- mid-transition between groups.
- - Intra-group instance migration/failover will check that the target
- group will be able to accept the instance network/storage wise, and
- fail otherwise. In the future we may be able to make some parameter
- changed during the move, but in the first version we expect an
- import/export if this is not possible.
- - From an allocation point of view, inter-group movements will be
- shown to a iallocator as a new allocation over the target group.
- Only in a future version we may add allocator extensions to decide
- which group the instance should be in. In the meantime we expect
- Ganeti administrators to either put instances on different groups by
- filling all groups first, or to have their own strategy based on the
- instance needs.
+ - Cluster verify will signal an error if an instance has nodes
+ belonging to different groups. Additionally, changing the group of a
+ given node will be initially only allowed if the node is empty, as a
+ straightforward mechanism to avoid creating such situation.
+ - Inter-group instance migration will have the same operation modes as
+ new instance allocation, defined above: letting an iallocator plugin
+ decide the target group, possibly restricting the set of node groups
+ to consider, or specifying a target primary and secondary nodes. In
+ both cases, the target group or nodes must be able to accept the
+ instance network- and storage-wise; the operation will fail
+ otherwise, though in the future we may be able to allow some
+ parameter to be changed together with the move (in the meantime, an
+ import/export will be required in this scenario).
Internal changes
++++++++++++++++
We expect the following changes for cluster management:
- - Frequent multinode operations, such as os-diagnose or cluster-verify
- will act one group at a time. The default group will be used if none
- is passed. Command line tools will have a way to easily target all
- groups, by generating one job per group.
+ - Frequent multinode operations, such as os-diagnose or cluster-verify,
+ will act on one group at a time, which will have to be specified in
+ all cases, except for clusters with just one group. Command line
+ tools will also have a way to easily target all groups, by
+ generating one job per group.
- Groups will have a human-readable name, but will internally always
- be referenced by a UUID, which will be immutable. For example the
- cluster object will contain the UUID of the default group, each node
- will contain the UUID of the group it belongs to, etc. This is done
+ be referenced by a UUID, which will be immutable; for example, nodes
+ will contain the UUID of the group they belong to. This is done
to simplify referencing while keeping it easy to handle renames and
movements. If we see that this works well, we'll transition other
config objects (instances, nodes) to the same model.
Other work and future changes
+++++++++++++++++++++++++++++
-Commands like gnt-cluster command/copyfile will continue to work on the
-whole cluster, but it will be possible to target one group only by
-specifying it.
+Commands like ``gnt-cluster command``/``gnt-cluster copyfile`` will
+continue to work on the whole cluster, but it will be possible to target
+one group only by specifying it.
Commands which allow selection of sets of resources (for example
-gnt-instance start/stop) will be able to select them by node group as
-well.
+``gnt-instance start``/``gnt-instance stop``) will be able to select
+them by node group as well.
Initially node groups won't be taggable objects, to simplify the first
implementation, but we expect this to be easy to add in a future version
should we see it's useful.
We envision groups as a good place to enhance cluster scalability. In
-the future we may want to use them ad units for configuration diffusion,
+the future we may want to use them as units for configuration diffusion,
to allow a better master scalability. For example it could be possible
to change some all-nodes RPCs to contact each group once, from the
master, and make one node in the group perform internal diffusion. We
won't implement this in the first version, but we'll evaluate it for the
future, if we see scalability problems on big multi-group clusters.
-When Ganeti will support more storage models (eg. SANs, sheepdog, ceph)
+When Ganeti will support more storage models (e.g. SANs, Sheepdog, Ceph)
we expect groups to be the basis for this, allowing for example a
-different sheepdog/ceph cluster, or a different SAN to be connected to
+different Sheepdog/Ceph cluster, or a different SAN to be connected to
each group. In some cases this will mean that inter-group move operation
will be necessarily performed with instance downtime, unless the
hypervisor has block-migrate functionality, and we implement support for
cluster state. While this is still acceptable for smaller clusters where
a small number of allocations/removal are presumed to occur between two
periodic capacity calculations, on bigger clusters where we aim to
-parallelise heavily between node groups this is no longer true.
+parallelize heavily between node groups this is no longer true.
- the total node memory, CPU count are very seldom changing; the total
node disk space is also slow changing, but can change at runtime; the
free memory and free disk will change significantly for some jobs, but
- on a short timescale; in general, these values will mostly “constant”
+ on a short timescale; in general, these values will be mostly “constant”
during the lifetime of a job
- we already have a periodic set of jobs that query the node and
instance state, driven the by :command:`ganeti-watcher` command, and
we're just discarding the results after acting on them
-Given the above, it makes sense to cache inside the master daemon the
-results of node and instance state (with a focus on the node state).
+Given the above, it makes sense to cache the results of node and instance
+state (with a focus on the node state) inside the master daemon.
The cache will not be serialised to disk, and will be for the most part
transparent to the outside of the master daemon.
consistent). Partial results will not update the cache (see next
paragraph).
-Since the there will be no way to feed the cache from outside, and we
+Since there will be no way to feed the cache from outside, and we
would like to have a consistent cache view when driven by the watcher,
we'll introduce a new OpCode/LU for the watcher to run, instead of the
current separate opcodes (see below in the watcher section).
node will not invalidate the capacity, as we're more interested in “at
least available” correctness, not “at most available”.
-Cache invalidations
-+++++++++++++++++++
+Cache invalidation
+++++++++++++++++++
If a partial node query is done (e.g. just for the node free space), and
the returned values don't match with the cache, then the entire node
instances they lock. If an LU uses the BGL, then it will invalidate the
entire cache. In time, it is expected that LUs will be modified to not
invalidate, if they are not expected to change the node's and/or
-instance's state (e.g. ``LUConnectConsole``, or
-``LUActivateInstanceDisks``).
+instance's state (e.g. ``LUInstanceConsole``, or
+``LUInstanceActivateDisks``).
Invalidation of a node's properties will also invalidate the capacity
data associated with that node.
allocation on one group from exclusive blocking jobs on other node
groups.
-The capacity calculations will also use the cache—this is detailed in
+The capacity calculations will also use the cache. This is detailed in
the respective sections.
Watcher operation
This method will feed the cluster state (for the complete set of node
group, or alternative just a subset) to the iallocator plugin (either
-the specified one, or the default is none is specified), and return the
+the specified one, or the default if none is specified), and return the
new capacity in the format currently exported by the htools suite and
known as the “tiered specs” (see :manpage:`hspace(1)`).
specification, the total available count is the count for the given
entry, plus the sum of counts for higher specifications.
-Also note that the node group information is provided just
-informationally, not for allocation decisions.
+Node flags
+----------
+
+Current state and shortcomings
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Currently all nodes are, from the point of view of their capabilities,
+homogeneous. This means the cluster considers all nodes capable of
+becoming master candidates, and of hosting instances.
+
+This prevents some deployment scenarios: e.g. having a Ganeti instance
+(in another cluster) be just a master candidate, in case all other
+master candidates go down (but not, of course, host instances), or
+having a node in a remote location just host instances but not become
+master, etc.
+
+Proposed changes
+~~~~~~~~~~~~~~~~
+
+Two new capability flags will be added to the node:
+
+- master_capable, denoting whether the node can become a master
+ candidate or master
+- vm_capable, denoting whether the node can host instances
+
+In terms of the other flags, master_capable is a stronger version of
+"not master candidate", and vm_capable is a stronger version of
+"drained".
+
+For the master_capable flag, it will affect auto-promotion code and node
+modifications.
+
+The vm_capable flag will affect the iallocator protocol, capacity
+calculations, node checks in cluster verify, and will interact in novel
+ways with locking (unfortunately).
+
+It is envisaged that most nodes will be both vm_capable and
+master_capable, and just a few will have one of these flags
+removed. Ganeti itself will allow clearing of both flags, even though
+this doesn't make much sense currently.
+
+
+.. _jqueue-job-priority-design:
Job priorities
--------------
A job's priority can never go below -20. If a job hits priority -20, it
must acquire its locks in blocking mode.
-Opcode priorities are synchronized to disk in order to be restored after
+Opcode priorities are synchronised to disk in order to be restored after
a restart or crash of the master daemon.
Priorities also need to be considered inside the locking library to
netutils: Utilities for handling common network tasks
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Currently common util functions are kept in the utils modules. Since
-this module grows bigger and bigger network-related functions are moved
-to a separate module named *netutils*. Additionally all these utilities
-will be IPv6-enabled.
+Currently common utility functions are kept in the ``utils`` module.
+Since this module grows bigger and bigger network-related functions are
+moved to a separate module named *netutils*. Additionally all these
+utilities will be IPv6-enabled.
Cluster initialization
~~~~~~~~~~~~~~~~~~~~~~
Privilege Separation
--------------------
-Current state and short comings
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-As of Ganeti 2.2 we introduced privilege separation. This was affecting
-just Ganeti RAPI and also that just in a quickly short term solution. In
-this release we iterate again over it and make it more advanced and
-stable. This also means we'll remove the privilege separation again from
-the core and put it completely external so the daemons will be started
-on the final user already.
+Current state and shortcomings
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Additionally this involves removing SSH code out auf bootstrap and core
-component and put it into a separate script. This means every
-daemon/script will assume that a working ssh setup is in place.
+In Ganeti 2.2 we introduced privilege separation for the RAPI daemon.
+This was done directly in the daemon's code in the process of
+daemonizing itself. Doing so leads to several potential issues. For
+example, a file could be opened while the code is still running as
+``root`` and for some reason not be closed again. Even after changing
+the user ID, the file descriptor can be written to.
Implementation
~~~~~~~~~~~~~~
-We need to partially revert changes done in Ganeti 2.2 to move on the
-long term solution. This involves removing the drop privileges code in
-``daemons.py`` as this is already done on startup time by
-``start-stop-daemon`` util.
+To address these shortcomings, daemons will be started under the target
+user right away. The ``start-stop-daemon`` utility used to start daemons
+supports the ``--chuid`` option to change user and group ID before
+starting the executable.
+
+The intermediate solution for the RAPI daemon from Ganeti 2.2 will be
+removed again.
-The ssh code will be separated into one single script called upon
-``gnt-node add`` which guarantees that the SSH setup is done and
-functioning.
+Files written by the daemons may need to have an explicit owner and
+group set (easily done through ``utils.WriteFile``).
-Additionally some of the utils.WriteFile calls needs to be adjusted
-for the new permissions and ownerships.
+All SSH-related code is removed from the ``ganeti.bootstrap`` module and
+core components and moved to a separate script. The core code will
+simply assume a working SSH setup to be in place.
Security Domains
~~~~~~~~~~~~~~~~
1. Public: ``0755`` respectively ``0644``
2. Ganeti wide: shared between the daemons (gntdaemons)
-3. Secret files: shared just between a specified set of daemons/users
+3. Secret files: shared among a specific set of daemons/users
So for point 3 this tables shows the correlation of the sets to groups
and their users:
Set Group Users Description
=== ========== ============================== ==========================
A gntrapi gntrapi, gntmasterd Share data between
- gntrapi & gntmasterd
+ gntrapi and gntmasterd
B gntadmins gntrapi, gntmasterd, *users* Shared between users who
needs to call gntmasterd
C gntconfd gntconfd, gntmasterd Share data between
- gntconfd & gntmasterd
+ gntconfd and gntmasterd
D gntmasterd gntmasterd masterd only; Currently
only to redistribute the
configuration, has access
gnt-node {add|remove}
gnt-instance {console}
-Directory structure & permissions
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Directory structure and permissions
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Here's how we propose to change the filesystem hierachy and their
+Here's how we propose to change the filesystem hierarchy and their
permissions.
Assuming it follows the defaults: ``gnt${daemon}`` for user and