this node's hardware resources; it runs on all nodes which are in a
cluster
- the :command:`ganeti-confd` daemon (Ganeti 2.1+) which runs on all
- nodes, but is only functional on master candidate nodes
+ nodes, but is only functional on master candidate nodes; this daemon
+ can be disabled at configuration time if you don't need its
+ functionality
- the :command:`ganeti-rapi` daemon which runs on the master node and
offers an HTTP-based API for the cluster
- the :command:`ganeti-masterd` daemon which runs on the master node and
allows control of the cluster
+Beside the node role, there are other node flags that influence its
+behaviour:
+
+- the *master_capable* flag denotes whether the node can ever become a
+ master candidate; setting this to 'no' means that auto-promotion will
+ never make this node a master candidate; this flag can be useful for a
+ remote node that only runs local instances, and having it become a
+ master is impractical due to networking or other constraints
+- the *vm_capable* flag denotes whether the node can host instances or
+ not; for example, one might use a non-vm_capable node just as a master
+ candidate, for configuration backups; setting this flag to no
+ disallows placement of instances of this node, deactivates hypervisor
+ and related checks on it (e.g. bridge checks, LVM check, etc.), and
+ removes it from cluster capacity computations
+
+
Instance
~~~~~~~~
the instance sees the same virtual drive in all cases, the node-level
configuration varies between them.
-There are four disk templates you can choose from:
+There are five disk templates you can choose from:
diskless
The instance has no disks. Only used for special purpose operating
to obtain a highly available instance that can be failed over to a
remote node should the primary one fail.
+rbd
+ The instance will use Volumes inside a RADOS cluster as backend for its
+ disks. It will access them using the RADOS block device (RBD).
+
IAllocator
~~~~~~~~~~
internall by Ganeti as an *OpCode* (abbreviation from operation
code). These OpCodes are executed as part of a *Job*. The OpCodes in a
single Job are processed serially by Ganeti, but different Jobs will be
-processed (depending on resource availability) in parallel.
+processed (depending on resource availability) in parallel. They will
+not be executed in the submission order, but depending on resource
+availability, locks and (starting with Ganeti 2.3) priority. An earlier
+job may have to wait for a lock while a newer job doesn't need any locks
+and can be executed right away. Operations requiring a certain order
+need to be submitted as a single job, or the client must submit one job
+at a time and wait for it to finish before continuing.
For example, shutting down the entire cluster can be done by running the
command ``gnt-instance shutdown --all``, which will submit for each
The above command has the minimum required options; other options you
can give include, among others:
-- The memory size (``-B memory``)
+- The maximum/minimum memory size (``-B maxmem``, ``-B minmem``)
+ (``-B memory`` can be used to specify only one size)
- The number of virtual CPUs (``-B vcpus``)
gnt-instance remove INSTANCE_NAME
+.. _instance-startup-label:
+
Startup/shutdown
~~~~~~~~~~~~~~~~
gnt-instance startup INSTANCE_NAME
-While the command to stop one is::
+Ganeti will start an instance with up to its maximum instance memory. If
+not enough memory is available Ganeti will use all the available memory
+down to the instance minumum memory. If not even that amount of memory
+is free Ganeti will refuse to start the instance.
+
+Note, that this will not work when an instance is in a permanently
+stopped state ``offline``. In this case, you will first have to
+put it back to online mode by running::
+
+ gnt-instance modify --online INSTANCE_NAME
+
+The command to stop the running instance is::
gnt-instance shutdown INSTANCE_NAME
+If you want to shut the instance down more permanently, so that it
+does not require dynamically allocated resources (memory and vcpus),
+after shutting down an instance, execute the following::
+
+ gnt-instance modify --offline INSTANCE_NAME
+
.. warning:: Do not use the Xen or KVM commands directly to stop
instances. If you run for example ``xm shutdown`` or ``xm destroy``
- on an instance Ganeti will automatically restart it (via the
- :command:`ganeti-watcher` command which is launched via cron).
+ on an instance Ganeti will automatically restart it (via
+ the :command:`ganeti-watcher` command which is launched via cron).
Querying instances
~~~~~~~~~~~~~~~~~~
status), etc. This is harder to parse and is more expensive than the
list operation, but returns much more detailed information.
+Changing an instance's runtime memory
++++++++++++++++++++++++++++++++++++++
+
+Ganeti will always make sure an instance has a value between its maximum
+and its minimum memory available as runtime memory. As of version 2.6
+Ganeti will only choose a size different than the maximum size when
+starting up, failing over, or migrating an instance on a node with less
+than the maximum memory available. It won't resize other instances in
+order to free up space for an instance.
+
+If you find that you need more memory on a node any instance can be
+manually resized without downtime, with the command::
+
+ gnt-instance modify -m SIZE INSTANCE_NAME
+
+The same command can also be used to increase the memory available on an
+instance, provided that enough free memory is available on its node, and
+the specified size is not larger than the maximum memory size the
+instance had when it was first booted (an instance will be unable to see
+new memory above the maximum that was specified to the hypervisor at its
+boot time, if it needs to grow further a reboot becomes necessary).
Export/Import
+++++++++++++
passing the disk information you specify the current volumes::
gnt-instance add -t plain -n HOME_NODE ... \
- --disk 0:adopt=lv_name INSTANCE_NAME
+ --disk 0:adopt=lv_name[,vg=vg_name] INSTANCE_NAME
This will take over the given logical volumes, rename them to the Ganeti
standard (UUID-based), and without installing the OS on them start
be seamless for the instance. For more than one disk, just pass another
disk parameter (e.g. ``--disk 1:adopt=...``).
+Instance kernel selection
++++++++++++++++++++++++++
+
+The kernel that instances uses to bootup can come either from the node,
+or from instances themselves, depending on the setup.
+
+Xen-PVM
+~~~~~~~
+
+With Xen PVM, there are three options.
+
+First, you can use a kernel from the node, by setting the hypervisor
+parameters as such:
+
+- ``kernel_path`` to a valid file on the node (and appropriately
+ ``initrd_path``)
+- ``kernel_args`` optionally set to a valid Linux setting (e.g. ``ro``)
+- ``root_path`` to a valid setting (e.g. ``/dev/xvda1``)
+- ``bootloader_path`` and ``bootloader_args`` to empty
+
+Alternatively, you can delegate the kernel management to instances, and
+use either ``pvgrub`` or the deprecated ``pygrub``. For this, you must
+install the kernels and initrds in the instance and create a valid GRUB
+v1 configuration file.
+
+For ``pvgrub`` (new in version 2.4.2), you need to set:
+
+- ``kernel_path`` to point to the ``pvgrub`` loader present on the node
+ (e.g. ``/usr/lib/xen/boot/pv-grub-x86_32.gz``)
+- ``kernel_args`` to the path to the GRUB config file, relative to the
+ instance (e.g. ``(hd0,0)/grub/menu.lst``)
+- ``root_path`` **must** be empty
+- ``bootloader_path`` and ``bootloader_args`` to empty
+
+While ``pygrub`` is deprecated, here is how you can configure it:
+
+- ``bootloader_path`` to the pygrub binary (e.g. ``/usr/bin/pygrub``)
+- the other settings are not important
+
+More information can be found in the Xen wiki pages for `pvgrub
+<http://wiki.xensource.com/xenwiki/PvGrub>`_ and `pygrub
+<http://wiki.xensource.com/xenwiki/PyGrub>`_.
+
+KVM
+~~~
+
+For KVM also the kernel can be loaded either way.
+
+For loading the kernels from the node, you need to set:
+
+- ``kernel_path`` to a valid value
+- ``initrd_path`` optionally set if you use an initrd
+- ``kernel_args`` optionally set to a valid value (e.g. ``ro``)
+
+If you want instead to have the instance boot from its disk (and execute
+its bootloader), simply set the ``kernel_path`` parameter to an empty
+string, and all the others will be ignored.
+
Instance HA features
--------------------
That's it. After the command completes the secondary node is now the
primary, and vice-versa.
+The instance will be started with an amount of memory between its
+``maxmem`` and its ``minmem`` value, depending on the free memory on its
+target node, or the operation will fail if that's not possible. See
+:ref:`instance-startup-label` for details.
+
+If the instance's disk template is of type rbd, then you can specify
+the target node (which can be any node) explicitly, or specify an
+iallocator plugin. If you omit both, the default iallocator will be
+used to determine the target node::
+
+ gnt-instance failover -n TARGET_NODE INSTANCE_NAME
+
Live migrating an instance
~~~~~~~~~~~~~~~~~~~~~~~~~~
long the migration will take. In any case, for both KVM and Xen
hypervisors, the migration will be transparent to the instance.
+If the destination node has less memory than the instance's current
+runtime memory, but at least the instance's minimum memory available
+Ganeti will automatically reduce the instance runtime memory before
+migrating it, unless the ``--no-runtime-changes`` option is passed, in
+which case the target node should have at least the instance's current
+runtime memory free.
+
+If the instance's disk template is of type rbd, then you can specify
+the target node (which can be any node) explicitly, or specify an
+iallocator plugin. If you omit both, the default iallocator will be
+used to determine the target node::
+
+ gnt-instance migrate -n TARGET_NODE INSTANCE_NAME
+
Moving an instance (offline)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Since the process involves copying all data from the working node to the
target node, it will take a while, depending on the instance's disk
-size, node I/O system and network speed. But it is (baring any network
+size, node I/O system and network speed. But it is (barring any network
interruption) completely transparent for the instance.
Re-creating disks for non-redundant instances
# later convert it to redundant
gnt-instance stop INSTANCE
- gnt-instance modify -t drbd INSTANCE
+ gnt-instance modify -t drbd -n NEW_SECONDARY INSTANCE
gnt-instance start INSTANCE
# and convert it back
If you want to promote a different node to the master role (for whatever
reason), run on any other master-candidate node the command::
- gnt-cluster masterfailover
+ gnt-cluster master-failover
and the node you ran it on is now the new master. In case you try to run
this on a non master-candidate node, you will get an error telling you
6. Remove the ganeti state directory (``rm -rf /var/lib/ganeti/*``),
replacing the path with the correct path for your installation.
+7. If using RBD, run ``rbd unmap /dev/rbdN`` to unmap the RBD disks.
+ Then remove the RBD disk images used by Ganeti, identified by their
+ UUIDs (``rbd rm uuid.rbd.diskN``).
+
On the master node, remove the cluster from the master-netdev (usually
``xen-br0`` for bridged mode, otherwise ``eth0`` or similar), by running
``ip a del $clusterip/32 dev xen-br0`` (use the correct cluster ip and
Note that the set of characters present in a tag and the maximum tag
length are restricted. Currently the maximum length is 128 characters,
there can be at most 4096 tags per object, and the set of characters is
-comprised by alphanumeric characters and additionally ``.+*/:-``.
+comprised by alphanumeric characters and additionally ``.+*/:@-``.
Operations
++++++++++
Mon Oct 26 00:22:52 2009 adding instance instance1 to cluster config
Mon Oct 26 00:22:52 2009 - INFO: Waiting for instance instance1 to sync disks.
…
- Mon Oct 26 00:23:03 2009 creating os for instance xen-devi-18.fra.corp.google.com on node mpgntac4.fra.corp.google.com
+ Mon Oct 26 00:23:03 2009 creating os for instance instance1 on node node1
Mon Oct 26 00:23:03 2009 * running the instance OS create scripts...
Mon Oct 26 00:23:13 2009 * starting instance...
node1#
There are two queues for jobs: the *current* and the *archive*
queue. Jobs are initially submitted to the current queue, and they stay
in that queue until they have finished execution (either successfully or
-not). At that point, they can be moved into the archive queue, and the
-ganeti-watcher script will do this automatically after 6 hours. The
-ganeti-cleaner script will remove the jobs from the archive directory
+not). At that point, they can be moved into the archive queue using e.g.
+``gnt-job autoarchive all``. The ``ganeti-watcher`` script will do this
+automatically 6 hours after a job is finished. The ``ganeti-cleaner``
+script will then remove archived the jobs from the archive directory
after three weeks.
-Note that only jobs in the current queue can be viewed via the list and
-info commands; Ganeti itself doesn't examine the archive directory. If
-you need to see an older job, either move the file manually in the
-top-level queue directory, or look at its contents (it's a
-JSON-formatted file).
+Note that ``gnt-job list`` only shows jobs in the current queue.
+Archived jobs can be viewed using ``gnt-job info <id>``.
+
+Special Ganeti deployments
+--------------------------
+
+Since Ganeti 2.4, it is possible to extend the Ganeti deployment with
+two custom scenarios: Ganeti inside Ganeti and multi-site model.
+
+Running Ganeti under Ganeti
++++++++++++++++++++++++++++
+
+It is sometimes useful to be able to use a Ganeti instance as a Ganeti
+node (part of another cluster, usually). One example scenario is two
+small clusters, where we want to have an additional master candidate
+that holds the cluster configuration and can be used for helping with
+the master voting process.
+
+However, these Ganeti instance should not host instances themselves, and
+should not be considered in the normal capacity planning, evacuation
+strategies, etc. In order to accomplish this, mark these nodes as
+non-``vm_capable``::
+
+ node1# gnt-node modify --vm-capable=no node3
+
+The vm_capable status can be listed as usual via ``gnt-node list``::
+
+ node1# gnt-node list -oname,vm_capable
+ Node VMCapable
+ node1 Y
+ node2 Y
+ node3 N
+
+When this flag is set, the cluster will not do any operations that
+relate to instances on such nodes, e.g. hypervisor operations,
+disk-related operations, etc. Basically they will just keep the ssconf
+files, and if master candidates the full configuration.
+
+Multi-site model
+++++++++++++++++
+
+If Ganeti is deployed in multi-site model, with each site being a node
+group (so that instances are not relocated across the WAN by mistake),
+it is conceivable that either the WAN latency is high or that some sites
+have a lower reliability than others. In this case, it doesn't make
+sense to replicate the job information across all sites (or even outside
+of a “central” node group), so it should be possible to restrict which
+nodes can become master candidates via the auto-promotion algorithm.
+
+Ganeti 2.4 introduces for this purpose a new ``master_capable`` flag,
+which (when unset) prevents nodes from being marked as master
+candidates, either manually or automatically.
+
+As usual, the node modify operation can change this flag::
+
+ node1# gnt-node modify --auto-promote --master-capable=no node3
+ Fri Jan 7 06:23:07 2011 - INFO: Demoting from master candidate
+ Fri Jan 7 06:23:08 2011 - INFO: Promoted nodes to master candidate role: node4
+ Modified node node3
+ - master_capable -> False
+ - master_candidate -> False
+
+And the node list operation will list this flag::
+
+ node1# gnt-node list -oname,master_capable node1 node2 node3
+ Node MasterCapable
+ node1 Y
+ node2 Y
+ node3 N
+
+Note that marking a node both not ``vm_capable`` and not
+``master_capable`` makes the node practically unusable from Ganeti's
+point of view. Hence these two flags should be used probably in
+contrast: some nodes will be only master candidates (master_capable but
+not vm_capable), and other nodes will only hold instances (vm_capable
+but not master_capable).
+
Ganeti tools
------------
More information about the upgrade procedure is listed on the wiki at
http://code.google.com/p/ganeti/wiki/UpgradeNotes.
+There is also a script designed to upgrade from Ganeti 1.2 to 2.0,
+called ``cfgupgrade12``.
+
cfgshell
++++++++
instance OS definitions are executing properly the rename, import and
export operations.
+sanitize-config
++++++++++++++++
+
+This tool takes the Ganeti configuration and outputs a "sanitized"
+version, by randomizing or clearing:
+
+- DRBD secrets and cluster public key (always)
+- host names (optional)
+- IPs (optional)
+- OS names (optional)
+- LV names (optional, only useful for very old clusters which still have
+ instances whose LVs are based on the instance name)
+
+By default, all optional items are activated except the LV name
+randomization. When passing ``--no-randomization``, which disables the
+optional items (i.e. just the DRBD secrets and cluster public keys are
+randomized), the resulting file can be used as a safety copy of the
+cluster config - while not trivial, the layout of the cluster can be
+recreated from it and if the instance disks have not been lost it
+permits recovery from the loss of all master candidates.
+
+move-instance
++++++++++++++
+
+See :doc:`separate documentation for move-instance <move-instance>`.
+
+.. TODO: document cluster-merge tool
+
+
Other Ganeti projects
---------------------
-There are two other Ganeti-related projects that can be useful in a
-Ganeti deployment. These can be downloaded from the project site
-(http://code.google.com/p/ganeti/) and the repositories are also on the
-project git site (http://git.ganeti.org).
+Below is a list (which might not be up-to-date) of additional projects
+that can be useful in a Ganeti deployment. They can be downloaded from
+the project site (http://code.google.com/p/ganeti/) and the repositories
+are also on the project git site (http://git.ganeti.org).
NBMA tools
++++++++++
ganeti-htools
+++++++++++++
-The ``ganeti-htools`` software consists of a set of tools:
-
-- ``hail``: an advanced iallocator script compared to Ganeti's builtin
- one
-- ``hbal``: a tool for rebalancing the cluster, i.e. moving instances
- around in order to better use the resources on the nodes
-- ``hspace``: a tool for estimating the available capacity of a cluster,
- so that capacity planning can be done efficiently
+Before Ganeti version 2.5, this was a standalone project; since that
+version it is integrated into the Ganeti codebase (see
+:doc:`install-quick` for instructions on how to enable it). If you run
+an older Ganeti version, you will have to download and build it
+separately.
For more information and installation instructions, see the README file
in the source archive.
projects / ganeti-local / blobdiff