+.. _instance-relocation-label:
+
+Instance relocation
+~~~~~~~~~~~~~~~~~~~
+
+While it is not possible to move an instance from nodes ``(A, B)`` to
+nodes ``(C, D)`` in a single move, it is possible to do so in a few
+steps::
+
+ # instance is located on A, B
+ node1# gnt-instance replace -n nodeC instance1
+ # instance has moved from (A, B) to (A, C)
+ # we now flip the primary/secondary nodes
+ node1# gnt-instance migrate instance1
+ # instance lives on (C, A)
+ # we can then change A to D via:
+ node1# gnt-instance replace -n nodeD instance1
+
+Which brings it into the final configuration of ``(C, D)``. Note that we
+needed to do two replace-disks operation (two copies of the instance
+disks), because we needed to get rid of both the original nodes (A and
+B).
+
+Node operations
+---------------
+
+There are much fewer node operations available than for instances, but
+they are equivalently important for maintaining a healthy cluster.
+
+Add/readd
++++++++++
+
+It is at any time possible to extend the cluster with one more node, by
+using the node add operation::
+
+ gnt-node add NEW_NODE
+
+If the cluster has a replication network defined, then you need to pass
+the ``-s REPLICATION_IP`` parameter to this option.
+
+A variation of this command can be used to re-configure a node if its
+Ganeti configuration is broken, for example if it has been reinstalled
+by mistake::
+
+ gnt-node add --readd EXISTING_NODE
+
+This will reinitialise the node as if it's been newly added, but while
+keeping its existing configuration in the cluster (primary/secondary IP,
+etc.), in other words you won't need to use ``-s`` here.
+
+Changing the node role
+++++++++++++++++++++++
+
+A node can be in different roles, as explained in the
+:ref:`terminology-label` section. Promoting a node to the master role is
+special, while the other roles are handled all via a single command.
+
+Failing over the master node
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you want to promote a different node to the master role (for whatever
+reason), run on any other master-candidate node the command::
+
+ gnt-cluster master-failover
+
+and the node you ran it on is now the new master. In case you try to run
+this on a non master-candidate node, you will get an error telling you
+which nodes are valid.
+
+Changing between the other roles
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``gnt-node modify`` command can be used to select a new role::
+
+ # change to master candidate
+ gnt-node modify -C yes NODE
+ # change to drained status
+ gnt-node modify -D yes NODE
+ # change to offline status
+ gnt-node modify -O yes NODE
+ # change to regular mode (reset all flags)
+ gnt-node modify -O no -D no -C no NODE
+
+Note that the cluster requires that at any point in time, a certain
+number of nodes are master candidates, so changing from master candidate
+to other roles might fail. It is recommended to either force the
+operation (via the ``--force`` option) or first change the number of
+master candidates in the cluster - see :ref:`cluster-config-label`.
+
+Evacuating nodes
+++++++++++++++++
+
+There are two steps of moving instances off a node:
+
+- moving the primary instances (actually converting them into secondary
+ instances)
+- moving the secondary instances (including any instances converted in
+ the step above)
+
+Primary instance conversion
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For this step, you can use either individual instance move
+commands (as seen in :ref:`instance-change-primary-label`) or the bulk
+per-node versions; these are::
+
+ gnt-node migrate NODE
+ gnt-node evacuate NODE
+
+Note that the instance “move” command doesn't currently have a node
+equivalent.
+
+Both these commands, or the equivalent per-instance command, will make
+this node the secondary node for the respective instances, whereas their
+current secondary node will become primary. Note that it is not possible
+to change in one step the primary node to another node as primary, while
+keeping the same secondary node.
+
+Secondary instance evacuation
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For the evacuation of secondary instances, a command called
+:command:`gnt-node evacuate` is provided and its syntax is::
+
+ gnt-node evacuate -I IALLOCATOR_SCRIPT NODE
+ gnt-node evacuate -n DESTINATION_NODE NODE
+
+The first version will compute the new secondary for each instance in
+turn using the given iallocator script, whereas the second one will
+simply move all instances to DESTINATION_NODE.
+
+Removal
++++++++
+
+Once a node no longer has any instances (neither primary nor secondary),
+it's easy to remove it from the cluster::
+
+ gnt-node remove NODE_NAME
+
+This will deconfigure the node, stop the ganeti daemons on it and leave
+it hopefully like before it joined to the cluster.
+
+Storage handling
+++++++++++++++++
+
+When using LVM (either standalone or with DRBD), it can become tedious
+to debug and fix it in case of errors. Furthermore, even file-based
+storage can become complicated to handle manually on many hosts. Ganeti
+provides a couple of commands to help with automation.
+
+Logical volumes
+~~~~~~~~~~~~~~~
+
+This is a command specific to LVM handling. It allows listing the
+logical volumes on a given node or on all nodes and their association to
+instances via the ``volumes`` command::
+
+ node1# gnt-node volumes
+ Node PhysDev VG Name Size Instance
+ node1 /dev/sdb1 xenvg e61fbc97-….disk0 512M instance17
+ node1 /dev/sdb1 xenvg ebd1a7d1-….disk0 512M instance19
+ node2 /dev/sdb1 xenvg 0af08a3d-….disk0 512M instance20
+ node2 /dev/sdb1 xenvg cc012285-….disk0 512M instance16
+ node2 /dev/sdb1 xenvg f0fac192-….disk0 512M instance18
+
+The above command maps each logical volume to a volume group and
+underlying physical volume and (possibly) to an instance.
+
+.. _storage-units-label:
+
+Generalized storage handling
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. versionadded:: 2.1
+
+Starting with Ganeti 2.1, a new storage framework has been implemented
+that tries to abstract the handling of the storage type the cluster
+uses.
+
+First is listing the backend storage and their space situation::
+
+ node1# gnt-node list-storage
+ Node Name Size Used Free
+ node1 /dev/sda7 673.8G 0M 673.8G
+ node1 /dev/sdb1 698.6G 1.5G 697.1G
+ node2 /dev/sda7 673.8G 0M 673.8G
+ node2 /dev/sdb1 698.6G 1.0G 697.6G
+
+The default is to list LVM physical volumes. It's also possible to list
+the LVM volume groups::
+
+ node1# gnt-node list-storage -t lvm-vg
+ Node Name Size
+ node1 xenvg 1.3T
+ node2 xenvg 1.3T
+
+Next is repairing storage units, which is currently only implemented for
+volume groups and does the equivalent of ``vgreduce --removemissing``::
+
+ node1# gnt-node repair-storage node2 lvm-vg xenvg
+ Sun Oct 25 22:21:45 2009 Repairing storage unit 'xenvg' on node2 ...
+
+Last is the modification of volume properties, which is (again) only
+implemented for LVM physical volumes and allows toggling the
+``allocatable`` value::
+
+ node1# gnt-node modify-storage --allocatable=no node2 lvm-pv /dev/sdb1
+
+Use of the storage commands
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+All these commands are needed when recovering a node from a disk
+failure:
+
+- first, we need to recover from complete LVM failure (due to missing
+ disk), by running the ``repair-storage`` command
+- second, we need to change allocation on any partially-broken disk
+ (i.e. LVM still sees it, but it has bad blocks) by running
+ ``modify-storage``
+- then we can evacuate the instances as needed
+
+
+Cluster operations
+------------------
+
+Beside the cluster initialisation command (which is detailed in the
+:doc:`install` document) and the master failover command which is
+explained under node handling, there are a couple of other cluster
+operations available.
+
+.. _cluster-config-label:
+
+Standard operations
++++++++++++++++++++
+
+One of the few commands that can be run on any node (not only the
+master) is the ``getmaster`` command::
+
+ node2# gnt-cluster getmaster
+ node1.example.com
+ node2#
+
+It is possible to query and change global cluster parameters via the
+``info`` and ``modify`` commands::
+
+ node1# gnt-cluster info
+ Cluster name: cluster.example.com
+ Cluster UUID: 07805e6f-f0af-4310-95f1-572862ee939c
+ Creation time: 2009-09-25 05:04:15
+ Modification time: 2009-10-18 22:11:47
+ Master node: node1.example.com
+ Architecture (this node): 64bit (x86_64)
+ …
+ Tags: foo
+ Default hypervisor: xen-pvm
+ Enabled hypervisors: xen-pvm
+ Hypervisor parameters:
+ - xen-pvm:
+ root_path: /dev/sda1
+ …
+ Cluster parameters:
+ - candidate pool size: 10
+ …
+ Default instance parameters:
+ - default:
+ memory: 128
+ …
+ Default nic parameters:
+ - default:
+ link: xen-br0
+ …
+
+There various parameters above can be changed via the ``modify``
+commands as follows:
+
+- the hypervisor parameters can be changed via ``modify -H
+ xen-pvm:root_path=…``, and so on for other hypervisors/key/values
+- the "default instance parameters" are changeable via ``modify -B
+ parameter=value…`` syntax
+- the cluster parameters are changeable via separate options to the
+ modify command (e.g. ``--candidate-pool-size``, etc.)
+
+For detailed option list see the :manpage:`gnt-cluster(8)` man page.
+
+The cluster version can be obtained via the ``version`` command::
+ node1# gnt-cluster version
+ Software version: 2.1.0
+ Internode protocol: 20
+ Configuration format: 2010000
+ OS api version: 15
+ Export interface: 0
+
+This is not very useful except when debugging Ganeti.
+
+Global node commands
+++++++++++++++++++++
+
+There are two commands provided for replicating files to all nodes of a
+cluster and for running commands on all the nodes::
+
+ node1# gnt-cluster copyfile /path/to/file
+ node1# gnt-cluster command ls -l /path/to/file
+
+These are simple wrappers over scp/ssh and more advanced usage can be
+obtained using :manpage:`dsh(1)` and similar commands. But they are
+useful to update an OS script from the master node, for example.
+
+Cluster verification
+++++++++++++++++++++
+
+There are three commands that relate to global cluster checks. The first
+one is ``verify`` which gives an overview on the cluster state,
+highlighting any issues. In normal operation, this command should return
+no ``ERROR`` messages::
+
+ node1# gnt-cluster verify
+ Sun Oct 25 23:08:58 2009 * Verifying global settings
+ Sun Oct 25 23:08:58 2009 * Gathering data (2 nodes)
+ Sun Oct 25 23:09:00 2009 * Verifying node status
+ Sun Oct 25 23:09:00 2009 * Verifying instance status
+ Sun Oct 25 23:09:00 2009 * Verifying orphan volumes
+ Sun Oct 25 23:09:00 2009 * Verifying remaining instances
+ Sun Oct 25 23:09:00 2009 * Verifying N+1 Memory redundancy
+ Sun Oct 25 23:09:00 2009 * Other Notes
+ Sun Oct 25 23:09:00 2009 - NOTICE: 5 non-redundant instance(s) found.
+ Sun Oct 25 23:09:00 2009 * Hooks Results
+
+The second command is ``verify-disks``, which checks that the instance's
+disks have the correct status based on the desired instance state
+(up/down)::
+
+ node1# gnt-cluster verify-disks
+
+Note that this command will show no output when disks are healthy.
+
+The last command is used to repair any discrepancies in Ganeti's
+recorded disk size and the actual disk size (disk size information is
+needed for proper activation and growth of DRBD-based disks)::
+
+ node1# gnt-cluster repair-disk-sizes
+ Sun Oct 25 23:13:16 2009 - INFO: Disk 0 of instance instance1 has mismatched size, correcting: recorded 512, actual 2048
+ Sun Oct 25 23:13:17 2009 - WARNING: Invalid result from node node4, ignoring node results
+
+The above shows one instance having wrong disk size, and a node which
+returned invalid data, and thus we ignored all primary instances of that
+node.
+
+Configuration redistribution
+++++++++++++++++++++++++++++
+
+If the verify command complains about file mismatches between the master
+and other nodes, due to some node problems or if you manually modified
+configuration files, you can force an push of the master configuration
+to all other nodes via the ``redist-conf`` command::
+
+ node1# gnt-cluster redist-conf
+ node1#
+
+This command will be silent unless there are problems sending updates to
+the other nodes.
+
+
+Cluster renaming
+++++++++++++++++
+
+It is possible to rename a cluster, or to change its IP address, via the
+``rename`` command. If only the IP has changed, you need to pass the
+current name and Ganeti will realise its IP has changed::
+
+ node1# gnt-cluster rename cluster.example.com
+ This will rename the cluster to 'cluster.example.com'. If
+ you are connected over the network to the cluster name, the operation
+ is very dangerous as the IP address will be removed from the node and
+ the change may not go through. Continue?
+ y/[n]/?: y
+ Failure: prerequisites not met for this operation:
+ Neither the name nor the IP address of the cluster has changed
+
+In the above output, neither value has changed since the cluster
+initialisation so the operation is not completed.
+
+Queue operations
+++++++++++++++++
+
+The job queue execution in Ganeti 2.0 and higher can be inspected,
+suspended and resumed via the ``queue`` command::
+
+ node1~# gnt-cluster queue info
+ The drain flag is unset
+ node1~# gnt-cluster queue drain
+ node1~# gnt-instance stop instance1
+ Failed to submit job for instance1: Job queue is drained, refusing job
+ node1~# gnt-cluster queue info
+ The drain flag is set
+ node1~# gnt-cluster queue undrain
+
+This is most useful if you have an active cluster and you need to
+upgrade the Ganeti software, or simply restart the software on any node:
+
+#. suspend the queue via ``queue drain``
+#. wait until there are no more running jobs via ``gnt-job list``
+#. restart the master or another node, or upgrade the software
+#. resume the queue via ``queue undrain``
+
+.. note:: this command only stores a local flag file, and if you
+ failover the master, it will not have effect on the new master.
+
+
+Watcher control
++++++++++++++++
+
+The :manpage:`ganeti-watcher` is a program, usually scheduled via
+``cron``, that takes care of cluster maintenance operations (restarting
+downed instances, activating down DRBD disks, etc.). However, during
+maintenance and troubleshooting, this can get in your way; disabling it
+via commenting out the cron job is not so good as this can be
+forgotten. Thus there are some commands for automated control of the
+watcher: ``pause``, ``info`` and ``continue``::
+
+ node1~# gnt-cluster watcher info
+ The watcher is not paused.
+ node1~# gnt-cluster watcher pause 1h
+ The watcher is paused until Mon Oct 26 00:30:37 2009.
+ node1~# gnt-cluster watcher info
+ The watcher is paused until Mon Oct 26 00:30:37 2009.
+ node1~# ganeti-watcher -d
+ 2009-10-25 23:30:47,984: pid=28867 ganeti-watcher:486 DEBUG Pause has been set, exiting
+ node1~# gnt-cluster watcher continue
+ The watcher is no longer paused.
+ node1~# ganeti-watcher -d
+ 2009-10-25 23:31:04,789: pid=28976 ganeti-watcher:345 DEBUG Archived 0 jobs, left 0
+ 2009-10-25 23:31:05,884: pid=28976 ganeti-watcher:280 DEBUG Got data from cluster, writing instance status file
+ 2009-10-25 23:31:06,061: pid=28976 ganeti-watcher:150 DEBUG Data didn't change, just touching status file
+ node1~# gnt-cluster watcher info
+ The watcher is not paused.
+ node1~#
+
+The exact details of the argument to the ``pause`` command are available
+in the manpage.
+
+.. note:: this command only stores a local flag file, and if you
+ failover the master, it will not have effect on the new master.
+
+Node auto-maintenance
++++++++++++++++++++++
+
+If the cluster parameter ``maintain_node_health`` is enabled (see the
+manpage for :command:`gnt-cluster`, the init and modify subcommands),
+then the following will happen automatically:
+
+- the watcher will shutdown any instances running on offline nodes
+- the watcher will deactivate any DRBD devices on offline nodes
+
+In the future, more actions are planned, so only enable this parameter
+if the nodes are completely dedicated to Ganeti; otherwise it might be
+possible to lose data due to auto-maintenance actions.
+
+Removing a cluster entirely
++++++++++++++++++++++++++++
+
+The usual method to cleanup a cluster is to run ``gnt-cluster destroy``
+however if the Ganeti installation is broken in any way then this will
+not run.
+
+It is possible in such a case to cleanup manually most if not all traces
+of a cluster installation by following these steps on all of the nodes:
+
+1. Shutdown all instances. This depends on the virtualisation method
+ used (Xen, KVM, etc.):
+
+ - Xen: run ``xm list`` and ``xm destroy`` on all the non-Domain-0
+ instances
+ - KVM: kill all the KVM processes
+ - chroot: kill all processes under the chroot mountpoints
+
+2. If using DRBD, shutdown all DRBD minors (which should by at this time
+ no-longer in use by instances); on each node, run ``drbdsetup
+ /dev/drbdN down`` for each active DRBD minor.
+
+3. If using LVM, cleanup the Ganeti volume group; if only Ganeti created
+ logical volumes (and you are not sharing the volume group with the
+ OS, for example), then simply running ``lvremove -f xenvg`` (replace
+ 'xenvg' with your volume group name) should do the required cleanup.
+
+4. If using file-based storage, remove recursively all files and
+ directories under your file-storage directory: ``rm -rf
+ /srv/ganeti/file-storage/*`` replacing the path with the correct path
+ for your cluster.
+
+5. Stop the ganeti daemons (``/etc/init.d/ganeti stop``) and kill any
+ that remain alive (``pgrep ganeti`` and ``pkill ganeti``).
+
+6. Remove the ganeti state directory (``rm -rf /var/lib/ganeti/*``),
+ replacing the path with the correct path for your installation.
+
+On the master node, remove the cluster from the master-netdev (usually
+``xen-br0`` for bridged mode, otherwise ``eth0`` or similar), by running
+``ip a del $clusterip/32 dev xen-br0`` (use the correct cluster ip and
+network device name).
+
+At this point, the machines are ready for a cluster creation; in case
+you want to remove Ganeti completely, you need to also undo some of the
+SSH changes and log directories:
+
+- ``rm -rf /var/log/ganeti /srv/ganeti`` (replace with the correct
+ paths)
+- remove from ``/root/.ssh`` the keys that Ganeti added (check the
+ ``authorized_keys`` and ``id_dsa`` files)
+- regenerate the host's SSH keys (check the OpenSSH startup scripts)
+- uninstall Ganeti
+
+Otherwise, if you plan to re-create the cluster, you can just go ahead
+and rerun ``gnt-cluster init``.
+
+Tags handling
+-------------
+
+The tags handling (addition, removal, listing) is similar for all the
+objects that support it (instances, nodes, and the cluster).
+
+Limitations
++++++++++++
+
+Note that the set of characters present in a tag and the maximum tag
+length are restricted. Currently the maximum length is 128 characters,
+there can be at most 4096 tags per object, and the set of characters is
+comprised by alphanumeric characters and additionally ``.+*/:@-``.
+
+Operations
+++++++++++
+
+Tags can be added via ``add-tags``::
+
+ gnt-instance add-tags INSTANCE a b c
+ gnt-node add-tags INSTANCE a b c
+ gnt-cluster add-tags a b c
+
+
+The above commands add three tags to an instance, to a node and to the
+cluster. Note that the cluster command only takes tags as arguments,
+whereas the node and instance commands first required the node and
+instance name.
+
+Tags can also be added from a file, via the ``--from=FILENAME``
+argument. The file is expected to contain one tag per line.
+
+Tags can also be remove via a syntax very similar to the add one::
+
+ gnt-instance remove-tags INSTANCE a b c
+
+And listed via::
+
+ gnt-instance list-tags
+ gnt-node list-tags
+ gnt-cluster list-tags
+
+Global tag search
++++++++++++++++++
+
+It is also possible to execute a global search on the all tags defined
+in the cluster configuration, via a cluster command::
+
+ gnt-cluster search-tags REGEXP
+
+The parameter expected is a regular expression (see
+:manpage:`regex(7)`). This will return all tags that match the search,
+together with the object they are defined in (the names being show in a
+hierarchical kind of way)::
+
+ node1# gnt-cluster search-tags o
+ /cluster foo
+ /instances/instance1 owner:bar
+
+
+Job operations
+--------------
+
+The various jobs submitted by the instance/node/cluster commands can be
+examined, canceled and archived by various invocations of the
+``gnt-job`` command.
+
+First is the job list command::
+
+ node1# gnt-job list
+ 17771 success INSTANCE_QUERY_DATA
+ 17773 success CLUSTER_VERIFY_DISKS
+ 17775 success CLUSTER_REPAIR_DISK_SIZES
+ 17776 error CLUSTER_RENAME(cluster.example.com)
+ 17780 success CLUSTER_REDIST_CONF
+ 17792 success INSTANCE_REBOOT(instance1.example.com)
+
+More detailed information about a job can be found via the ``info``
+command::
+
+ node1# gnt-job info 17776
+ Job ID: 17776
+ Status: error
+ Received: 2009-10-25 23:18:02.180569
+ Processing start: 2009-10-25 23:18:02.200335 (delta 0.019766s)
+ Processing end: 2009-10-25 23:18:02.279743 (delta 0.079408s)
+ Total processing time: 0.099174 seconds
+ Opcodes:
+ OP_CLUSTER_RENAME
+ Status: error
+ Processing start: 2009-10-25 23:18:02.200335
+ Processing end: 2009-10-25 23:18:02.252282
+ Input fields:
+ name: cluster.example.com
+ Result:
+ OpPrereqError
+ [Neither the name nor the IP address of the cluster has changed]
+ Execution log:
+
+During the execution of a job, it's possible to follow the output of a
+job, similar to the log that one get from the ``gnt-`` commands, via the
+watch command::
+
+ node1# gnt-instance add --submit … instance1
+ JobID: 17818
+ node1# gnt-job watch 17818
+ Output from job 17818 follows
+ -----------------------------
+ Mon Oct 26 00:22:48 2009 - INFO: Selected nodes for instance instance1 via iallocator dumb: node1, node2
+ Mon Oct 26 00:22:49 2009 * creating instance disks...
+ Mon Oct 26 00:22:52 2009 adding instance instance1 to cluster config
+ Mon Oct 26 00:22:52 2009 - INFO: Waiting for instance instance1 to sync disks.
+ …
+ Mon Oct 26 00:23:03 2009 creating os for instance instance1 on node node1
+ Mon Oct 26 00:23:03 2009 * running the instance OS create scripts...
+ Mon Oct 26 00:23:13 2009 * starting instance...
+ node1#
+
+This is useful if you need to follow a job's progress from multiple
+terminals.
+
+A job that has not yet started to run can be canceled::
+
+ node1# gnt-job cancel 17810
+
+But not one that has already started execution::
+
+ node1# gnt-job cancel 17805
+ Job 17805 is no longer waiting in the queue
+
+There are two queues for jobs: the *current* and the *archive*
+queue. Jobs are initially submitted to the current queue, and they stay
+in that queue until they have finished execution (either successfully or
+not). At that point, they can be moved into the archive queue, and the
+ganeti-watcher script will do this automatically after 6 hours. The
+ganeti-cleaner script will remove the jobs from the archive directory
+after three weeks.
+
+Note that only jobs in the current queue can be viewed via the list and
+info commands; Ganeti itself doesn't examine the archive directory. If
+you need to see an older job, either move the file manually in the
+top-level queue directory, or look at its contents (it's a
+JSON-formatted file).
+
+Special Ganeti deployments
+--------------------------
+
+Since Ganeti 2.4, it is possible to extend the Ganeti deployment with
+two custom scenarios: Ganeti inside Ganeti and multi-site model.
+
+Running Ganeti under Ganeti
++++++++++++++++++++++++++++
+
+It is sometimes useful to be able to use a Ganeti instance as a Ganeti
+node (part of another cluster, usually). One example scenario is two
+small clusters, where we want to have an additional master candidate
+that holds the cluster configuration and can be used for helping with
+the master voting process.
+
+However, these Ganeti instance should not host instances themselves, and
+should not be considered in the normal capacity planning, evacuation
+strategies, etc. In order to accomplish this, mark these nodes as
+non-``vm_capable``::
+
+ node1# gnt-node modify --vm-capable=no node3
+
+The vm_capable status can be listed as usual via ``gnt-node list``::
+
+ node1# gnt-node list -oname,vm_capable
+ Node VMCapable
+ node1 Y
+ node2 Y
+ node3 N
+
+When this flag is set, the cluster will not do any operations that
+relate to instances on such nodes, e.g. hypervisor operations,
+disk-related operations, etc. Basically they will just keep the ssconf
+files, and if master candidates the full configuration.
+
+Multi-site model
+++++++++++++++++
+
+If Ganeti is deployed in multi-site model, with each site being a node
+group (so that instances are not relocated across the WAN by mistake),
+it is conceivable that either the WAN latency is high or that some sites
+have a lower reliability than others. In this case, it doesn't make
+sense to replicate the job information across all sites (or even outside
+of a “central” node group), so it should be possible to restrict which
+nodes can become master candidates via the auto-promotion algorithm.
+
+Ganeti 2.4 introduces for this purpose a new ``master_capable`` flag,
+which (when unset) prevents nodes from being marked as master
+candidates, either manually or automatically.
+
+As usual, the node modify operation can change this flag::
+
+ node1# gnt-node modify --auto-promote --master-capable=no node3
+ Fri Jan 7 06:23:07 2011 - INFO: Demoting from master candidate
+ Fri Jan 7 06:23:08 2011 - INFO: Promoted nodes to master candidate role: node4
+ Modified node node3
+ - master_capable -> False
+ - master_candidate -> False
+
+And the node list operation will list this flag::
+
+ node1# gnt-node list -oname,master_capable node1 node2 node3
+ Node MasterCapable
+ node1 Y
+ node2 Y
+ node3 N
+
+Note that marking a node both not ``vm_capable`` and not
+``master_capable`` makes the node practically unusable from Ganeti's
+point of view. Hence these two flags should be used probably in
+contrast: some nodes will be only master candidates (master_capable but
+not vm_capable), and other nodes will only hold instances (vm_capable
+but not master_capable).
+
+
+Ganeti tools
+------------
+
+Beside the usual ``gnt-`` and ``ganeti-`` commands which are provided
+and installed in ``$prefix/sbin`` at install time, there are a couple of
+other tools installed which are used seldom but can be helpful in some
+cases.
+
+lvmstrap
+++++++++
+
+The ``lvmstrap`` tool, introduced in :ref:`configure-lvm-label` section,
+has two modes of operation:
+
+- ``diskinfo`` shows the discovered disks on the system and their status
+- ``create`` takes all not-in-use disks and creates a volume group out
+ of them
+
+.. warning:: The ``create`` argument to this command causes data-loss!
+
+cfgupgrade
+++++++++++
+
+The ``cfgupgrade`` tools is used to upgrade between major (and minor)
+Ganeti versions. Point-releases are usually transparent for the admin.
+
+More information about the upgrade procedure is listed on the wiki at
+http://code.google.com/p/ganeti/wiki/UpgradeNotes.
+
+There is also a script designed to upgrade from Ganeti 1.2 to 2.0,
+called ``cfgupgrade12``.
+
+cfgshell
+++++++++
+
+.. note:: This command is not actively maintained; make sure you backup
+ your configuration before using it
+
+This can be used as an alternative to direct editing of the
+main configuration file if Ganeti has a bug and prevents you, for
+example, from removing an instance or a node from the configuration
+file.
+
+.. _burnin-label:
+
+burnin
+++++++
+
+.. warning:: This command will erase existing instances if given as
+ arguments!
+
+This tool is used to exercise either the hardware of machines or
+alternatively the Ganeti software. It is safe to run on an existing
+cluster **as long as you don't pass it existing instance names**.
+
+The command will, by default, execute a comprehensive set of operations
+against a list of instances, these being:
+
+- creation
+- disk replacement (for redundant instances)
+- failover and migration (for redundant instances)
+- move (for non-redundant instances)
+- disk growth
+- add disks, remove disk
+- add NICs, remove NICs
+- export and then import
+- rename
+- reboot
+- shutdown/startup
+- and finally removal of the test instances
+
+Executing all these operations will test that the hardware performs
+well: the creation, disk replace, disk add and disk growth will exercise
+the storage and network; the migrate command will test the memory of the
+systems. Depending on the passed options, it can also test that the
+instance OS definitions are executing properly the rename, import and
+export operations.
+
+sanitize-config
++++++++++++++++
+
+This tool takes the Ganeti configuration and outputs a "sanitized"
+version, by randomizing or clearing:
+
+- DRBD secrets and cluster public key (always)
+- host names (optional)
+- IPs (optional)
+- OS names (optional)
+- LV names (optional, only useful for very old clusters which still have
+ instances whose LVs are based on the instance name)
+
+By default, all optional items are activated except the LV name
+randomization. When passing ``--no-randomization``, which disables the
+optional items (i.e. just the DRBD secrets and cluster public keys are
+randomized), the resulting file can be used as a safety copy of the
+cluster config - while not trivial, the layout of the cluster can be
+recreated from it and if the instance disks have not been lost it
+permits recovery from the loss of all master candidates.
+
+move-instance
++++++++++++++
+
+See :doc:`separate documentation for move-instance <move-instance>`.
+
+.. TODO: document cluster-merge tool
+
+
+Other Ganeti projects
+---------------------