Documentation updates

Our admin guide was very very trivial. This patch updates it to contain
advice on when to use which commands, removes the instance
administration part from the installation guide (moved to the admin
guide), and adds a walkthrough document that should be useable as a
starting point for new admins.

The patch also adds emacs variables to the documents, and rewraps some
which were not already at 72 chars.

The doc updates also show backwards-compatible commands for Ganeti 2.0,
as we don't have a good up-to-date 2.0 document and people might refer
to this set of documentation even when running that.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

diff --git a/Makefile.am b/Makefile.am
index dd6d0bd..1ceaa41 100644 (file)
--- a/Makefile.am
+++ b/Makefile.am
@@ -148,9 +148,10 @@ docrst = \
        doc/index.rst \
        doc/install.rst \
        doc/locking.rst \
        doc/index.rst \
        doc/install.rst \
        doc/locking.rst \

+       doc/news.rst \

        doc/rapi.rst \
        doc/security.rst \
        doc/rapi.rst \
        doc/security.rst \

-       doc/news.rst
+       doc/walkthrough.rst

 
 doc/html/.stamp: $(docrst) $(docpng) doc/conf.py configure.ac
        @test -n "$(SPHINX)" || \
 
 doc/html/.stamp: $(docrst) $(docpng) doc/conf.py configure.ac
        @test -n "$(SPHINX)" || \

diff --git a/doc/admin.rst b/doc/admin.rst
index 3d0003d..61f572b 100644 (file)
--- a/doc/admin.rst
+++ b/doc/admin.rst
@@ -5,83 +5,222 @@ Documents Ganeti version |version|
 
 .. contents::
 
 
 .. contents::
 

+.. highlight:: text
+

 Introduction
 ------------
 
 Introduction
 ------------
 

-Ganeti is a virtualization cluster management software. You are
-expected to be a system administrator familiar with your Linux
-distribution and the Xen or KVM virtualization environments before
-using it.
-
+Ganeti is a virtualization cluster management software. You are expected
+to be a system administrator familiar with your Linux distribution and
+the Xen or KVM virtualization environments before using it.

 
 The various components of Ganeti all have man pages and interactive
 
 The various components of Ganeti all have man pages and interactive

-help. This manual though will help you getting familiar with the
-system by explaining the most common operations, grouped by related
-use.
+help. This manual though will help you getting familiar with the system
+by explaining the most common operations, grouped by related use.

 
 After a terminology glossary and a section on the prerequisites needed
 
 After a terminology glossary and a section on the prerequisites needed

-to use this manual, the rest of this document is divided in three main
-sections, which group different features of Ganeti:
+to use this manual, the rest of this document is divided in sections
+for the different targets that a command affects: instance, nodes, etc.

 
 

-- Instance Management
-- High Availability Features
-- Debugging Features
+.. _terminology-label:

 
 Ganeti terminology
 
 Ganeti terminology

-~~~~~~~~~~~~~~~~~~
+++++++++++++++++++

 
 

-This section provides a small introduction to Ganeti terminology,
-which might be useful to read the rest of the document.
+This section provides a small introduction to Ganeti terminology, which
+might be useful when reading the rest of the document.

 
 Cluster
 
 Cluster

-  A set of machines (nodes) that cooperate to offer a coherent
-  highly available virtualization service.
+~~~~~~~

 
 

-Node
-  A physical machine which is member of a cluster.
-  Nodes are the basic cluster infrastructure, and are
-  not fault tolerant.
+A set of machines (nodes) that cooperate to offer a coherent, highly
+available virtualization service under a single administration domain.

 
 

-Master node
-  The node which controls the Cluster, from which all
-  Ganeti commands must be given.
+Node
+~~~~
+
+A physical machine which is member of a cluster.  Nodes are the basic
+cluster infrastructure, and they don't need to be fault tolerant in
+order to achieve high availability for instances.
+
+Node can be added and removed (if they host no instances) at will from
+the cluster. In a HA cluster and only with HA instances, the loss of any
+single node will not cause disk data loss for any instance; of course,
+a node crash will cause the crash of the its primary instances.
+
+A node belonging to a cluster can be in one of the following roles at a
+given time:
+
+- *master* node, which is the node from which the cluster is controlled
+- *master candidate* node, only nodes in this role have the full cluster
+  configuration and knowledge, and only master candidates can become the
+  master node
+- *regular* node, which is the state in which most nodes will be on
+  bigger clusters (>20 nodes)
+- *drained* node, nodes in this state are functioning normally but the
+  cannot receive new instances; the intention is that nodes in this role
+  have some issue and they are being evacuated for hardware repairs
+- *offline* node, in which there is a record in the cluster
+  configuration about the node, but the daemons on the master node will
+  not talk to this node; any instances declared as having an offline
+  node as either primary or secondary will be flagged as an error in the
+  cluster verify operation
+
+Depending on the role, each node will run a set of daemons:
+
+- the :command:`ganeti-noded` daemon, which control the manipulation of
+  this node's hardware resources; it runs on all nodes which are in a
+  cluster
+- the :command:`ganeti-confd` daemon (Ganeti 2.1+) which runs on all
+  nodes, but is only functional on master candidate nodes
+- the :command:`ganeti-rapi` daemon which runs on the master node and
+  offers an HTTP-based API for the cluster
+- the :command:`ganeti-masterd` daemon which runs on the master node and
+  allows control of the cluster

 
 Instance
 
 Instance

-  A virtual machine which runs on a cluster. It can be a
-  fault tolerant highly available entity.
+~~~~~~~~

 
 

-Pool
-  A pool is a set of clusters sharing the same network.
+A virtual machine which runs on a cluster. It can be a fault tolerant,
+highly available entity.

 
 

-Meta-Cluster
-  Anything that concerns more than one cluster.
+An instance has various parameters, which are classified in three
+categories: hypervisor related-parameters (called ``hvparams``), general
+parameters (called ``beparams``) and per network-card parameters (called
+``nicparams``). All these parameters can be modified either at instance
+level or via defaults at cluster level.

 
 

-Prerequisites
+Disk template

 ~~~~~~~~~~~~~
 
 ~~~~~~~~~~~~~
 

-You need to have your Ganeti cluster installed and configured before
-you try any of the commands in this document. Please follow the
-*Ganeti installation tutorial* for instructions on how to do that.
+The are multiple options for the storage provided to an instance; while
+the instance sees the same virtual drive in all cases, the node-level
+configuration varies between them.

 
 

-Managing Instances
-------------------
+There are four disk templates you can choose from:

 
 

-Adding/Removing an instance
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
+diskless
+  The instance has no disks. Only used for special purpose operating
+  systems or for testing.
+
+file
+  The instance will use plain files as backend for its disks. No
+  redundancy is provided, and this is somewhat more difficult to
+  configure for high performance.
+
+plain
+  The instance will use LVM devices as backend for its disks. No
+  redundancy is provided.
+
+drbd
+  .. note:: This is only valid for multi-node clusters using DRBD 8.0+
+
+  A mirror is set between the local node and a remote one, which must be
+  specified with the second value of the --node option. Use this option
+  to obtain a highly available instance that can be failed over to a
+  remote node should the primary one fail.
+
+IAllocator
+~~~~~~~~~~
+
+A framework for using external (user-provided) scripts to compute the
+placement of instances on the cluster nodes. This eliminates the need to
+manually specify nodes in instance add, instance moves, node evacuate,
+etc.
+
+In order for Ganeti to be able to use these scripts, they must be place
+in the iallocator directory (usually ``lib/ganeti/iallocators`` under
+the installation prefix, e.g. ``/usr/local``).
+
+“Primary” and “secondary” concepts
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+An instance has a primary and depending on the disk configuration, might
+also have a secondary node. The instance always runs on the primary node
+and only uses its secondary node for disk replication.
+
+Similarly, the term of primary and secondary instances when talking
+about a node refers to the set of instances having the given node as
+primary, respectively secondary.
+
+Tags
+~~~~
+
+Tags are short strings that can be attached to either to cluster itself,
+or to nodes or instances. They are useful as a very simplistic
+information store for helping with cluster administration, for example
+by attaching owner information to each instance after it's created::

 
 

-Adding a new virtual instance to your Ganeti cluster is really easy.
-The command is::
+  gnt-instance add … instance1
+  gnt-instance add-tags instance1 owner:user2
+
+And then by listing each instance and its tags, this information could
+be used for contacting the users of each instance.
+
+Jobs and OpCodes
+~~~~~~~~~~~~~~~~
+
+While not directly visible by an end-user, it's useful to know that a
+basic cluster operation (e.g. starting an instance) is represented
+internall by Ganeti as an *OpCode* (abbreviation from operation
+code). These OpCodes are executed as part of a *Job*. The OpCodes in a
+single Job are processed serially by Ganeti, but different Jobs will be
+processed (depending on resource availability) in parallel.
+
+For example, shutting down the entire cluster can be done by running the
+command ``gnt-instance shutdown --all``, which will submit for each
+instance a separate job containing the “shutdown instance” OpCode.
+
+
+Prerequisites
++++++++++++++
+
+You need to have your Ganeti cluster installed and configured before you
+try any of the commands in this document. Please follow the
+:doc:`install` for instructions on how to do that.
+
+Instance management
+-------------------
+
+Adding an instance
+++++++++++++++++++
+
+The add operation might seem complex due to the many parameters it
+accepts, but once you have understood the (few) required parameters and
+the customisation capabilities you will see it is an easy operation.
+
+The add operation requires at minimum five parameters:
+
+- the OS for the instance
+- the disk template
+- the disk count and size
+- the node specification or alternatively the iallocator to use
+- and finally the instance name
+
+The OS for the instance must be visible in the output of the command
+``gnt-os list`` and specifies which guest OS to install on the instance.
+
+The disk template specifies what kind of storage to use as backend for
+the (virtual) disks presented to the instance; note that for instances
+with multiple virtual disks, they all must be of the same type.
+
+The node(s) on which the instance will run can be given either manually,
+via the ``-n`` option, or computed automatically by Ganeti, if you have
+installed any iallocator script.
+
+With the above parameters in mind, the command is::

 
   gnt-instance add \
 
   gnt-instance add \

-    -n TARGET_NODE:SECONDARY_NODE -o OS_TYPE -t DISK_TEMPLATE \
+    -n TARGET_NODE:SECONDARY_NODE \
+    -o OS_TYPE \
+    -t DISK_TEMPLATE -s DISK_SIZE \

     INSTANCE_NAME
 
 The instance name must be resolvable (e.g. exist in DNS) and usually
     INSTANCE_NAME
 
 The instance name must be resolvable (e.g. exist in DNS) and usually

-to an address in the same subnet as the cluster itself. Options you
-can give to this command include:
+points to an address in the same subnet as the cluster itself.

 
 

-- The disk size (``-s``) for a single-disk instance, or multiple
-  ``--disk N:size=SIZE`` options for multi-instance disks
+The above command has the minimum required options; other options you
+can give include, among others:

 
 - The memory size (``-B memory``)
 
 
 - The memory size (``-B memory``)
 

@@ -91,49 +230,33 @@ can give to this command include:
   instance is created. The IP and/or bridge of the NIC can be changed
   via ``--nic 0:ip=IP,bridge=BRIDGE``
 
   instance is created. The IP and/or bridge of the NIC can be changed
   via ``--nic 0:ip=IP,bridge=BRIDGE``
 

+See the manpage for gnt-instance for the detailed option list.

 
 

-There are four types of disk template you can choose from:
-
-diskless
-  The instance has no disks. Only used for special purpouse operating
-  systems or for testing.
-
-file
-  The instance will use plain files as backend for its disks. No
-  redundancy is provided, and this is somewhat more difficult to
-  configure for high performance.
-
-plain
-  The instance will use LVM devices as backend for its disks. No
-  redundancy is provided.
-
-drbd
-  .. note:: This is only valid for multi-node clusters using DRBD 8.0.x
-
-  A mirror is set between the local node and a remote one, which must
-  be specified with the second value of the --node option. Use this
-  option to obtain a highly available instance that can be failed over
-  to a remote node should the primary one fail.
+For example if you want to create an highly available instance, with a
+single disk of 50GB and the default memory size, having primary node
+``node1`` and secondary node ``node3``, use the following command::

 
 

-For example if you want to create an highly available instance use the
-drbd disk templates::
+  gnt-instance add -n node1:node3 -o debootstrap -t drbd \
+    instance1

 
 

-  gnt-instance add -n TARGET_NODE:SECONDARY_NODE -o OS_TYPE -t drbd \
-    INSTANCE_NAME
+There is a also a command for batch instance creation from a
+specification file, see the ``batch-create`` operation in the
+gnt-instance manual page.

 
 

-To know which operating systems your cluster supports you can use
-the command::
+Regular instance operations
++++++++++++++++++++++++++++

 
 

-  gnt-os list
+Removal
+~~~~~~~

 
 

-Removing an instance is even easier than creating one. This operation
-is irrereversible and destroys all the contents of your instance. Use
-with care::
+Removing an instance is even easier than creating one. This operation is
+irreversible and destroys all the contents of your instance. Use with
+care::

 
   gnt-instance remove INSTANCE_NAME
 
 
   gnt-instance remove INSTANCE_NAME
 

-Starting/Stopping an instance
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Startup/shutdown
+~~~~~~~~~~~~~~~~

 
 Instances are automatically started at instance creation time. To
 manually start one which is currently stopped you can run::
 
 Instances are automatically started at instance creation time. To
 manually start one which is currently stopped you can run::

@@ -144,31 +267,58 @@ While the command to stop one is::
 
   gnt-instance shutdown INSTANCE_NAME
 
 
   gnt-instance shutdown INSTANCE_NAME
 

+.. warning:: Do not use the Xen or KVM commands directly to stop
+   instances. If you run for example ``xm shutdown`` or ``xm destroy``
+   on an instance Ganeti will automatically restart it (via the
+   :command:`ganeti-watcher` command which is launched via cron).
+
+Querying instances
+~~~~~~~~~~~~~~~~~~
+
+There are two ways to get information about instances: listing
+instances, which does a tabular output containing a given set of fields
+about each instance, and querying detailed information about a set of
+instances.
+

 The command to see all the instances configured and their status is::
 
   gnt-instance list
 
 The command to see all the instances configured and their status is::
 
   gnt-instance list
 

-Do not use the Xen commands to stop instances. If you run for example
-xm shutdown or xm destroy on an instance Ganeti will automatically
-restart it (via the ``ganeti-watcher``).
+The command can return a custom set of information when using the ``-o``
+option (as always, check the manpage for a detailed specification). Each
+instance will be represented on a line, thus making it easy to parse
+this output via the usual shell utilities (grep, sed, etc.).
+
+To get more detailed information about an instance, you can run::
+
+  gnt-instance info INSTANCE
+
+which will give a multi-line block of information about the instance,
+it's hardware resources (especially its disks and their redundancy
+status), etc. This is harder to parse and is more expensive than the
+list operation, but returns much more detailed information.

 
 

-Exporting/Importing an instance
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 
 

-You can create a snapshot of an instance disk and Ganeti
+Export/Import
++++++++++++++
+
+You can create a snapshot of an instance disk and its Ganeti

 configuration, which then you can backup, or import into another
 cluster. The way to export an instance is::
 
   gnt-backup export -n TARGET_NODE INSTANCE_NAME
 
 configuration, which then you can backup, or import into another
 cluster. The way to export an instance is::
 
   gnt-backup export -n TARGET_NODE INSTANCE_NAME
 

+

 The target node can be any node in the cluster with enough space under
 The target node can be any node in the cluster with enough space under

-``/srv/ganeti`` to hold the instance image. Use the *--noshutdown*
-option to snapshot an instance without rebooting it. Any previous
-snapshot of the same instance existing cluster-wide under
-``/srv/ganeti`` will be removed by this operation: if you want to keep
-them move them out of the Ganeti exports directory.
+``/srv/ganeti`` to hold the instance image. Use the ``--noshutdown``
+option to snapshot an instance without rebooting it. Note that Ganeti
+only keeps one snapshot for an instance - any previous snapshot of the
+same instance existing cluster-wide under ``/srv/ganeti`` will be
+removed by this operation: if you want to keep them, you need to move
+them out of the Ganeti exports directory.

 
 

-Importing an instance is similar to creating a new one. The command is::
+Importing an instance is similar to creating a new one, but additionally
+one must specify the location of the snapshot. The command is::

 
   gnt-backup import -n TARGET_NODE -t DISK_TEMPLATE \
     --src-node=NODE --src-dir=DIR INSTANCE_NAME
 
   gnt-backup import -n TARGET_NODE -t DISK_TEMPLATE \
     --src-node=NODE --src-dir=DIR INSTANCE_NAME

@@ -176,11 +326,26 @@ Importing an instance is similar to creating a new one. The command is::
 Most of the options available for the command :command:`gnt-instance
 add` are supported here too.
 
 Most of the options available for the command :command:`gnt-instance
 add` are supported here too.
 

-High availability features
---------------------------
+Instance HA features
+--------------------

 
 .. note:: This section only applies to multi-node clusters
 
 
 .. note:: This section only applies to multi-node clusters
 

+.. _instance-change-primary-label:
+
+Changing the primary node
++++++++++++++++++++++++++
+
+There are three ways to exchange an instance's primary and secondary
+nodes; the right one to choose depends on how the instance has been
+created and the status of its current primary node. See
+:ref:`rest-redundancy-label` for information on changing the secondary
+node. Note that it's only possible to change the primary node to the
+secondary and vice-versa; a direct change of the primary node with a
+third node, while keeping the current secondary is not possible in a
+single step, only via multiple operations as detailed in
+:ref:`instance-relocation-label`.
+

 Failing over an instance
 ~~~~~~~~~~~~~~~~~~~~~~~~
 
 Failing over an instance
 ~~~~~~~~~~~~~~~~~~~~~~~~
 

@@ -192,74 +357,182 @@ node you can just run::
   gnt-instance failover INSTANCE_NAME
 
 That's it. After the command completes the secondary node is now the
   gnt-instance failover INSTANCE_NAME
 
 That's it. After the command completes the secondary node is now the

-primary, and vice versa.
+primary, and vice-versa.

 
 Live migrating an instance
 ~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 
 Live migrating an instance
 ~~~~~~~~~~~~~~~~~~~~~~~~~~
 

-If an instance is built in highly available mode, it currently runs
-and both its nodes are running fine, you can at migrate it over to its
-secondary node, without dowtime. On the master node you need to run::
+If an instance is built in highly available mode, it currently runs and
+both its nodes are running fine, you can at migrate it over to its
+secondary node, without downtime. On the master node you need to run::

 
   gnt-instance migrate INSTANCE_NAME
 
 
   gnt-instance migrate INSTANCE_NAME
 

-Replacing an instance disks
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The current load on the instance and its memory size will influence how
+long the migration will take. In any case, for both KVM and Xen
+hypervisors, the migration will be transparent to the instance.

 
 

-So what if instead the secondary node for an instance has failed, or
-you plan to remove a node from your cluster, and you failed over all
-its instances, but it's still secondary for some? The solution here is
-to replace the instance disks, changing the secondary node::
+Moving an instance (offline)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 
 

-  gnt-instance replace-disks -n NODE INSTANCE_NAME
+If an instance has not been create as mirrored, then the only way to
+change its primary node is to execute the move command::

 
 

-This process is a bit long, but involves no instance downtime, and at
-the end of it the instance has changed its secondary node, to which it
-can if necessary be failed over.
+  gnt-instance move -n NEW_NODE INSTANCE

 
 

-Failing over the master node
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+This has a few prerequisites:

 
 

-This is all good as long as the Ganeti Master Node is up. Should it go
-down, or should you wish to decommission it, just run on any other
-node the command::
+- the instance must be stopped
+- its current primary node must be on-line and healthy
+- the disks of the instance must not have any errors

 
 

-  gnt-cluster masterfailover
+Since this operation actually copies the data from the old node to the
+new node, expect it to take proportional to the size of the instance's
+disks and the speed of both the nodes' I/O system and their networking.

 
 

-and the node you ran it on is now the new master.
+Disk operations
++++++++++++++++

 
 

-Adding/Removing nodes
-~~~~~~~~~~~~~~~~~~~~~
+Disk failures are a common cause of errors in any server
+deployment. Ganeti offers protection from single-node failure if your
+instances were created in HA mode, and it also offers ways to restore
+redundancy after a failure.

 
 

-And of course, now that you know how to move instances around, it's
-easy to free up a node, and then you can remove it from the cluster::
+Preparing for disk operations
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 
 

-  gnt-node remove NODE_NAME
+It is important to note that for Ganeti to be able to do any disk
+operation, the Linux machines on top of which Ganeti must be consistent;
+for LVM, this means that the LVM commands must not return failures; it
+is common that after a complete disk failure, any LVM command aborts
+with an error similar to::

 
 

-and maybe add a new one::
+  # vgs
+  /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
+  /dev/sdb1: read failed after 0 of 4096 at 750153695232: Input/output
+  error
+  /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
+  Couldn't find device with uuid
+  't30jmN-4Rcf-Fr5e-CURS-pawt-z0jU-m1TgeJ'.
+  Couldn't find all physical volumes for volume group xenvg.

 
 

-  gnt-node add --secondary-ip=ADDRESS NODE_NAME
+Before restoring an instance's disks to healthy status, it's needed to
+fix the volume group used by Ganeti so that we can actually create and
+manage the logical volumes. This is usually done in a multi-step
+process:

 
 

-Debugging Features
-------------------
+#. first, if the disk is completely gone and LVM commands exit with
+   “Couldn't find device with uuid…” then you need to run the command::
+
+    vgreduce --removemissing VOLUME_GROUP
+
+#. after the above command, the LVM commands should be executing
+   normally (warnings are normal, but the commands will not fail
+   completely).
+
+#. if the failed disk is still visible in the output of the ``pvs``
+   command, you need to deactivate it from allocations by running::
+
+    pvs -x n /dev/DISK

 
 

-At some point you might need to do some debugging operations on your
-cluster or on your instances. This section will help you with the most
-used debugging functionalities.
+At this point, the volume group should be consistent and any bad
+physical volumes should not longer be available for allocation.
+
+Note that since version 2.1 Ganeti provides some commands to automate
+these two operations, see :ref:`storage-units-label`.
+
+.. _rest-redundancy-label:
+
+Restoring redundancy for DRBD-based instances
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A DRBD instance has two nodes, and the storage on one of them has
+failed. Depending on which node (primary or secondary) has failed, you
+have three options at hand:
+
+- if the storage on the primary node has failed, you need to re-create
+  the disks on it
+- if the storage on the secondary node has failed, you can either
+  re-create the disks on it or change the secondary and recreate
+  redundancy on the new secondary node
+
+Of course, at any point it's possible to force re-creation of disks even
+though everything is already fine.
+
+For all three cases, the ``replace-disks`` operation can be used::
+
+  # re-create disks on the primary node
+  gnt-instance replace-disks -p INSTANCE_NAME
+  # re-create disks on the current secondary
+  gnt-instance replace-disks -s INSTANCE_NAME
+  # change the secondary node, via manual specification
+  gnt-instance replace-disks -n NODE INSTANCE_NAME
+  # change the secondary node, via an iallocator script
+  gnt-instance replace-disks -I SCRIPT INSTANCE_NAME
+  # since Ganeti 2.1: automatically fix the primary or secondary node
+  gnt-instance replace-disks -a INSTANCE_NAME
+
+Since the process involves copying all data from the working node to the
+target node, it will take a while, depending on the instance's disk
+size, node I/O system and network speed. But it is (baring any network
+interruption) completely transparent for the instance.
+
+Re-creating disks for non-redundant instances
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. versionadded:: 2.1
+
+For non-redundant instances, there isn't a copy (except backups) to
+re-create the disks. But it's possible to at-least re-create empty
+disks, after which a reinstall can be run, via the ``recreate-disks``
+command::
+
+  gnt-instance recreate-disks INSTANCE
+
+Note that this will fail if the disks already exists.
+
+Debugging instances
++++++++++++++++++++

 
 Accessing an instance's disks
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 
 Accessing an instance's disks
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

-From an instance's primary node you have access to its disks. Never
+From an instance's primary node you can have access to its disks. Never

 ever mount the underlying logical volume manually on a fault tolerant
 ever mount the underlying logical volume manually on a fault tolerant

-instance, or you risk breaking replication. The correct way to access
-them is to run the command::
+instance, or will break replication and your data will be
+inconsistent. The correct way to access an instance's disks is to run
+(on the master node, as usual) the command::
+
+  gnt-instance activate-disks INSTANCE
+
+And then, *on the primary node of the instance*, access the device that
+gets created. For example, you could mount the given disks, then edit
+files on the filesystem, etc.
+
+Note that with partitioned disks (as opposed to whole-disk filesystems),
+you will need to use a tool like :manpage:`kpartx(8)`::

 
 

-  gnt-instance activate-disks INSTANCE_NAME
+  node1# gnt-instance activate-disks instance1
+  …
+  node1# ssh node3
+  node3# kpartx -l /dev/…
+  node3# kpartx -a /dev/…
+  node3# mount /dev/mapper/… /mnt/
+  # edit files under mnt as desired
+  node3# umount /mnt/
+  node3# kpartx -d /dev/…
+  node3# exit
+  node1#

 
 

-And then access the device that gets created.  After you've finished
-you can deactivate them with the deactivate-disks command, which works
-in the same way.
+After you've finished you can deactivate them with the deactivate-disks
+command, which works in the same way::
+
+  gnt-instance deactivate-disks INSTANCE
+
+Note that if any process started by you is still using the disks, the
+above command will error out, and you **must** cleanup and ensure that
+the above command runs successfully before you start the instance,
+otherwise the instance will suffer corruption.

 
 Accessing an instance's console
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 Accessing an instance's console
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

@@ -268,65 +541,506 @@ The command to access a running instance's console is::
 
   gnt-instance console INSTANCE_NAME
 
 
   gnt-instance console INSTANCE_NAME
 

-Use the console normally and then type ``^]`` when
-done, to exit.
+Use the console normally and then type ``^]`` when done, to exit.
+
+Other instance operations
++++++++++++++++++++++++++
+
+Reboot
+~~~~~~

 
 

-Instance OS definitions Debugging
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+There is a wrapper command for rebooting instances::

 
 

-Should you have any problems with operating systems support the
-command to ran to see a complete status for all your nodes is::
+  gnt-instance reboot instance2
+
+By default, this does the equivalent of shutting down and then starting
+the instance, but it accepts parameters to perform a soft-reboot (via
+the hypervisor), a hard reboot (hypervisor shutdown and then startup) or
+a full one (the default, which also de-configures and then configures
+again the disks of the instance).
+
+Instance OS definitions debugging
++++++++++++++++++++++++++++++++++
+
+Should you have any problems with instance operating systems the command
+to see a complete status for all your nodes is::

 
    gnt-os diagnose
 
 
    gnt-os diagnose
 

-Cluster-wide debugging
-~~~~~~~~~~~~~~~~~~~~~~
+.. _instance-relocation-label:

 
 

-The :command:`gnt-cluster` command offers several options to run tests
-or execute cluster-wide operations. For example::
+Instance relocation
+~~~~~~~~~~~~~~~~~~~

 
 

-  gnt-cluster command
-  gnt-cluster copyfile
-  gnt-cluster verify
-  gnt-cluster verify-disks
-  gnt-cluster getmaster
-  gnt-cluster version
+While it is not possible to move an instance from nodes ``(A, B)`` to
+nodes ``(C, D)`` in a single move, it is possible to do so in a few
+steps::

 
 

-See the man page :manpage:`gnt-cluster` to know more about their usage.
+  # instance is located on A, B
+  node1# gnt-instance replace -n nodeC instance1
+  # instance has moved from (A, B) to (A, C)
+  # we now flip the primary/secondary nodes
+  node1# gnt-instance migrate instance1
+  # instance lives on (C, A)
+  # we can then change A to D via:
+  node1# gnt-instance replace -n nodeD instance1

 
 

-Removing a cluster entirely
+Which brings it into the final configuration of ``(C, D)``. Note that we
+needed to do two replace-disks operation (two copies of the instance
+disks), because we needed to get rid of both the original nodes (A and
+B).
+
+Node operations
+---------------
+
+There are much fewer node operations available than for instances, but
+they are equivalently important for maintaining a healthy cluster.
+
+Add/readd
++++++++++
+
+It is at any time possible to extend the cluster with one more node, by
+using the node add operation::
+
+  gnt-node add NEW_NODE
+
+If the cluster has a replication network defined, then you need to pass
+the ``-s REPLICATION_IP`` parameter to this option.
+
+A variation of this command can be used to re-configure a node if its
+Ganeti configuration is broken, for example if it has been reinstalled
+by mistake::
+
+  gnt-node add --readd EXISTING_NODE
+
+This will reinitialise the node as if it's been newly added, but while
+keeping its existing configuration in the cluster (primary/secondary IP,
+etc.), in other words you won't need to use ``-s`` here.
+
+Changing the node role
+++++++++++++++++++++++
+
+A node can be in different roles, as explained in the
+:ref:`terminology-label` section. Promoting a node to the master role is
+special, while the other roles are handled all via a single command.
+
+Failing over the master node
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you want to promote a different node to the master role (for whatever
+reason), run on any other master-candidate node the command::
+
+  gnt-cluster masterfailover
+
+and the node you ran it on is now the new master. In case you try to run
+this on a non master-candidate node, you will get an error telling you
+which nodes are valid.
+
+Changing between the other roles
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``gnt-node modify`` command can be used to select a new role::
+
+  # change to master candidate
+  gnt-node modify -C yes NODE
+  # change to drained status
+  gnt-node modify -D yes NODE
+  # change to offline status
+  gnt-node modify -O yes NODE
+  # change to regular mode (reset all flags)
+  gnt-node modify -O no -D no -C no NODE
+
+Note that the cluster requires that at any point in time, a certain
+number of nodes are master candidates, so changing from master candidate
+to other roles might fail. It is recommended to either force the
+operation (via the ``--force`` option) or first change the number of
+master candidates in the cluster - see :ref:`cluster-config-label`.
+
+Evacuating nodes
+++++++++++++++++
+
+There are two steps of moving instances off a node:
+
+- moving the primary instances (actually converting them into secondary
+  instances)
+- moving the secondary instances (including any instances converted in
+  the step above)
+
+Primary instance conversion
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For this step, you can use either individual instance move
+commands (as seen in :ref:`instance-change-primary-label`) or the bulk
+per-node versions; these are::
+
+  gnt-node migrate NODE
+  gnt-node evacuate NODE
+
+Note that the instance “move” command doesn't currently have a node
+equivalent.
+
+Both these commands, or the equivalent per-instance command, will make
+this node the secondary node for the respective instances, whereas their
+current secondary node will become primary. Note that it is not possible
+to change in one step the primary node to another node as primary, while
+keeping the same secondary node.
+
+Secondary instance evacuation
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For the evacuation of secondary instances, a command called
+:command:`gnt-node evacuate` is provided and its syntax is::
+
+  gnt-node evacuate -I IALLOCATOR_SCRIPT NODE
+  gnt-node evacuate -n DESTINATION_NODE NODE
+
+The first version will compute the new secondary for each instance in
+turn using the given iallocator script, whereas the second one will
+simply move all instances to DESTINATION_NODE.
+
+Removal
++++++++
+
+Once a node no longer has any instances (neither primary nor secondary),
+it's easy to remove it from the cluster::
+
+  gnt-node remove NODE_NAME
+
+This will deconfigure the node, stop the ganeti daemons on it and leave
+it hopefully like before it joined to the cluster.
+
+Storage handling
+++++++++++++++++
+
+When using LVM (either standalone or with DRBD), it can become tedious
+to debug and fix it in case of errors. Furthermore, even file-based
+storage can become complicated to handle manually on many hosts. Ganeti
+provides a couple of commands to help with automation.
+
+Logical volumes
+~~~~~~~~~~~~~~~
+
+This is a command specific to LVM handling. It allows listing the
+logical volumes on a given node or on all nodes and their association to
+instances via the ``volumes`` command::
+
+  node1# gnt-node volumes
+  Node  PhysDev   VG    Name             Size Instance
+  node1 /dev/sdb1 xenvg e61fbc97-….disk0 512M instance17
+  node1 /dev/sdb1 xenvg ebd1a7d1-….disk0 512M instance19
+  node2 /dev/sdb1 xenvg 0af08a3d-….disk0 512M instance20
+  node2 /dev/sdb1 xenvg cc012285-….disk0 512M instance16
+  node2 /dev/sdb1 xenvg f0fac192-….disk0 512M instance18
+
+The above command maps each logical volume to a volume group and
+underlying physical volume and (possibly) to an instance.
+
+.. _storage-units-label:
+
+Generalized storage handling
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. versionadded:: 2.1
+
+Starting with Ganeti 2.1, a new storage framework has been implemented
+that tries to abstract the handling of the storage type the cluster
+uses.
+
+First is listing the backend storage and their space situation::
+
+  node1# gnt-node list-storage
+  Node  Name        Size Used   Free
+  node1 /dev/sda7 673.8G   0M 673.8G
+  node1 /dev/sdb1 698.6G 1.5G 697.1G
+  node2 /dev/sda7 673.8G   0M 673.8G
+  node2 /dev/sdb1 698.6G 1.0G 697.6G
+
+The default is to list LVM physical volumes. It's also possible to list
+the LVM volume groups::
+
+  node1# gnt-node list-storage -t lvm-vg
+  Node  Name  Size
+  node1 xenvg 1.3T
+  node2 xenvg 1.3T
+
+Next is repairing storage units, which is currently only implemented for
+volume groups and does the equivalent of ``vgreduce --removemissing``::
+
+  node1# gnt-node repair-storage node2 lvm-vg xenvg
+  Sun Oct 25 22:21:45 2009 Repairing storage unit 'xenvg' on node2 ...
+
+Last is the modification of volume properties, which is (again) only
+implemented for LVM physical volumes and allows toggling the
+``allocatable`` value::
+
+  node1# gnt-node modify-storage --allocatable=no node2 lvm-pv /dev/sdb1
+
+Use of the storage commands

 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

-The usual method to cleanup a cluster is to run ``gnt-cluster
-destroy`` however if the Ganeti installation is broken in any way then
-this will not run.
+All these commands are needed when recovering a node from a disk
+failure:
+
+- first, we need to recover from complete LVM failure (due to missing
+  disk), by running the ``repair-storage`` command
+- second, we need to change allocation on any partially-broken disk
+  (i.e. LVM still sees it, but it has bad blocks) by running
+  ``modify-storage``
+- then we can evacuate the instances as needed

 
 

-It is possible in such a case to cleanup manually most if not all
-traces of a cluster installation by following these steps on all of
-the nodes:

 
 

-1. Shutdown all instances. This depends on the virtualisation
-   method used (Xen, KVM, etc.):
+Cluster operations
+------------------
+
+Beside the cluster initialisation command (which is detailed in the
+:doc:`install` document) and the master failover command which is
+explained under node handling, there are a couple of other cluster
+operations available.
+
+.. _cluster-config-label:
+
+Standard operations
++++++++++++++++++++
+
+One of the few commands that can be run on any node (not only the
+master) is the ``getmaster`` command::
+
+  node2# gnt-cluster getmaster
+  node1.example.com
+  node2#
+
+It is possible to query and change global cluster parameters via the
+``info`` and ``modify`` commands::
+
+  node1# gnt-cluster info
+  Cluster name: cluster.example.com
+  Cluster UUID: 07805e6f-f0af-4310-95f1-572862ee939c
+  Creation time: 2009-09-25 05:04:15
+  Modification time: 2009-10-18 22:11:47
+  Master node: node1.example.com
+  Architecture (this node): 64bit (x86_64)
+  …
+  Tags: foo
+  Default hypervisor: xen-pvm
+  Enabled hypervisors: xen-pvm
+  Hypervisor parameters:
+    - xen-pvm:
+        root_path: /dev/sda1
+        …
+  Cluster parameters:
+    - candidate pool size: 10
+      …
+  Default instance parameters:
+    - default:
+        memory: 128
+        …
+  Default nic parameters:
+    - default:
+        link: xen-br0
+        …
+
+There various parameters above can be changed via the ``modify``
+commands as follows:
+
+- the hypervisor parameters can be changed via ``modify -H
+  xen-pvm:root_path=…``, and so on for other hypervisors/key/values
+- the "default instance parameters" are changeable via ``modify -B
+  parameter=value…`` syntax
+- the cluster parameters are changeable via separate options to the
+  modify command (e.g. ``--candidate-pool-size``, etc.)
+
+For detailed option list see the :manpage:`gnt-cluster(8)` man page.
+
+The cluster version can be obtained via the ``version`` command::
+  node1# gnt-cluster version
+  Software version: 2.1.0
+  Internode protocol: 20
+  Configuration format: 2010000
+  OS api version: 15
+  Export interface: 0
+
+This is not very useful except when debugging Ganeti.
+
+Global node commands
+++++++++++++++++++++
+
+There are two commands provided for replicating files to all nodes of a
+cluster and for running commands on all the nodes::
+
+  node1# gnt-cluster copyfile /path/to/file
+  node1# gnt-cluster command ls -l /path/to/file
+
+These are simple wrappers over scp/ssh and more advanced usage can be
+obtained using :manpage:`dsh(1)` and similar commands. But they are
+useful to update an OS script from the master node, for example.
+
+Cluster verification
+++++++++++++++++++++
+
+There are three commands that relate to global cluster checks. The first
+one is ``verify`` which gives an overview on the cluster state,
+highlighting any issues. In normal operation, this command should return
+no ``ERROR`` messages::
+
+  node1# gnt-cluster verify
+  Sun Oct 25 23:08:58 2009 * Verifying global settings
+  Sun Oct 25 23:08:58 2009 * Gathering data (2 nodes)
+  Sun Oct 25 23:09:00 2009 * Verifying node status
+  Sun Oct 25 23:09:00 2009 * Verifying instance status
+  Sun Oct 25 23:09:00 2009 * Verifying orphan volumes
+  Sun Oct 25 23:09:00 2009 * Verifying remaining instances
+  Sun Oct 25 23:09:00 2009 * Verifying N+1 Memory redundancy
+  Sun Oct 25 23:09:00 2009 * Other Notes
+  Sun Oct 25 23:09:00 2009   - NOTICE: 5 non-redundant instance(s) found.
+  Sun Oct 25 23:09:00 2009 * Hooks Results
+
+The second command is ``verify-disks``, which checks that the instance's
+disks have the correct status based on the desired instance state
+(up/down)::
+
+  node1# gnt-cluster verify-disks
+
+Note that this command will show no output when disks are healthy.
+
+The last command is used to repair any discrepancies in Ganeti's
+recorded disk size and the actual disk size (disk size information is
+needed for proper activation and growth of DRBD-based disks)::
+
+  node1# gnt-cluster repair-disk-sizes
+  Sun Oct 25 23:13:16 2009  - INFO: Disk 0 of instance instance1 has mismatched size, correcting: recorded 512, actual 2048
+  Sun Oct 25 23:13:17 2009  - WARNING: Invalid result from node node4, ignoring node results
+
+The above shows one instance having wrong disk size, and a node which
+returned invalid data, and thus we ignored all primary instances of that
+node.
+
+Configuration redistribution
+++++++++++++++++++++++++++++
+
+If the verify command complains about file mismatches between the master
+and other nodes, due to some node problems or if you manually modified
+configuration files, you can force an push of the master configuration
+to all other nodes via the ``redist-conf`` command::
+
+  node1# gnt-cluster redist-conf
+  node1#
+
+This command will be silent unless there are problems sending updates to
+the other nodes.
+
+
+Cluster renaming
+++++++++++++++++
+
+It is possible to rename a cluster, or to change its IP address, via the
+``rename`` command. If only the IP has changed, you need to pass the
+current name and Ganeti will realise its IP has changed::
+
+  node1# gnt-cluster rename cluster.example.com
+  This will rename the cluster to 'cluster.example.com'. If
+  you are connected over the network to the cluster name, the operation
+  is very dangerous as the IP address will be removed from the node and
+  the change may not go through. Continue?
+  y/[n]/?: y
+  Failure: prerequisites not met for this operation:
+  Neither the name nor the IP address of the cluster has changed
+
+In the above output, neither value has changed since the cluster
+initialisation so the operation is not completed.
+
+Queue operations
+++++++++++++++++
+
+The job queue execution in Ganeti 2.0 and higher can be inspected,
+suspended and resumed via the ``queue`` command::
+
+  node1~# gnt-cluster queue info
+  The drain flag is unset
+  node1~# gnt-cluster queue drain
+  node1~# gnt-instance stop instance1
+  Failed to submit job for instance1: Job queue is drained, refusing job
+  node1~# gnt-cluster queue info
+  The drain flag is set
+  node1~# gnt-cluster queue undrain
+
+This is most useful if you have an active cluster and you need to
+upgrade the Ganeti software, or simply restart the software on any node:
+
+#. suspend the queue via ``queue drain``
+#. wait until there are no more running jobs via ``gnt-job list``
+#. restart the master or another node, or upgrade the software
+#. resume the queue via ``queue undrain``
+
+.. note:: this command only stores a local flag file, and if you
+   failover the master, it will not have effect on the new master.
+
+
+Watcher control
++++++++++++++++
+
+The :manpage:`ganeti-watcher` is a program, usually scheduled via
+``cron``, that takes care of cluster maintenance operations (restarting
+downed instances, activating down DRBD disks, etc.). However, during
+maintenance and troubleshooting, this can get in your way; disabling it
+via commenting out the cron job is not so good as this can be
+forgotten. Thus there are some commands for automated control of the
+watcher: ``pause``, ``info`` and ``continue``::
+
+  node1~# gnt-cluster watcher info
+  The watcher is not paused.
+  node1~# gnt-cluster watcher pause 1h
+  The watcher is paused until Mon Oct 26 00:30:37 2009.
+  node1~# gnt-cluster watcher info
+  The watcher is paused until Mon Oct 26 00:30:37 2009.
+  node1~# ganeti-watcher -d
+  2009-10-25 23:30:47,984:  pid=28867 ganeti-watcher:486 DEBUG Pause has been set, exiting
+  node1~# gnt-cluster watcher continue
+  The watcher is no longer paused.
+  node1~# ganeti-watcher -d
+  2009-10-25 23:31:04,789:  pid=28976 ganeti-watcher:345 DEBUG Archived 0 jobs, left 0
+  2009-10-25 23:31:05,884:  pid=28976 ganeti-watcher:280 DEBUG Got data from cluster, writing instance status file
+  2009-10-25 23:31:06,061:  pid=28976 ganeti-watcher:150 DEBUG Data didn't change, just touching status file
+  node1~# gnt-cluster watcher info
+  The watcher is not paused.
+  node1~#
+
+The exact details of the argument to the ``pause`` command are available
+in the manpage.
+
+.. note:: this command only stores a local flag file, and if you
+   failover the master, it will not have effect on the new master.
+
+Removing a cluster entirely
++++++++++++++++++++++++++++
+
+The usual method to cleanup a cluster is to run ``gnt-cluster destroy``
+however if the Ganeti installation is broken in any way then this will
+not run.
+
+It is possible in such a case to cleanup manually most if not all traces
+of a cluster installation by following these steps on all of the nodes:
+
+1. Shutdown all instances. This depends on the virtualisation method
+   used (Xen, KVM, etc.):

 
   - Xen: run ``xm list`` and ``xm destroy`` on all the non-Domain-0
     instances
   - KVM: kill all the KVM processes
   - chroot: kill all processes under the chroot mountpoints
 
 
   - Xen: run ``xm list`` and ``xm destroy`` on all the non-Domain-0
     instances
   - KVM: kill all the KVM processes
   - chroot: kill all processes under the chroot mountpoints
 

-2. If using DRBD, shutdown all DRBD minors (which should by at this
-   time no-longer in use by instances); on each node, run ``drbdsetup
+2. If using DRBD, shutdown all DRBD minors (which should by at this time
+   no-longer in use by instances); on each node, run ``drbdsetup

    /dev/drbdN down`` for each active DRBD minor.
 
    /dev/drbdN down`` for each active DRBD minor.
 

-3. If using LVM, cleanup the Ganeti volume group; if only Ganeti
-   created logical volumes (and you are not sharing the volume group
-   with the OS, for example), then simply running ``lvremove -f
-   xenvg`` (replace 'xenvg' with your volume group name) should do the
-   required cleanup.
+3. If using LVM, cleanup the Ganeti volume group; if only Ganeti created
+   logical volumes (and you are not sharing the volume group with the
+   OS, for example), then simply running ``lvremove -f xenvg`` (replace
+   'xenvg' with your volume group name) should do the required cleanup.

 
 4. If using file-based storage, remove recursively all files and
    directories under your file-storage directory: ``rm -rf
 
 4. If using file-based storage, remove recursively all files and
    directories under your file-storage directory: ``rm -rf

-   /srv/ganeti/file-storage/*`` replacing the path with the correct
-   path for your cluster.
+   /srv/ganeti/file-storage/*`` replacing the path with the correct path
+   for your cluster.

 
 5. Stop the ganeti daemons (``/etc/init.d/ganeti stop``) and kill any
    that remain alive (``pgrep ganeti`` and ``pkill ganeti``).
 
 5. Stop the ganeti daemons (``/etc/init.d/ganeti stop``) and kill any
    that remain alive (``pgrep ganeti`` and ``pkill ganeti``).

@@ -335,22 +1049,277 @@ the nodes:
    replacing the path with the correct path for your installation.
 
 On the master node, remove the cluster from the master-netdev (usually
    replacing the path with the correct path for your installation.
 
 On the master node, remove the cluster from the master-netdev (usually

-``xen-br0`` for bridged mode, otherwise ``eth0`` or similar), by
-running ``ip a del $clusterip/32 dev xen-br0`` (use the correct
-cluster ip and network device name).
+``xen-br0`` for bridged mode, otherwise ``eth0`` or similar), by running
+``ip a del $clusterip/32 dev xen-br0`` (use the correct cluster ip and
+network device name).

 
 At this point, the machines are ready for a cluster creation; in case
 
 At this point, the machines are ready for a cluster creation; in case

-you want to remove Ganeti completely, you need to also undo some of
-the SSH changes and log directories:
+you want to remove Ganeti completely, you need to also undo some of the
+SSH changes and log directories:

 
 - ``rm -rf /var/log/ganeti /srv/ganeti`` (replace with the correct
   paths)
 
 - ``rm -rf /var/log/ganeti /srv/ganeti`` (replace with the correct
   paths)

-- remove from ``/root/.ssh`` the keys that Ganeti added (check
-  the ``authorized_keys`` and ``id_dsa`` files)
+- remove from ``/root/.ssh`` the keys that Ganeti added (check the
+  ``authorized_keys`` and ``id_dsa`` files)

 - regenerate the host's SSH keys (check the OpenSSH startup scripts)
 - uninstall Ganeti
 
 Otherwise, if you plan to re-create the cluster, you can just go ahead
 and rerun ``gnt-cluster init``.
 
 - regenerate the host's SSH keys (check the OpenSSH startup scripts)
 - uninstall Ganeti
 
 Otherwise, if you plan to re-create the cluster, you can just go ahead
 and rerun ``gnt-cluster init``.
 

+Tags handling
+-------------
+
+The tags handling (addition, removal, listing) is similar for all the
+objects that support it (instances, nodes, and the cluster).
+
+Limitations
++++++++++++
+
+Note that the set of characters present in a tag and the maximum tag
+length are restricted. Currently the maximum length is 128 characters,
+there can be at most 4096 tags per object, and the set of characters is
+comprised by alphanumeric characters and additionally ``.+*/:-``.
+
+Operations
+++++++++++
+
+Tags can be added via ``add-tags``::
+
+  gnt-instance add-tags INSTANCE a b c
+  gnt-node add-tags INSTANCE a b c
+  gnt-cluster add-tags a b c
+
+
+The above commands add three tags to an instance, to a node and to the
+cluster. Note that the cluster command only takes tags as arguments,
+whereas the node and instance commands first required the node and
+instance name.
+
+Tags can also be added from a file, via the ``--from=FILENAME``
+argument. The file is expected to contain one tag per line.
+
+Tags can also be remove via a syntax very similar to the add one::
+
+  gnt-instance remove-tags INSTANCE a b c
+
+And listed via::
+
+  gnt-instance list-tags
+  gnt-node list-tags
+  gnt-cluster list-tags
+
+Global tag search
++++++++++++++++++
+
+It is also possible to execute a global search on the all tags defined
+in the cluster configuration, via a cluster command::
+
+  gnt-cluster search-tags REGEXP
+
+The parameter expected is a regular expression (see
+:manpage:`regex(7)`). This will return all tags that match the search,
+together with the object they are defined in (the names being show in a
+hierarchical kind of way)::
+
+  node1# gnt-cluster search-tags o
+  /cluster foo
+  /instances/instance1 owner:bar
+
+
+Job operations
+--------------
+
+The various jobs submitted by the instance/node/cluster commands can be
+examined, canceled and archived by various invocations of the
+``gnt-job`` command.
+
+First is the job list command::
+
+  node1# gnt-job list
+  17771 success INSTANCE_QUERY_DATA
+  17773 success CLUSTER_VERIFY_DISKS
+  17775 success CLUSTER_REPAIR_DISK_SIZES
+  17776 error   CLUSTER_RENAME(cluster.example.com)
+  17780 success CLUSTER_REDIST_CONF
+  17792 success INSTANCE_REBOOT(instance1.example.com)
+
+More detailed information about a job can be found via the ``info``
+command::
+
+  node1# gnt-job info 17776
+  Job ID: 17776
+    Status: error
+    Received:         2009-10-25 23:18:02.180569
+    Processing start: 2009-10-25 23:18:02.200335 (delta 0.019766s)
+    Processing end:   2009-10-25 23:18:02.279743 (delta 0.079408s)
+    Total processing time: 0.099174 seconds
+    Opcodes:
+      OP_CLUSTER_RENAME
+        Status: error
+        Processing start: 2009-10-25 23:18:02.200335
+        Processing end:   2009-10-25 23:18:02.252282
+        Input fields:
+          name: cluster.example.com
+        Result:
+          OpPrereqError
+          [Neither the name nor the IP address of the cluster has changed]
+        Execution log:
+
+During the execution of a job, it's possible to follow the output of a
+job, similar to the log that one get from the ``gnt-`` commands, via the
+watch command::
+
+  node1# gnt-instance add --submit … instance1
+  JobID: 17818
+  node1# gnt-job watch 17818
+  Output from job 17818 follows
+  -----------------------------
+  Mon Oct 26 00:22:48 2009  - INFO: Selected nodes for instance instance1 via iallocator dumb: node1, node2
+  Mon Oct 26 00:22:49 2009 * creating instance disks...
+  Mon Oct 26 00:22:52 2009 adding instance instance1 to cluster config
+  Mon Oct 26 00:22:52 2009  - INFO: Waiting for instance instance1 to sync disks.
+  …
+  Mon Oct 26 00:23:03 2009 creating os for instance xen-devi-18.fra.corp.google.com on node mpgntac4.fra.corp.google.com
+  Mon Oct 26 00:23:03 2009 * running the instance OS create scripts...
+  Mon Oct 26 00:23:13 2009 * starting instance...
+  node1#
+
+This is useful if you need to follow a job's progress from multiple
+terminals.
+
+A job that has not yet started to run can be canceled::
+
+  node1# gnt-job cancel 17810
+
+But not one that has already started execution::
+
+  node1# gnt-job cancel 17805
+  Job 17805 is no longer waiting in the queue
+
+There are two queues for jobs: the *current* and the *archive*
+queue. Jobs are initially submitted to the current queue, and they stay
+in that queue until they have finished execution (either successfully or
+not). At that point, they can be moved into the archive queue, and the
+ganeti-watcher script will do this automatically after 6 hours. The
+ganeti-cleaner script will remove the jobs from the archive directory
+after three weeks.
+
+Note that only jobs in the current queue can be viewed via the list and
+info commands; Ganeti itself doesn't examine the archive directory. If
+you need to see an older job, either move the file manually in the
+top-level queue directory, or look at its contents (it's a
+JSON-formatted file).
+
+Ganeti tools
+------------
+
+Beside the usual ``gnt-`` and ``ganeti-`` commands which are provided
+and installed in ``$prefix/sbin`` at install time, there are a couple of
+other tools installed which are used seldom but can be helpful in some
+cases.
+
+lvmstrap
+++++++++
+
+The ``lvmstrap`` tool, introduced in :ref:`configure-lvm-label` section,
+has two modes of operation:
+
+- ``diskinfo`` shows the discovered disks on the system and their status
+- ``create`` takes all not-in-use disks and creates a volume group out
+  of them
+
+.. warning:: The ``create`` argument to this command causes data-loss!
+
+cfgupgrade
+++++++++++
+
+The ``cfgupgrade`` tools is used to upgrade between major (and minor)
+Ganeti versions. Point-releases are usually transparent for the admin.
+
+More information about the upgrade procedure is listed on the wiki at
+http://code.google.com/p/ganeti/wiki/UpgradeNotes.
+
+cfgshell
+++++++++
+
+.. note:: This command is not actively maintained; make sure you backup
+   your configuration before using it
+
+This can be used as an alternative to direct editing of the
+main configuration file if Ganeti has a bug and prevents you, for
+example, from removing an instance or a node from the configuration
+file.
+
+.. _burnin-label:
+
+burnin
+++++++
+
+.. warning:: This command will erase existing instances if given as
+   arguments!
+
+This tool is used to exercise either the hardware of machines or
+alternatively the Ganeti software. It is safe to run on an existing
+cluster **as long as you don't pass it existing instance names**.
+
+The command will, by default, execute a comprehensive set of operations
+against a list of instances, these being:
+
+- creation
+- disk replacement (for redundant instances)
+- failover and migration (for redundant instances)
+- move (for non-redundant instances)
+- disk growth
+- add disks, remove disk
+- add NICs, remove NICs
+- export and then import
+- rename
+- reboot
+- shutdown/startup
+- and finally removal of the test instances
+
+Executing all these operations will test that the hardware performs
+well: the creation, disk replace, disk add and disk growth will exercise
+the storage and network; the migrate command will test the memory of the
+systems. Depending on the passed options, it can also test that the
+instance OS definitions are executing properly the rename, import and
+export operations.
+
+Other Ganeti projects
+---------------------
+
+There are two other Ganeti-related projects that can be useful in a
+Ganeti deployment. These can be downloaded from the project site
+(http://code.google.com/p/ganeti/) and the repositories are also on the
+project git site (http://git.ganeti.org).
+
+NBMA tools
+++++++++++
+
+The ``ganeti-nbma`` software is designed to allow instances to live on a
+separate, virtual network from the nodes, and in an environment where
+nodes are not guaranteed to be able to reach each other via multicasting
+or broadcasting. For more information see the README in the source
+archive.
+
+ganeti-htools
++++++++++++++
+
+The ``ganeti-htools`` software consists of a set of tools:
+
+- ``hail``: an advanced iallocator script compared to Ganeti's builtin
+  one
+- ``hbal``: a tool for rebalancing the cluster, i.e. moving instances
+  around in order to better use the resources on the nodes
+- ``hspace``: a tool for estimating the available capacity of a cluster,
+  so that capacity planning can be done efficiently
+
+For more information and installation instructions, see the README file
+in the source archive.
+

 .. vim: set textwidth=72 :
 .. vim: set textwidth=72 :

+.. Local Variables:
+.. mode: rst
+.. fill-column: 72
+.. End:

diff --git a/doc/index.rst b/doc/index.rst
index 059016a..38983d7 100644 (file)
--- a/doc/index.rst
+++ b/doc/index.rst
@@ -11,6 +11,7 @@ Contents:
 
    install.rst
    admin.rst
 
    install.rst
    admin.rst

+   walkthrough.rst

    security.rst
    design-2.0.rst
    design-2.1.rst
    security.rst
    design-2.0.rst
    design-2.1.rst

diff --git a/doc/install.rst b/doc/install.rst
index 5c53f0f..4029a82 100644 (file)
--- a/doc/install.rst
+++ b/doc/install.rst
@@ -5,26 +5,28 @@ Documents Ganeti version |version|
 
 .. contents::
 
 
 .. contents::
 

+.. highlight:: text
+

 Introduction
 ------------
 
 Ganeti is a cluster virtualization management system based on Xen or
 Introduction
 ------------
 
 Ganeti is a cluster virtualization management system based on Xen or

-KVM. This document explains how to bootstrap a Ganeti node (Xen
-*dom0*), create a running cluster and install virtual instance (Xen
-*domU*).  You need to repeat most of the steps in this document for
-every node you want to install, but of course we recommend creating
-some semi-automatic procedure if you plan to deploy Ganeti on a
-medium/large scale.
+KVM. This document explains how to bootstrap a Ganeti node (Xen *dom0*,
+the host Linux system for KVM), create a running cluster and install
+virtual instances (Xen *domUs*, KVM guests).  You need to repeat most of
+the steps in this document for every node you want to install, but of
+course we recommend creating some semi-automatic procedure if you plan
+to deploy Ganeti on a medium/large scale.

 
 A basic Ganeti terminology glossary is provided in the introductory
 
 A basic Ganeti terminology glossary is provided in the introductory

-section of the *Ganeti administrator's guide*. Please refer to that
-document if you are uncertain about the terms we are using.
+section of the :doc:`admin`. Please refer to that document if you are
+uncertain about the terms we are using.

 
 

-Ganeti has been developed for Linux and is distribution-agnostic.
+Ganeti has been developed for Linux and should be distribution-agnostic.

 This documentation will use Debian Lenny as an example system but the
 This documentation will use Debian Lenny as an example system but the

-examples can easily be translated to any other distribution. ou are
-expected to be familiar with your distribution, its package management
-system, and Xen or KVM before trying to use Ganeti.
+examples can be translated to any other distribution. You are expected
+to be familiar with your distribution, its package management system,
+and Xen or KVM before trying to use Ganeti.

 
 This document is divided into two main sections:
 
 
 This document is divided into two main sections:
 

@@ -33,9 +35,9 @@ This document is divided into two main sections:
 - Configuration of the environment for Ganeti
 
 Each of these is divided into sub-sections. While a full Ganeti system
 - Configuration of the environment for Ganeti
 
 Each of these is divided into sub-sections. While a full Ganeti system

-will need all of the steps specified, some are not strictly required
-for every environment. Which ones they are, and why, is specified in
-the corresponding sections.
+will need all of the steps specified, some are not strictly required for
+every environment. Which ones they are, and why, is specified in the
+corresponding sections.

 
 Installing the base system and base components
 ----------------------------------------------
 
 Installing the base system and base components
 ----------------------------------------------

@@ -43,15 +45,14 @@ Installing the base system and base components
 Hardware requirements
 +++++++++++++++++++++
 
 Hardware requirements
 +++++++++++++++++++++
 

-Any system supported by your Linux distribution is fine. 64-bit
-systems are better as they can support more memory.
+Any system supported by your Linux distribution is fine. 64-bit systems
+are better as they can support more memory.

 
 

-Any disk drive recognized by Linux (``IDE``/``SCSI``/``SATA``/etc.)
-is supported in Ganeti. Note that no shared storage (e.g.  ``SAN``) is
-needed to get high-availability features (but of course, one can be
-used to store the images). It is highly recommended to use more than
-one disk drive to improve speed. But Ganeti also works with one disk
-per machine.
+Any disk drive recognized by Linux (``IDE``/``SCSI``/``SATA``/etc.) is
+supported in Ganeti. Note that no shared storage (e.g. ``SAN``) is
+needed to get high-availability features (but of course, one can be used
+to store the images). It is highly recommended to use more than one disk
+drive to improve speed. But Ganeti also works with one disk per machine.

 
 Installing the base system
 ++++++++++++++++++++++++++
 
 Installing the base system
 ++++++++++++++++++++++++++

@@ -59,15 +60,14 @@ Installing the base system
 **Mandatory** on all nodes.
 
 It is advised to start with a clean, minimal install of the operating
 **Mandatory** on all nodes.
 
 It is advised to start with a clean, minimal install of the operating

-system. The only requirement you need to be aware of at this stage is
-to partition leaving enough space for a big (**minimum** 20GiB) LVM
-volume group which will then host your instance filesystems, if you
-want to use all Ganeti features. The volume group name Ganeti 2.0 uses
-(by default) is ``xenvg``.
-
-You can also use file-based storage only, without LVM, but this setup
-is not detailed in this document.
+system. The only requirement you need to be aware of at this stage is to
+partition leaving enough space for a big (**minimum** 20GiB) LVM volume
+group which will then host your instance filesystems, if you want to use
+all Ganeti features. The volume group name Ganeti uses (by default) is
+``xenvg``.

 
 

+You can also use file-based storage only, without LVM, but this setup is
+not detailed in this document.

 
 While you can use an existing system, please note that the Ganeti
 installation is intrusive in terms of changes to the system
 
 While you can use an existing system, please note that the Ganeti
 installation is intrusive in terms of changes to the system

@@ -135,10 +135,10 @@ virtualization environments in mind the only two currently useable on a
 live system are Xen and KVM. Supported Xen versions are: 3.0.3, 3.0.4
 and 3.1.  Supported KVM version are 72 and above.
 
 live system are Xen and KVM. Supported Xen versions are: 3.0.3, 3.0.4
 and 3.1.  Supported KVM version are 72 and above.
 

-Please follow your distribution's recommended way to install and set
-up Xen, or install Xen from the upstream source, if you wish,
-following their manual. For KVM, make sure you have a KVM-enabled
-kernel and the KVM tools.
+Please follow your distribution's recommended way to install and set up
+Xen, or install Xen from the upstream source, if you wish, following
+their manual. For KVM, make sure you have a KVM-enabled kernel and the
+KVM tools.

 
 After installing Xen, you need to reboot into your new system. On some
 distributions this might involve configuring GRUB appropriately, whereas
 
 After installing Xen, you need to reboot into your new system. On some
 distributions this might involve configuring GRUB appropriately, whereas

@@ -147,26 +147,25 @@ kernels. For KVM no reboot should be necessary.
 
 .. admonition:: Xen on Debian
 
 
 .. admonition:: Xen on Debian
 

-   Under Lenny or Etch you can install the relevant
-   ``xen-linux-system`` package, which will pull in both the
-   hypervisor and the relevant kernel. Also, if you are installing a
-   32-bit Lenny/Etch, you should install the ``libc6-xen`` package
-   (run ``apt-get install libc6-xen``).
+   Under Lenny or Etch you can install the relevant ``xen-linux-system``
+   package, which will pull in both the hypervisor and the relevant
+   kernel. Also, if you are installing a 32-bit Lenny/Etch, you should
+   install the ``libc6-xen`` package (run ``apt-get install
+   libc6-xen``).

 
 Xen settings
 ~~~~~~~~~~~~
 
 It's recommended that dom0 is restricted to a low amount of memory
 
 Xen settings
 ~~~~~~~~~~~~
 
 It's recommended that dom0 is restricted to a low amount of memory

-(512MiB or 1GiB is reasonable) and that memory ballooning is disabled
-in the file ``/etc/xen/xend-config.sxp`` by setting
-the value ``dom0-min-mem`` to 0,
-like this::
+(512MiB or 1GiB is reasonable) and that memory ballooning is disabled in
+the file ``/etc/xen/xend-config.sxp`` by setting the value
+``dom0-min-mem`` to 0, like this::

 
   (dom0-min-mem 0)
 
 For optimum performance when running both CPU and I/O intensive
 
   (dom0-min-mem 0)
 
 For optimum performance when running both CPU and I/O intensive

-instances, it's also recommended that the dom0 is restricted to one
-CPU only, for example by booting with the kernel parameter ``nosmp``.
+instances, it's also recommended that the dom0 is restricted to one CPU
+only, for example by booting with the kernel parameter ``nosmp``.

 
 It is recommended that you disable xen's automatic save of virtual
 machines at system shutdown and subsequent restore of them at reboot.
 
 It is recommended that you disable xen's automatic save of virtual
 machines at system shutdown and subsequent restore of them at reboot.

@@ -195,8 +194,7 @@ The last line assumes that all your nodes have secondary IPs in the
      ## Xen hypervisor options to use with the default Xen boot option
      # xenhopt=dom0_mem=1024M
 
      ## Xen hypervisor options to use with the default Xen boot option
      # xenhopt=dom0_mem=1024M
 

-   and the ``xenkopt`` needs to include the ``nosmp`` option like
-   this::
+   and the ``xenkopt`` needs to include the ``nosmp`` option like this::

 
      ## Xen Linux kernel options to use with the default Xen boot option
      # xenkopt=nosmp
 
      ## Xen Linux kernel options to use with the default Xen boot option
      # xenkopt=nosmp

@@ -207,8 +205,8 @@ The last line assumes that all your nodes have secondary IPs in the
 
      /sbin/update-grub
 
 
      /sbin/update-grub
 

-If you want to run HVM instances too with Ganeti and want VNC access
-to the console of your instances, set the following two entries in
+If you want to run HVM instances too with Ganeti and want VNC access to
+the console of your instances, set the following two entries in

 ``/etc/xen/xend-config.sxp``::
 
   (vnc-listen '0.0.0.0') (vncpasswd '')
 ``/etc/xen/xend-config.sxp``::
 
   (vnc-listen '0.0.0.0') (vncpasswd '')

@@ -221,12 +219,11 @@ Selecting the instance kernel
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 After you have installed Xen, you need to tell Ganeti exactly what
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 After you have installed Xen, you need to tell Ganeti exactly what

-kernel to use for the instances it will create. This is done by
-creating a symlink from your actual kernel to
-``/boot/vmlinuz-2.6-xenU``, and one from your initrd
-to ``/boot/initrd-2.6-xenU``. Note that if you don't
-use an initrd for the domU kernel, you don't need
-to create the initrd symlink.
+kernel to use for the instances it will create. This is done by creating
+a symlink from your actual kernel to ``/boot/vmlinuz-2.6-xenU``, and one
+from your initrd to ``/boot/initrd-2.6-xenU`` [#defkernel]_. Note that
+if you don't use an initrd for the domU kernel, you don't need to create
+the initrd symlink.

 
 .. admonition:: Debian
 
 
 .. admonition:: Debian
 

@@ -240,45 +237,52 @@ to create the initrd symlink.
 Installing DRBD
 +++++++++++++++
 
 Installing DRBD
 +++++++++++++++
 

-Recommended on all nodes: DRBD_ is required if you want to use the
-high availability (HA) features of Ganeti, but optional if you don't
-require HA or only run Ganeti on single-node clusters. You can upgrade
-a non-HA cluster to an HA one later, but you might need to export and
+Recommended on all nodes: DRBD_ is required if you want to use the high
+availability (HA) features of Ganeti, but optional if you don't require
+them or only run Ganeti on single-node clusters. You can upgrade a
+non-HA cluster to an HA one later, but you might need to export and

 re-import all your instances to take advantage of the new features.
 
 .. _DRBD: http://www.drbd.org/
 
 re-import all your instances to take advantage of the new features.
 
 .. _DRBD: http://www.drbd.org/
 

-Supported DRBD versions: 8.0.x. It's recommended to have at least
-version 8.0.12.
+Supported DRBD versions: 8.0+. It's recommended to have at least version
+8.0.12. Note that for version 8.2 and newer it is needed to pass the
+``usermode_helper=/bin/true`` parameter to the module, either by
+configuring ``/etc/modules`` or when inserting it manually.

 
 Now the bad news: unless your distribution already provides it
 
 Now the bad news: unless your distribution already provides it

-installing DRBD might involve recompiling your kernel or anyway
-fiddling with it. Hopefully at least the Xen-ified kernel source to
-start from will be provided.
+installing DRBD might involve recompiling your kernel or anyway fiddling
+with it. Hopefully at least the Xen-ified kernel source to start from
+will be provided (if you intend to use Xen).

 
 The good news is that you don't need to configure DRBD at all. Ganeti
 
 The good news is that you don't need to configure DRBD at all. Ganeti

-will do it for you for every instance you set up.  If you have the
-DRBD utils installed and the module in your kernel you're fine. Please
-check that your system is configured to load the module at every boot,
-and that it passes the following option to the module
-``minor_count=255``. This will allow you to use up to 128 instances
-per node (for most clusters 128 should be enough, though).
+will do it for you for every instance you set up.  If you have the DRBD
+utils installed and the module in your kernel you're fine. Please check
+that your system is configured to load the module at every boot, and
+that it passes the following option to the module:
+``minor_count=NUMBER``. We recommend that you use 128 as the value of
+the minor_count - this will allow you to use up to 64 instances in total
+per node (both primary and secondary, when using only one disk per
+instance). You can increase the number up to 255 if you need more
+instances on a node.
+

 
 .. admonition:: Debian
 
 
 .. admonition:: Debian
 

-   On Debian, you can just install (build) the DRBD 8.0.x module with
-   the following commands (make sure you are running the Xen kernel)::
+   On Debian, you can just install (build) the DRBD module with the
+   following commands, making sure you are running the target (Xen or
+   KVM) kernel::

 
      apt-get install drbd8-source drbd8-utils
      m-a update
      m-a a-i drbd8
 
      apt-get install drbd8-source drbd8-utils
      m-a update
      m-a a-i drbd8

-     echo drbd minor_count=128 >> /etc/modules
+     echo drbd minor_count=128 usermode_helper=/bin/true >> /etc/modules

      depmod -a
      depmod -a

-     modprobe drbd minor_count=128
+     modprobe drbd minor_count=128 usermode_helper=/bin/true

 
 

-   It is also recommended that you comment out the default resources
-   in the ``/etc/drbd.conf`` file, so that the init script doesn't try
-   to configure any drbd devices. You can do this by prefixing all
+   It is also recommended that you comment out the default resources in
+   the ``/etc/drbd.conf`` file, so that the init script doesn't try to
+   configure any drbd devices. You can do this by prefixing all

    *resource* lines in the file with the keyword *skip*, like this::
 
      skip resource r0 {
    *resource* lines in the file with the keyword *skip*, like this::
 
      skip resource r0 {

@@ -346,10 +350,10 @@ such a bridge at startup, but your distribution might have a different
 way to do things, and you'll definitely need to manually set it up under
 KVM.
 
 way to do things, and you'll definitely need to manually set it up under
 KVM.
 

-Beware that the default name Ganeti uses is ``xen-br0`` (which was
-used in Xen 2.0) while Xen 3.0 uses ``xenbr0`` by default. The default
-bridge your Ganeti cluster will use for new instances can be specified
-at cluster initialization time.
+Beware that the default name Ganeti uses is ``xen-br0`` (which was used
+in Xen 2.0) while Xen 3.0 uses ``xenbr0`` by default. The default bridge
+your Ganeti cluster will use for new instances can be specified at
+cluster initialization time.

 
 If you want to run in "routing mode" you need to specify that at cluster
 init time (using the --nicparam option), and then no bridge will be
 
 If you want to run in "routing mode" you need to specify that at cluster
 init time (using the --nicparam option), and then no bridge will be

@@ -408,6 +412,8 @@ commands::
   bridge name     bridge id               STP enabled     interfaces
   xen-br0         8000.0020fc1ed55d       no              eth0
 
   bridge name     bridge id               STP enabled     interfaces
   xen-br0         8000.0020fc1ed55d       no              eth0
 

+.. _configure-lvm-label:
+

 Configuring LVM
 +++++++++++++++
 
 Configuring LVM
 +++++++++++++++
 

@@ -415,10 +421,10 @@ Configuring LVM
 
 The volume group is required to be at least 20GiB.
 
 
 The volume group is required to be at least 20GiB.
 

-If you haven't configured your LVM volume group at install time you
-need to do it before trying to initialize the Ganeti cluster. This is
-done by formatting the devices/partitions you want to use for it and
-then adding them to the relevant volume group::
+If you haven't configured your LVM volume group at install time you need
+to do it before trying to initialize the Ganeti cluster. This is done by
+formatting the devices/partitions you want to use for it and then adding
+them to the relevant volume group::

 
   pvcreate /dev/sda3
   vgcreate xenvg /dev/sda3
 
   pvcreate /dev/sda3
   vgcreate xenvg /dev/sda3

@@ -437,12 +443,16 @@ command::
 
 Optional: it is recommended to configure LVM not to scan the DRBD
 devices for physical volumes. This can be accomplished by editing
 
 Optional: it is recommended to configure LVM not to scan the DRBD
 devices for physical volumes. This can be accomplished by editing

-``/etc/lvm/lvm.conf`` and adding the
-``/dev/drbd[0-9]+`` regular expression to the
-``filter`` variable, like this::
+``/etc/lvm/lvm.conf`` and adding the ``/dev/drbd[0-9]+`` regular
+expression to the ``filter`` variable, like this::

 
   filter = ["r|/dev/cdrom|", "r|/dev/drbd[0-9]+|" ]
 
 
   filter = ["r|/dev/cdrom|", "r|/dev/drbd[0-9]+|" ]
 

+Note that with Ganeti a helper script is provided - ``lvmstrap`` which
+will erase and configure as LVM any not in-use disk on your system. This
+is dangerous and it's recommended to read its ``--help`` output if you
+want to use it.
+

 Installing Ganeti
 +++++++++++++++++
 
 Installing Ganeti
 +++++++++++++++++
 

@@ -459,17 +469,36 @@ and install it (replace 2.0.0 with the latest version)::
   make install
   mkdir /srv/ganeti/ /srv/ganeti/os /srv/ganeti/export
 
   make install
   mkdir /srv/ganeti/ /srv/ganeti/os /srv/ganeti/export
 

-You also need to copy the file
-``doc/examples/ganeti.initd`` from the source archive
-to ``/etc/init.d/ganeti`` and register it with your
+You also need to copy the file ``doc/examples/ganeti.initd`` from the
+source archive to ``/etc/init.d/ganeti`` and register it with your

 distribution's startup scripts, for example in Debian::
 
   update-rc.d ganeti defaults 20 80
 
 distribution's startup scripts, for example in Debian::
 
   update-rc.d ganeti defaults 20 80
 

-In order to automatically restart failed instances, you need to setup
-a cron job run the *ganeti-watcher* command. A sample cron file is
-provided in the source at ``doc/examples/ganeti.cron`` and you can
-copy that (eventually altering the path) to ``/etc/cron.d/ganeti``.
+In order to automatically restart failed instances, you need to setup a
+cron job run the *ganeti-watcher* command. A sample cron file is
+provided in the source at ``doc/examples/ganeti.cron`` and you can copy
+that (eventually altering the path) to ``/etc/cron.d/ganeti``.
+
+What gets installed
+~~~~~~~~~~~~~~~~~~~
+
+The above ``make install`` invocation, or installing via your
+distribution mechanisms, will install on the system:
+
+- a set of python libraries under the *ganeti* namespace (depending on
+  the python version this can be located in either
+  ``lib/python-$ver/site-packages`` or various other locations)
+- a set of programs under ``/usr/local/sbin`` or ``/usr/sbin``
+- man pages for the above programs
+- a set of tools under the ``lib/ganeti/tools`` directory
+- an example iallocator script (see the admin guide for details) under
+  ``lib/ganeti/iallocators``
+- a cron job that is needed for cluster maintenance
+- an init script for automatic startup of Ganeti daemons
+- provided but not installed automatically by ``make install`` is a bash
+  completion script that hopefully will ease working with the many
+  cluster commands

 
 Installing the Operating System support packages
 ++++++++++++++++++++++++++++++++++++++++++++++++
 
 Installing the Operating System support packages
 ++++++++++++++++++++++++++++++++++++++++++++++++

@@ -479,10 +508,9 @@ Installing the Operating System support packages
 To be able to install instances you need to have an Operating System
 installation script. An example OS that works under Debian and can
 install Debian and Ubuntu instace OSes is provided on the project web
 To be able to install instances you need to have an Operating System
 installation script. An example OS that works under Debian and can
 install Debian and Ubuntu instace OSes is provided on the project web

-site.  Download it from the project page and follow the instructions
-in the ``README`` file.  Here is the installation procedure (replace
-0.7 with the latest version that is compatible with your ganeti
-version)::
+site.  Download it from the project page and follow the instructions in
+the ``README`` file.  Here is the installation procedure (replace 0.7
+with the latest version that is compatible with your ganeti version)::

 
   cd /usr/local/src/
   wget http://ganeti.googlecode.com/files/ganeti-instance-debootstrap-0.7.tar.gz
 
   cd /usr/local/src/
   wget http://ganeti.googlecode.com/files/ganeti-instance-debootstrap-0.7.tar.gz

@@ -511,21 +539,21 @@ Alternatively, you can create your own OS definitions. See the manpage
 Initializing the cluster
 ++++++++++++++++++++++++
 
 Initializing the cluster
 ++++++++++++++++++++++++
 

-**Mandatory** on one node per cluster.
+**Mandatory** once per cluster, on the first node.

 
 

-The last step is to initialize the cluster. After you've repeated the
+The last step is to initialize the cluster. After you have repeated the

 above process on all of your nodes, choose one as the master, and
 execute::
 
   gnt-cluster init <CLUSTERNAME>
 
 above process on all of your nodes, choose one as the master, and
 execute::
 
   gnt-cluster init <CLUSTERNAME>
 

-The *CLUSTERNAME* is a hostname, which must be resolvable (e.g. it
-must exist in DNS or in ``/etc/hosts``) by all the nodes in the
-cluster. You must choose a name different from any of the nodes names
-for a multi-node cluster. In general the best choice is to have a
-unique name for a cluster, even if it consists of only one machine, as
-you will be able to expand it later without any problems. Please note
-that the hostname used for this must resolve to an IP address reserved
+The *CLUSTERNAME* is a hostname, which must be resolvable (e.g. it must
+exist in DNS or in ``/etc/hosts``) by all the nodes in the cluster. You
+must choose a name different from any of the nodes names for a
+multi-node cluster. In general the best choice is to have a unique name
+for a cluster, even if it consists of only one machine, as you will be
+able to expand it later without any problems. Please note that the
+hostname used for this must resolve to an IP address reserved

 **exclusively** for this purpose, and cannot be the name of the first
 (master) node.
 
 **exclusively** for this purpose, and cannot be the name of the first
 (master) node.
 

@@ -534,18 +562,18 @@ all, use ``--nicparams``.
 
 If the bridge name you are using is not ``xen-br0``, use the *-b
 <BRIDGENAME>* option to specify the bridge name. In this case, you
 
 If the bridge name you are using is not ``xen-br0``, use the *-b
 <BRIDGENAME>* option to specify the bridge name. In this case, you

-should also use the *--master-netdev <BRIDGENAME>* option with the
-same BRIDGENAME argument.
+should also use the *--master-netdev <BRIDGENAME>* option with the same
+BRIDGENAME argument.

 
 You can use a different name than ``xenvg`` for the volume group (but
 note that the name must be identical on all nodes). In this case you
 
 You can use a different name than ``xenvg`` for the volume group (but
 note that the name must be identical on all nodes). In this case you

-need to specify it by passing the *-g <VGNAME>* option to
-``gnt-cluster init``.
+need to specify it by passing the *-g <VGNAME>* option to ``gnt-cluster
+init``.

 
 

-To set up the cluster as an HVM cluster, use the
+To set up the cluster as an Xen HVM cluster, use the

 ``--enabled-hypervisors=xen-hvm`` option to enable the HVM hypervisor
 ``--enabled-hypervisors=xen-hvm`` option to enable the HVM hypervisor

-(you can also add ``,xen-pvm`` to enable the PVM one too). You will
-also need to create the VNC cluster password file
+(you can also add ``,xen-pvm`` to enable the PVM one too). You will also
+need to create the VNC cluster password file

 ``/etc/ganeti/vnc-cluster-password`` which contains one line with the
 default VNC password for the cluster.
 
 ``/etc/ganeti/vnc-cluster-password`` which contains one line with the
 default VNC password for the cluster.
 

@@ -560,9 +588,9 @@ Joining the nodes to the cluster
 
 **Mandatory** for all the other nodes.
 
 
 **Mandatory** for all the other nodes.
 

-After you have initialized your cluster you need to join the other
-nodes to it. You can do so by executing the following command on the
-master node::
+After you have initialized your cluster you need to join the other nodes
+to it. You can do so by executing the following command on the master
+node::

 
   gnt-node add <NODENAME>
 
 
   gnt-node add <NODENAME>
 

@@ -577,92 +605,49 @@ improve performance or to enhance security) you need to configure an
 additional interface for each node.  Use the *-s* option with
 ``gnt-cluster init`` and ``gnt-node add`` to specify the IP address of
 this secondary interface to use for each node. Note that if you
 additional interface for each node.  Use the *-s* option with
 ``gnt-cluster init`` and ``gnt-node add`` to specify the IP address of
 this secondary interface to use for each node. Note that if you

-specified this option at cluster setup time, you must afterwards use
-it for every node add operation.
+specified this option at cluster setup time, you must afterwards use it
+for every node add operation.

 
 Testing the setup
 +++++++++++++++++
 
 
 Testing the setup
 +++++++++++++++++
 

-Execute the ``gnt-node list`` command to see all nodes in the
-cluster::
+Execute the ``gnt-node list`` command to see all nodes in the cluster::

 
   # gnt-node list
   Node              DTotal  DFree MTotal MNode MFree Pinst Sinst
   node1.example.com 197404 197404   2047  1896   125     0     0
 
 
   # gnt-node list
   Node              DTotal  DFree MTotal MNode MFree Pinst Sinst
   node1.example.com 197404 197404   2047  1896   125     0     0
 

-Setting up and managing virtual instances
------------------------------------------
-
-Setting up virtual instances
-++++++++++++++++++++++++++++
-
-This step shows how to setup a virtual instance with either
-non-mirrored disks (``plain``) or with network mirrored disks
-(``drbd``).  All commands need to be executed on the Ganeti master
-node (the one on which ``gnt-cluster init`` was run).  Verify that the
-OS scripts are present on all cluster nodes with ``gnt-os list``.
-
-
-To create a virtual instance, you need a hostname which is resolvable
-(DNS or ``/etc/hosts`` on all nodes). The following command will
-create a non-mirrored instance for you::
-
-  gnt-instance add -t plain -s 1G -n node1 -o debootstrap instance1.example.com
-  * creating instance disks...
-  adding instance instance1.example.com to cluster config
-   - INFO: Waiting for instance instance1.example.com to sync disks.
-   - INFO: Instance instance1.example.com's disks are in sync.
-  creating os for instance instance1.example.com on node node1.example.com
-  * running the instance OS create scripts...
-  * starting instance...
-
-The above instance will have no network interface enabled. You can
-access it over the virtual console with ``gnt-instance console
-inst1``. There is no password for root. As this is a Debian instance,
-you can modify the ``/etc/network/interfaces`` file to setup the
-network interface (eth0 is the name of the interface provided to the
-instance).
-
-To create a network mirrored instance, change the argument to the *-t*
-option from ``plain`` to ``drbd`` and specify the node on which the
-mirror should reside with the second value of the *--node* option,
-like this (note that the command output includes timestamps which have
-been removed for clarity)::
-
-  # gnt-instance add -t drbd -s 1G -n node1:node2 -o debootstrap instance2
-  * creating instance disks...
-  adding instance instance2.example.com to cluster config
-   - INFO: Waiting for instance instance2.example.com to sync disks.
-   - INFO: - device disk/0: 35.50% done, 11 estimated seconds remaining
-   - INFO: - device disk/0: 100.00% done, 0 estimated seconds remaining
-   - INFO: Instance instance2.example.com's disks are in sync.
-  creating os for instance instance2.example.com on node node1.example.com
-  * running the instance OS create scripts...
-  * starting instance...
-
-Managing virtual instances
-++++++++++++++++++++++++++
-
-All commands need to be executed on the Ganeti master node.
-
-To access the console of an instance, run::
-
-  gnt-instance console INSTANCENAME
+The above shows a couple of things:

 
 

-To shutdown an instance, run::
+- The various Ganeti daemons can talk to each other
+- Ganeti can examine the storage of the node (DTotal/DFree)
+- Ganeti can talk to the selected hypervisor (MTotal/MNode/MFree)

 
 

-  gnt-instance shutdown INSTANCENAME
+Cluster burnin
+~~~~~~~~~~~~~~

 
 

-To startup an instance, run::
+With Ganeti a tool called :command:`burnin` is provided that can test
+most of the Ganeti functionality. The tool is installed under the
+``lib/ganeti/tools`` directory (either under ``/usr`` or ``/usr/local``
+based on the installation method). See more details under
+:ref:`burnin-label`.

 
 

-  gnt-instance startup INSTANCENAME
+Further steps
+-------------

 
 

-To failover an instance to its secondary node (only possible with
-``drbd`` disk templates), run::
+You can now proceed either to the :doc:`admin`, or read the manpages of
+the various commands (:manpage:`ganeti(7)`, :manpage:`gnt-cluster(8)`,
+:manpage:`gnt-node(8)`, :manpage:`gnt-instance(8)`,
+:manpage:`gnt-job(8)`).

 
 

-  gnt-instance failover INSTANCENAME
+.. rubric:: Footnotes

 
 

-For more instance and cluster administration details, see the
-*Ganeti administrator's guide*.
+.. [#defkernel] The kernel and initrd paths can be changed at either
+   cluster level (which changes the default for all instances) or at
+   instance level.

 
 .. vim: set textwidth=72 :
 
 .. vim: set textwidth=72 :

+.. Local Variables:
+.. mode: rst
+.. fill-column: 72
+.. End:

diff --git a/doc/rapi.rst b/doc/rapi.rst
index ee64dec..dd06ca6 100644 (file)
--- a/doc/rapi.rst
+++ b/doc/rapi.rst
@@ -34,11 +34,11 @@ it.
 ``bulk``
 ++++++++
 
 ``bulk``
 ++++++++
 

-Bulk-mode means that for the resources which usually return just a
-list of child resources (e.g. ``/2/instances`` which returns just
-instance names), the output will instead contain detailed data for all
-these subresources. This is more efficient than query-ing the
-sub-resources themselves.
+Bulk-mode means that for the resources which usually return just a list
+of child resources (e.g. ``/2/instances`` which returns just instance
+names), the output will instead contain detailed data for all these
+subresources. This is more efficient than query-ing the sub-resources
+themselves.

 
 ``dry-run``
 +++++++++++
 
 ``dry-run``
 +++++++++++

@@ -47,9 +47,9 @@ The optional *dry-run* argument, if provided and set to a positive
 integer value (e.g. ``?dry-run=1``), signals to Ganeti that the job
 should not be executed, only the pre-execution checks will be done.
 
 integer value (e.g. ``?dry-run=1``), signals to Ganeti that the job
 should not be executed, only the pre-execution checks will be done.
 

-This is useful in trying to determine (without guarantees though, as
-in the meantime the cluster state could have changed) if the operation
-is likely to succeed or at least start executing.
+This is useful in trying to determine (without guarantees though, as in
+the meantime the cluster state could have changed) if the operation is
+likely to succeed or at least start executing.

 
 ``force``
 +++++++++++
 
 ``force``
 +++++++++++

@@ -60,8 +60,8 @@ inconsistent (e.g. because there are not enough master candidates).
 Usage examples
 --------------
 
 Usage examples
 --------------
 

-You can access the API using your favorite programming language as
-long as it supports network connections.
+You can access the API using your favorite programming language as long
+as it supports network connections.

 
 Shell
 +++++
 
 Shell
 +++++

@@ -80,7 +80,9 @@ or curl::
 Python
 ++++++
 
 Python
 ++++++
 

-.. highlight: python
+.. highlight:: python
+
+::

 
   import urllib2
   f = urllib2.urlopen('https://CLUSTERNAME:5080/2/info')
 
   import urllib2
   f = urllib2.urlopen('https://CLUSTERNAME:5080/2/info')

@@ -91,9 +93,9 @@ JavaScript
 ++++++++++
 
 .. warning:: While it's possible to use JavaScript, it poses several
 ++++++++++
 
 .. warning:: While it's possible to use JavaScript, it poses several

-  potential problems, including browser blocking request due to
-  non-standard ports or different domain names. Fetching the data on
-  the webserver is easier.
+   potential problems, including browser blocking request due to
+   non-standard ports or different domain names. Fetching the data on
+   the webserver is easier.

 
 .. highlight:: javascript
 
 
 .. highlight:: javascript
 

@@ -230,9 +232,9 @@ Example::
       }
     ]
 
       }
     ]
 

-If the optional *bulk* argument is provided and set to a true value
-(i.e ``?bulk=1``), the output contains detailed information about
-instances as a list.
+If the optional *bulk* argument is provided and set to a true value (i.e
+``?bulk=1``), the output contains detailed information about instances
+as a list.

 
 Example::
 
 
 Example::
 

@@ -271,9 +273,9 @@ Creates an instance.
 
 If the optional *dry-run* argument is provided and set to a positive
 integer valu (e.g. ``?dry-run=1``), the job will not be actually
 
 If the optional *dry-run* argument is provided and set to a positive
 integer valu (e.g. ``?dry-run=1``), the job will not be actually

-executed, only the pre-execution checks will be done. Query-ing the
-job result will return, in both dry-run and normal case, the list of
-nodes selected for the instance.
+executed, only the pre-execution checks will be done. Query-ing the job
+result will return, in both dry-run and normal case, the list of nodes
+selected for the instance.

 
 Returns: a job ID that can be used later for polling.
 
 
 Returns: a job ID that can be used later for polling.
 

@@ -423,8 +425,8 @@ It supports the ``dry-run`` argument.
 
 Delete a tag.
 
 
 Delete a tag.
 

-In order to delete a set of tags, the DELETE request should be
-addressed to URI like::
+In order to delete a set of tags, the DELETE request should be addressed
+to URI like::

 
     /tags?tag=[tag]&tag=[tag]
 
 
     /tags?tag=[tag]&tag=[tag]
 

@@ -464,8 +466,8 @@ The result includes:
 
 - id: job ID as a number
 - status: current job status as a string
 
 - id: job ID as a number
 - status: current job status as a string

-- ops: involved OpCodes as a list of dictionaries for each
-  opcodes in the job
+- ops: involved OpCodes as a list of dictionaries for each opcodes in
+  the job

 - opstatus: OpCodes status as a list
 - opresult: OpCodes results as a list of lists
 
 - opstatus: OpCodes status as a list
 - opresult: OpCodes results as a list of lists
 

@@ -499,9 +501,9 @@ Example::
       }
     ]
 
       }
     ]
 

-If the optional 'bulk' argument is provided and set to 'true' value
-(i.e '?bulk=1'), the output contains detailed information about nodes
-as a list.
+If the optional 'bulk' argument is provided and set to 'true' value (i.e
+'?bulk=1'), the output contains detailed information about nodes as a
+list.

 
 Example::
 
 
 Example::
 

@@ -662,8 +664,8 @@ It supports the ``dry-run`` argument.
 
 Deletes tags.
 
 
 Deletes tags.
 

-In order to delete a set of tags, the DELETE request should be
-addressed to URI like::
+In order to delete a set of tags, the DELETE request should be addressed
+to URI like::

 
     /tags?tag=[tag]&tag=[tag]
 
 
     /tags?tag=[tag]&tag=[tag]
 

@@ -683,8 +685,7 @@ It supports the following commands: ``GET``.
 Return a list of all OSes.
 
 Can return error 500 in case of a problem. Since this is a costly
 Return a list of all OSes.
 
 Can return error 500 in case of a problem. Since this is a costly

-operation for Ganeti 2.0, it is not recommended to execute it too
-often.
+operation for Ganeti 2.0, it is not recommended to execute it too often.

 
 Example::
 
 
 Example::
 

@@ -722,8 +723,8 @@ It supports the ``dry-run`` argument.
 
 Deletes tags.
 
 
 Deletes tags.
 

-In order to delete a set of tags, the DELETE request should be
-addressed to URI like::
+In order to delete a set of tags, the DELETE request should be addressed
+to URI like::

 
     /tags?tag=[tag]&tag=[tag]
 
 
     /tags?tag=[tag]&tag=[tag]
 

@@ -735,15 +736,19 @@ It supports the ``dry-run`` argument.
 
 The version resource.
 
 
 The version resource.
 

-This resource should be used to determine the remote API version and
-to adapt clients accordingly.
+This resource should be used to determine the remote API version and to
+adapt clients accordingly.

 
 It supports the following commands: ``GET``.
 
 ``GET``
 ~~~~~~~
 
 
 It supports the following commands: ``GET``.
 
 ``GET``
 ~~~~~~~
 

-Returns the remote API version. Ganeti 1.2 returned ``1`` and Ganeti
-2.0 returns ``2``.
+Returns the remote API version. Ganeti 1.2 returned ``1`` and Ganeti 2.0
+returns ``2``.

 
 .. vim: set textwidth=72 :
 
 .. vim: set textwidth=72 :

+.. Local Variables:
+.. mode: rst
+.. fill-column: 72
+.. End:

diff --git a/doc/security.rst b/doc/security.rst
index 5e15746..e816ed3 100644 (file)
--- a/doc/security.rst
+++ b/doc/security.rst
@@ -100,3 +100,7 @@ will be set at source configure time. Symlinks or command line
 parameters may be used to use different files.
 
 .. vim: set textwidth=72 :
 parameters may be used to use different files.
 
 .. vim: set textwidth=72 :

+.. Local Variables:
+.. mode: rst
+.. fill-column: 72
+.. End:

diff --git a/doc/walkthrough.rst b/doc/walkthrough.rst
new file mode 100644 (file)
index 0000000..b05a677
--- /dev/null
+++ b/doc/walkthrough.rst
@@ -0,0 +1,1030 @@
+Ganeti walk-through
+===================
+
+Documents Ganeti version |version|
+
+.. contents::
+
+.. highlight:: text
+
+Introduction
+------------
+
+This document serves as a more example-oriented guide to Ganeti; while
+the administration guide shows a conceptual approach, here you will find
+a step-by-step example to managing instances and the cluster.
+
+Our simulated, example cluster will have three machines, named
+``node1``, ``node2``, ``node3``. Note that in real life machines will
+usually FQDNs but here we use short names for brevity. We will use a
+secondary network for replication data, ``192.168.2.0/24``, with nodes
+having the last octet the same as their index. The cluster name will be
+``example-cluster``. All nodes have the same simulated hardware
+configuration, two disks of 750GB, 32GB of memory and 4 CPUs.
+
+On this cluster, we will create up to seven instances, named
+``instance1`` to ``instance7``.
+
+
+Cluster creation
+----------------
+
+Follow the :doc:`install` document and prepare the nodes. Then it's time
+to initialise the cluster::
+
+  node1# gnt-cluster init -s 192.168.2.1 --enabled-hypervisors=xen-pvm cluster
+  node1#
+
+The creation was fine. Let's check that one node we have is functioning
+correctly::
+
+  node1# gnt-node list
+  Node  DTotal DFree MTotal MNode MFree Pinst Sinst
+  node1   1.3T  1.3T  32.0G  1.0G 30.5G     0     0
+  node1# gnt-cluster verify
+  Mon Oct 26 02:08:51 2009 * Verifying global settings
+  Mon Oct 26 02:08:51 2009 * Gathering data (1 nodes)
+  Mon Oct 26 02:08:52 2009 * Verifying node status
+  Mon Oct 26 02:08:52 2009 * Verifying instance status
+  Mon Oct 26 02:08:52 2009 * Verifying orphan volumes
+  Mon Oct 26 02:08:52 2009 * Verifying remaining instances
+  Mon Oct 26 02:08:52 2009 * Verifying N+1 Memory redundancy
+  Mon Oct 26 02:08:52 2009 * Other Notes
+  Mon Oct 26 02:08:52 2009 * Hooks Results
+  node1#
+
+Since this proceeded correctly, let's add the other two nodes::
+
+  node1# gnt-node add -s 192.168.2.2 node2
+  -- WARNING --
+  Performing this operation is going to replace the ssh daemon keypair
+  on the target machine (node2) with the ones of the current one
+  and grant full intra-cluster ssh root access to/from it
+
+  The authenticity of host 'node2 (192.168.1.2)' can't be established.
+  RSA key fingerprint is 9f:…
+  Are you sure you want to continue connecting (yes/no)? yes
+  root@node2's password:
+  Mon Oct 26 02:11:54 2009  - INFO: Node will be a master candidate
+  node1# gnt-node add -s 192.168.2.3 node3
+  -- WARNING --
+  Performing this operation is going to replace the ssh daemon keypair
+  on the target machine (node2) with the ones of the current one
+  and grant full intra-cluster ssh root access to/from it
+
+  The authenticity of host 'node3 (192.168.1.3)' can't be established.
+  RSA key fingerprint is 9f:…
+  Are you sure you want to continue connecting (yes/no)? yes
+  root@node2's password:
+  Mon Oct 26 02:11:54 2009  - INFO: Node will be a master candidate
+
+Checking the cluster status again::
+
+  node1# gnt-node list
+  Node  DTotal DFree MTotal MNode MFree Pinst Sinst
+  node1   1.3T  1.3T  32.0G  1.0G 30.5G     0     0
+  node2   1.3T  1.3T  32.0G  1.0G 30.5G     0     0
+  node3   1.3T  1.3T  32.0G  1.0G 30.5G     0     0
+  node1# gnt-cluster verify
+  Mon Oct 26 02:15:14 2009 * Verifying global settings
+  Mon Oct 26 02:15:14 2009 * Gathering data (3 nodes)
+  Mon Oct 26 02:15:16 2009 * Verifying node status
+  Mon Oct 26 02:15:16 2009 * Verifying instance status
+  Mon Oct 26 02:15:16 2009 * Verifying orphan volumes
+  Mon Oct 26 02:15:16 2009 * Verifying remaining instances
+  Mon Oct 26 02:15:16 2009 * Verifying N+1 Memory redundancy
+  Mon Oct 26 02:15:16 2009 * Other Notes
+  Mon Oct 26 02:15:16 2009 * Hooks Results
+  node1#
+
+And let's check that we have a valid OS::
+
+  node1# gnt-os list
+  Name
+  debootstrap
+  node1#
+
+Running a burnin
+----------------
+
+Now that the cluster is created, it is time to check that the hardware
+works correctly, that the hypervisor can actually create instances,
+etc. This is done via the debootstrap tool as described in the admin
+guide. Similar output lines are replaced with ``…`` in the below log::
+
+  node1# /usr/lib/ganeti/tools/burnin -o debootstrap -p instance{1..5}
+  - Testing global parameters
+  - Creating instances
+    * instance instance1
+      on node1, node2
+    * instance instance2
+      on node2, node3
+    …
+    * instance instance5
+      on node2, node3
+    * Submitted job ID(s) 157, 158, 159, 160, 161
+      waiting for job 157 for instance1
+      …
+      waiting for job 161 for instance5
+  - Replacing disks on the same nodes
+    * instance instance1
+      run replace_on_secondary
+      run replace_on_primary
+    …
+    * instance instance5
+      run replace_on_secondary
+      run replace_on_primary
+    * Submitted job ID(s) 162, 163, 164, 165, 166
+      waiting for job 162 for instance1
+      …
+  - Changing the secondary node
+    * instance instance1
+      run replace_new_secondary node3
+    * instance instance2
+      run replace_new_secondary node1
+    …
+    * instance instance5
+      run replace_new_secondary node1
+    * Submitted job ID(s) 167, 168, 169, 170, 171
+      waiting for job 167 for instance1
+      …
+  - Growing disks
+    * instance instance1
+      increase disk/0 by 128 MB
+    …
+    * instance instance5
+      increase disk/0 by 128 MB
+    * Submitted job ID(s) 173, 174, 175, 176, 177
+      waiting for job 173 for instance1
+      …
+  - Failing over instances
+    * instance instance1
+    …
+    * instance instance5
+    * Submitted job ID(s) 179, 180, 181, 182, 183
+      waiting for job 179 for instance1
+      …
+  - Migrating instances
+    * instance instance1
+      migration and migration cleanup
+    …
+    * instance instance5
+      migration and migration cleanup
+    * Submitted job ID(s) 184, 185, 186, 187, 188
+      waiting for job 184 for instance1
+      …
+  - Exporting and re-importing instances
+    * instance instance1
+      export to node node3
+      remove instance
+      import from node3 to node1, node2
+      remove export
+    …
+    * instance instance5
+      export to node node1
+      remove instance
+      import from node1 to node2, node3
+      remove export
+    * Submitted job ID(s) 196, 197, 198, 199, 200
+      waiting for job 196 for instance1
+      …
+  - Reinstalling instances
+    * instance instance1
+      reinstall without passing the OS
+      reinstall specifying the OS
+    …
+    * instance instance5
+      reinstall without passing the OS
+      reinstall specifying the OS
+    * Submitted job ID(s) 203, 204, 205, 206, 207
+      waiting for job 203 for instance1
+      …
+  - Rebooting instances
+    * instance instance1
+      reboot with type 'hard'
+      reboot with type 'soft'
+      reboot with type 'full'
+    …
+    * instance instance5
+      reboot with type 'hard'
+      reboot with type 'soft'
+      reboot with type 'full'
+    * Submitted job ID(s) 208, 209, 210, 211, 212
+      waiting for job 208 for instance1
+    …
+  - Adding and removing disks
+    * instance instance1
+      adding a disk
+      removing last disk
+    …
+    * instance instance5
+      adding a disk
+      removing last disk
+    * Submitted job ID(s) 213, 214, 215, 216, 217
+      waiting for job 213 for instance1
+      …
+  - Adding and removing NICs
+    * instance instance1
+      adding a NIC
+      removing last NIC
+    …
+    * instance instance5
+      adding a NIC
+      removing last NIC
+    * Submitted job ID(s) 218, 219, 220, 221, 222
+      waiting for job 218 for instance1
+      …
+  - Activating/deactivating disks
+    * instance instance1
+      activate disks when online
+      activate disks when offline
+      deactivate disks (when offline)
+    …
+    * instance instance5
+      activate disks when online
+      activate disks when offline
+      deactivate disks (when offline)
+    * Submitted job ID(s) 223, 224, 225, 226, 227
+      waiting for job 223 for instance1
+      …
+  - Stopping and starting instances
+    * instance instance1
+    …
+    * instance instance5
+    * Submitted job ID(s) 230, 231, 232, 233, 234
+      waiting for job 230 for instance1
+      …
+  - Removing instances
+    * instance instance1
+    …
+    * instance instance5
+    * Submitted job ID(s) 235, 236, 237, 238, 239
+      waiting for job 235 for instance1
+      …
+  node1#
+
+You can see in the above what operations the burnin does. Ideally, the
+burnin log would proceed successfully through all the steps and end
+cleanly, without throwing errors.
+
+Instance operations
+-------------------
+
+Creation
+++++++++
+
+At this point, Ganeti and the hardware seems to be functioning
+correctly, so we'll follow up with creating the instances manually::
+
+  node1# gnt-instance add -t drbd -o debootstrap -s 256m -n node1:node2 instance3
+  Mon Oct 26 04:06:52 2009  - INFO: Selected nodes for instance instance1 via iallocator hail: node2, node3
+  Mon Oct 26 04:06:53 2009 * creating instance disks...
+  Mon Oct 26 04:06:57 2009 adding instance instance1 to cluster config
+  Mon Oct 26 04:06:57 2009  - INFO: Waiting for instance instance1 to sync disks.
+  Mon Oct 26 04:06:57 2009  - INFO: - device disk/0: 20.00% done, 4 estimated seconds remaining
+  Mon Oct 26 04:07:01 2009  - INFO: Instance instance1's disks are in sync.
+  Mon Oct 26 04:07:01 2009 creating os for instance instance1 on node node2
+  Mon Oct 26 04:07:01 2009 * running the instance OS create scripts...
+  Mon Oct 26 04:07:14 2009 * starting instance...
+  node1# gnt-instance add -t drbd -o debootstrap -s 256m -n node1:node2 instanc<drbd -o debootstrap -s 256m -n node1:node2 instance2
+  Mon Oct 26 04:11:37 2009 * creating instance disks...
+  Mon Oct 26 04:11:40 2009 adding instance instance2 to cluster config
+  Mon Oct 26 04:11:41 2009  - INFO: Waiting for instance instance2 to sync disks.
+  Mon Oct 26 04:11:41 2009  - INFO: - device disk/0: 35.40% done, 1 estimated seconds remaining
+  Mon Oct 26 04:11:42 2009  - INFO: - device disk/0: 58.50% done, 1 estimated seconds remaining
+  Mon Oct 26 04:11:43 2009  - INFO: - device disk/0: 86.20% done, 0 estimated seconds remaining
+  Mon Oct 26 04:11:44 2009  - INFO: - device disk/0: 92.40% done, 0 estimated seconds remaining
+  Mon Oct 26 04:11:44 2009  - INFO: - device disk/0: 97.00% done, 0 estimated seconds remaining
+  Mon Oct 26 04:11:44 2009  - INFO: Instance instance2's disks are in sync.
+  Mon Oct 26 04:11:44 2009 creating os for instance instance2 on node node1
+  Mon Oct 26 04:11:44 2009 * running the instance OS create scripts...
+  Mon Oct 26 04:11:57 2009 * starting instance...
+  node1#
+
+The above shows one instance created via an iallocator script, and one
+being created with manual node assignment. The other three instances
+were also created and now it's time to check them::
+
+  node1# gnt-instance list
+  Instance  Hypervisor OS          Primary_node Status  Memory
+  instance1 xen-pvm    debootstrap node2        running   128M
+  instance2 xen-pvm    debootstrap node1        running   128M
+  instance3 xen-pvm    debootstrap node1        running   128M
+  instance4 xen-pvm    debootstrap node3        running   128M
+  instance5 xen-pvm    debootstrap node2        running   128M
+
+Accessing instances
++++++++++++++++++++
+
+Accessing an instance's console is easy::
+
+  node1# gnt-instance console instance2
+  [    0.000000] Bootdata ok (command line is root=/dev/sda1 ro)
+  [    0.000000] Linux version 2.6…
+  [    0.000000] BIOS-provided physical RAM map:
+  [    0.000000]  Xen: 0000000000000000 - 0000000008800000 (usable)
+  [13138176.018071] Built 1 zonelists.  Total pages: 34816
+  [13138176.018074] Kernel command line: root=/dev/sda1 ro
+  [13138176.018694] Initializing CPU#0
+  …
+  Checking file systems...fsck 1.41.3 (12-Oct-2008)
+  done.
+  Setting kernel variables (/etc/sysctl.conf)...done.
+  Mounting local filesystems...done.
+  Activating swapfile swap...done.
+  Setting up networking....
+  Configuring network interfaces...done.
+  Setting console screen modes and fonts.
+  INIT: Entering runlevel: 2
+  Starting enhanced syslogd: rsyslogd.
+  Starting periodic command scheduler: crond.
+
+  Debian GNU/Linux 5.0 instance2 tty1
+
+  instance2 login:
+
+At this moment you can login to the instance and, after configuring the
+network (and doing this on all instances), we can check their
+connectivity::
+
+  node1# fping instance{1..5}
+  instance1 is alive
+  instance2 is alive
+  instance3 is alive
+  instance4 is alive
+  instance5 is alive
+  node1#
+
+Removal
++++++++
+
+Removing unwanted instances is also easy::
+
+  node1# gnt-instance remove instance5
+  This will remove the volumes of the instance instance5 (including
+  mirrors), thus removing all the data of the instance. Continue?
+  y/[n]/?: y
+  node1#
+
+
+Recovering from hardware failures
+---------------------------------
+
+Recovering from node failure
+++++++++++++++++++++++++++++
+
+We are now left with four instances. Assume that at this point, node3,
+which has one primary and one secondary instance, crashes::
+
+  node1# gnt-node info node3
+  Node name: node3
+    primary ip: 172.24.227.1
+    secondary ip: 192.168.2.3
+    master candidate: True
+    drained: False
+    offline: False
+    primary for instances:
+      - instance4
+    secondary for instances:
+      - instance1
+  node1# fping node3
+  node3 is unreachable
+
+At this point, the primary instance of that node (instance4) is down,
+but the secondary instance (instance1) is not affected except it has
+lost disk redundancy::
+
+  node1# fping instance{1,4}
+  instance1 is alive
+  instance4 is unreachable
+  node1#
+
+If we try to check the status of instance4 via the instance info
+command, it fails because it tries to contact node3 which is down::
+
+  node1# gnt-instance info instance4
+  Failure: command execution error:
+  Error checking node node3: Connection failed (113: No route to host)
+  node1#
+
+So we need to mark node3 as being *offline*, and thus Ganeti won't talk
+to it anymore::
+
+  node1# gnt-node modify -O yes -f node3
+  Mon Oct 26 04:34:12 2009  - WARNING: Not enough master candidates (desired 10, new value will be 2)
+  Mon Oct 26 04:34:15 2009  - WARNING: Communication failure to node node3: Connection failed (113: No route to host)
+  Modified node node3
+   - offline -> True
+   - master_candidate -> auto-demotion due to offline
+  node1#
+
+And now we can failover the instance::
+
+  node1# gnt-instance failover --ignore-consistency instance4
+  Failover will happen to image instance4. This requires a shutdown of
+  the instance. Continue?
+  y/[n]/?: y
+  Mon Oct 26 04:35:34 2009 * checking disk consistency between source and target
+  Failure: command execution error:
+  Disk disk/0 is degraded on target node, aborting failover.
+  node1# gnt-instance failover --ignore-consistency instance4
+  Failover will happen to image instance4. This requires a shutdown of
+  the instance. Continue?
+  y/[n]/?: y
+  Mon Oct 26 04:35:47 2009 * checking disk consistency between source and target
+  Mon Oct 26 04:35:47 2009 * shutting down instance on source node
+  Mon Oct 26 04:35:47 2009  - WARNING: Could not shutdown instance instance4 on node node3. Proceeding anyway. Please make sure node node3 is down. Error details: Node is marked offline
+  Mon Oct 26 04:35:47 2009 * deactivating the instance's disks on source node
+  Mon Oct 26 04:35:47 2009  - WARNING: Could not shutdown block device disk/0 on node node3: Node is marked offline
+  Mon Oct 26 04:35:47 2009 * activating the instance's disks on target node
+  Mon Oct 26 04:35:47 2009  - WARNING: Could not prepare block device disk/0 on node node3 (is_primary=False, pass=1): Node is marked offline
+  Mon Oct 26 04:35:48 2009 * starting the instance on the target node
+  node1#
+
+Note in our first attempt, Ganeti refused to do the failover since it
+wasn't sure what is the status of the instance's disks. We pass the
+``--ignore-consistency`` flag and then we can failover::
+
+  node1# gnt-instance list
+  Instance  Hypervisor OS          Primary_node Status  Memory
+  instance1 xen-pvm    debootstrap node2        running   128M
+  instance2 xen-pvm    debootstrap node1        running   128M
+  instance3 xen-pvm    debootstrap node1        running   128M
+  instance4 xen-pvm    debootstrap node1        running   128M
+  node1#
+
+But at this point, both instance1 and instance4 are without disk
+redundancy::
+
+  node1# gnt-instance info instance1
+  Instance name: instance1
+  UUID: 45173e82-d1fa-417c-8758-7d582ab7eef4
+  Serial number: 2
+  Creation time: 2009-10-26 04:06:57
+  Modification time: 2009-10-26 04:07:14
+  State: configured to be up, actual state is up
+    Nodes:
+      - primary: node2
+      - secondaries: node3
+    Operating system: debootstrap
+    Allocated network port: None
+    Hypervisor: xen-pvm
+      - root_path: default (/dev/sda1)
+      - kernel_args: default (ro)
+      - use_bootloader: default (False)
+      - bootloader_args: default ()
+      - bootloader_path: default ()
+      - kernel_path: default (/boot/vmlinuz-2.6-xenU)
+      - initrd_path: default ()
+    Hardware:
+      - VCPUs: 1
+      - memory: 128MiB
+      - NICs:
+        - nic/0: MAC: aa:00:00:78:da:63, IP: None, mode: bridged, link: xen-br0
+    Disks:
+      - disk/0: drbd8, size 256M
+        access mode: rw
+        nodeA:       node2, minor=0
+        nodeB:       node3, minor=0
+        port:        11035
+        auth key:    8e950e3cec6854b0181fbc3a6058657701f2d458
+        on primary:  /dev/drbd0 (147:0) in sync, status *DEGRADED*
+        child devices:
+          - child 0: lvm, size 256M
+            logical_id: xenvg/22459cf8-117d-4bea-a1aa-791667d07800.disk0_data
+            on primary: /dev/xenvg/22459cf8-117d-4bea-a1aa-791667d07800.disk0_data (254:0)
+          - child 1: lvm, size 128M
+            logical_id: xenvg/22459cf8-117d-4bea-a1aa-791667d07800.disk0_meta
+            on primary: /dev/xenvg/22459cf8-117d-4bea-a1aa-791667d07800.disk0_meta (254:1)
+
+The output is similar for instance4. In order to recover this, we need
+to run the node evacuate command which will change from the current
+secondary node to a new one (in this case, we only have two working
+nodes, so all instances will be end on nodes one and two)::
+
+  node1# gnt-node evacuate -I hail node3
+  Relocate instance(s) 'instance1','instance4' from node
+   node3 using iallocator hail?
+  y/[n]/?: y
+  Mon Oct 26 05:05:39 2009  - INFO: Selected new secondary for instance 'instance1': node1
+  Mon Oct 26 05:05:40 2009  - INFO: Selected new secondary for instance 'instance4': node2
+  Mon Oct 26 05:05:40 2009 Replacing disk(s) 0 for instance1
+  Mon Oct 26 05:05:40 2009 STEP 1/6 Check device existence
+  Mon Oct 26 05:05:40 2009  - INFO: Checking disk/0 on node2
+  Mon Oct 26 05:05:40 2009  - INFO: Checking volume groups
+  Mon Oct 26 05:05:40 2009 STEP 2/6 Check peer consistency
+  Mon Oct 26 05:05:40 2009  - INFO: Checking disk/0 consistency on node node2
+  Mon Oct 26 05:05:40 2009 STEP 3/6 Allocate new storage
+  Mon Oct 26 05:05:40 2009  - INFO: Adding new local storage on node1 for disk/0
+  Mon Oct 26 05:05:41 2009 STEP 4/6 Changing drbd configuration
+  Mon Oct 26 05:05:41 2009  - INFO: activating a new drbd on node1 for disk/0
+  Mon Oct 26 05:05:42 2009  - INFO: Shutting down drbd for disk/0 on old node
+  Mon Oct 26 05:05:42 2009  - WARNING: Failed to shutdown drbd for disk/0 on oldnode: Node is marked offline
+  Mon Oct 26 05:05:42 2009       Hint: Please cleanup this device manually as soon as possible
+  Mon Oct 26 05:05:42 2009  - INFO: Detaching primary drbds from the network (=> standalone)
+  Mon Oct 26 05:05:42 2009  - INFO: Updating instance configuration
+  Mon Oct 26 05:05:45 2009  - INFO: Attaching primary drbds to new secondary (standalone => connected)
+  Mon Oct 26 05:05:46 2009 STEP 5/6 Sync devices
+  Mon Oct 26 05:05:46 2009  - INFO: Waiting for instance instance1 to sync disks.
+  Mon Oct 26 05:05:46 2009  - INFO: - device disk/0: 13.90% done, 7 estimated seconds remaining
+  Mon Oct 26 05:05:53 2009  - INFO: Instance instance1's disks are in sync.
+  Mon Oct 26 05:05:53 2009 STEP 6/6 Removing old storage
+  Mon Oct 26 05:05:53 2009  - INFO: Remove logical volumes for 0
+  Mon Oct 26 05:05:53 2009  - WARNING: Can't remove old LV: Node is marked offline
+  Mon Oct 26 05:05:53 2009       Hint: remove unused LVs manually
+  Mon Oct 26 05:05:53 2009  - WARNING: Can't remove old LV: Node is marked offline
+  Mon Oct 26 05:05:53 2009       Hint: remove unused LVs manually
+  Mon Oct 26 05:05:53 2009 Replacing disk(s) 0 for instance4
+  Mon Oct 26 05:05:53 2009 STEP 1/6 Check device existence
+  Mon Oct 26 05:05:53 2009  - INFO: Checking disk/0 on node1
+  Mon Oct 26 05:05:53 2009  - INFO: Checking volume groups
+  Mon Oct 26 05:05:53 2009 STEP 2/6 Check peer consistency
+  Mon Oct 26 05:05:53 2009  - INFO: Checking disk/0 consistency on node node1
+  Mon Oct 26 05:05:54 2009 STEP 3/6 Allocate new storage
+  Mon Oct 26 05:05:54 2009  - INFO: Adding new local storage on node2 for disk/0
+  Mon Oct 26 05:05:54 2009 STEP 4/6 Changing drbd configuration
+  Mon Oct 26 05:05:54 2009  - INFO: activating a new drbd on node2 for disk/0
+  Mon Oct 26 05:05:55 2009  - INFO: Shutting down drbd for disk/0 on old node
+  Mon Oct 26 05:05:55 2009  - WARNING: Failed to shutdown drbd for disk/0 on oldnode: Node is marked offline
+  Mon Oct 26 05:05:55 2009       Hint: Please cleanup this device manually as soon as possible
+  Mon Oct 26 05:05:55 2009  - INFO: Detaching primary drbds from the network (=> standalone)
+  Mon Oct 26 05:05:55 2009  - INFO: Updating instance configuration
+  Mon Oct 26 05:05:55 2009  - INFO: Attaching primary drbds to new secondary (standalone => connected)
+  Mon Oct 26 05:05:56 2009 STEP 5/6 Sync devices
+  Mon Oct 26 05:05:56 2009  - INFO: Waiting for instance instance4 to sync disks.
+  Mon Oct 26 05:05:56 2009  - INFO: - device disk/0: 12.40% done, 8 estimated seconds remaining
+  Mon Oct 26 05:06:04 2009  - INFO: Instance instance4's disks are in sync.
+  Mon Oct 26 05:06:04 2009 STEP 6/6 Removing old storage
+  Mon Oct 26 05:06:04 2009  - INFO: Remove logical volumes for 0
+  Mon Oct 26 05:06:04 2009  - WARNING: Can't remove old LV: Node is marked offline
+  Mon Oct 26 05:06:04 2009       Hint: remove unused LVs manually
+  Mon Oct 26 05:06:04 2009  - WARNING: Can't remove old LV: Node is marked offline
+  Mon Oct 26 05:06:04 2009       Hint: remove unused LVs manually
+  node1#
+
+And now node3 is completely free of instances and can be repaired::
+
+  node1# gnt-node list
+  Node  DTotal DFree MTotal MNode MFree Pinst Sinst
+  node1   1.3T  1.3T  32.0G  1.0G 30.2G     3     1
+  node2   1.3T  1.3T  32.0G  1.0G 30.4G     1     3
+  node3      ?     ?      ?     ?     ?     0     0
+
+Re-adding a node to the cluster
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+
+Let's say node3 has been repaired and is now ready to be
+reused. Re-adding it is simple::
+
+  node1# gnt-node add --readd node3
+  The authenticity of host 'node3 (172.24.227.1)' can't be established.
+  RSA key fingerprint is 9f:2e:5a:2e:e0:bd:00:09:e4:5c:32:f2:27:57:7a:f4.
+  Are you sure you want to continue connecting (yes/no)? yes
+  Mon Oct 26 05:27:39 2009  - INFO: Readding a node, the offline/drained flags were reset
+  Mon Oct 26 05:27:39 2009  - INFO: Node will be a master candidate
+
+And is now working again::
+
+  node1# gnt-node list
+  Node  DTotal DFree MTotal MNode MFree Pinst Sinst
+  node1   1.3T  1.3T  32.0G  1.0G 30.2G     3     1
+  node2   1.3T  1.3T  32.0G  1.0G 30.4G     1     3
+  node3   1.3T  1.3T  32.0G  1.0G 30.4G     0     0
+
+.. note:: If you have the ganeti-htools package installed, you can
+   shuffle the instances around to have a better use of the nodes.
+
+Disk failures
++++++++++++++
+
+A disk failure is simpler than a full node failure. First, a single disk
+failure should not cause data-loss for any redundant instance; only the
+performance of some instances might be reduced due to more network
+traffic.
+
+Let take the cluster status in the above listing, and check what volumes
+are in use::
+
+  node1# gnt-node volumes -o phys,instance node2
+  PhysDev   Instance
+  /dev/sdb1 instance4
+  /dev/sdb1 instance4
+  /dev/sdb1 instance1
+  /dev/sdb1 instance1
+  /dev/sdb1 instance3
+  /dev/sdb1 instance3
+  /dev/sdb1 instance2
+  /dev/sdb1 instance2
+  node1#
+
+You can see that all instances on node2 have logical volumes on
+``/dev/sdb1``. Let's simulate a disk failure on that disk::
+
+  node1# ssh node2
+  node2# echo offline > /sys/block/sdb/device/state
+  node2# vgs
+    /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
+    /dev/sdb1: read failed after 0 of 4096 at 750153695232: Input/output error
+    /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
+    Couldn't find device with uuid '954bJA-mNL0-7ydj-sdpW-nc2C-ZrCi-zFp91c'.
+    Couldn't find all physical volumes for volume group xenvg.
+    /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
+    /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
+    Couldn't find device with uuid '954bJA-mNL0-7ydj-sdpW-nc2C-ZrCi-zFp91c'.
+    Couldn't find all physical volumes for volume group xenvg.
+    Volume group xenvg not found
+  node2#
+
+At this point, the node is broken and if we are to examine
+instance2 we get (simplified output shown)::
+
+  node1# gnt-instance info instance2
+  Instance name: instance2
+  State: configured to be up, actual state is up
+    Nodes:
+      - primary: node1
+      - secondaries: node2
+    Disks:
+      - disk/0: drbd8, size 256M
+        on primary:   /dev/drbd0 (147:0) in sync, status ok
+        on secondary: /dev/drbd1 (147:1) in sync, status *DEGRADED* *MISSING DISK*
+
+This instance has a secondary only on node2. Let's verify a primary
+instance of node2::
+
+  node1# gnt-instance info instance1
+  Instance name: instance1
+  State: configured to be up, actual state is up
+    Nodes:
+      - primary: node2
+      - secondaries: node1
+    Disks:
+      - disk/0: drbd8, size 256M
+        on primary:   /dev/drbd0 (147:0) in sync, status *DEGRADED* *MISSING DISK*
+        on secondary: /dev/drbd3 (147:3) in sync, status ok
+  node1# gnt-instance console instance1
+
+  Debian GNU/Linux 5.0 instance1 tty1
+
+  instance1 login: root
+  Last login: Tue Oct 27 01:24:09 UTC 2009 on tty1
+  instance1:~# date > test
+  instance1:~# sync
+  instance1:~# cat test
+  Tue Oct 27 01:25:20 UTC 2009
+  instance1:~# dmesg|tail
+  [5439785.235448] NET: Registered protocol family 15
+  [5439785.235489] 802.1Q VLAN Support v1.8 Ben Greear <greearb@candelatech.com>
+  [5439785.235495] All bugs added by David S. Miller <davem@redhat.com>
+  [5439785.235517] XENBUS: Device with no driver: device/console/0
+  [5439785.236576] kjournald starting.  Commit interval 5 seconds
+  [5439785.236588] EXT3-fs: mounted filesystem with ordered data mode.
+  [5439785.236625] VFS: Mounted root (ext3 filesystem) readonly.
+  [5439785.236663] Freeing unused kernel memory: 172k freed
+  [5439787.533779] EXT3 FS on sda1, internal journal
+  [5440655.065431] eth0: no IPv6 routers present
+  instance1:~#
+
+As you can see, the instance is running fine and doesn't see any disk
+issues. It is now time to fix node2 and re-establish redundancy for the
+involved instances.
+
+.. note:: For Ganeti 2.0 we need to fix manually the volume group on
+   node2 by running ``vgreduce --removemissing xenvg``
+
+::
+
+  node1# gnt-node repair-storage node2 lvm-vg xenvg
+  Mon Oct 26 18:14:03 2009 Repairing storage unit 'xenvg' on node2 ...
+  node1# ssh node2 vgs
+    VG    #PV #LV #SN Attr   VSize   VFree
+    xenvg   1   8   0 wz--n- 673.84G 673.84G
+  node1#
+
+This has removed the 'bad' disk from the volume group, which is now left
+with only one PV. We can now replace the disks for the involved
+instances::
+
+  node1# for i in instance{1..4}; do gnt-instance replace-disks -a $i; done
+  Mon Oct 26 18:15:38 2009 Replacing disk(s) 0 for instance1
+  Mon Oct 26 18:15:38 2009 STEP 1/6 Check device existence
+  Mon Oct 26 18:15:38 2009  - INFO: Checking disk/0 on node1
+  Mon Oct 26 18:15:38 2009  - INFO: Checking disk/0 on node2
+  Mon Oct 26 18:15:38 2009  - INFO: Checking volume groups
+  Mon Oct 26 18:15:38 2009 STEP 2/6 Check peer consistency
+  Mon Oct 26 18:15:38 2009  - INFO: Checking disk/0 consistency on node node1
+  Mon Oct 26 18:15:39 2009 STEP 3/6 Allocate new storage
+  Mon Oct 26 18:15:39 2009  - INFO: Adding storage on node2 for disk/0
+  Mon Oct 26 18:15:39 2009 STEP 4/6 Changing drbd configuration
+  Mon Oct 26 18:15:39 2009  - INFO: Detaching disk/0 drbd from local storage
+  Mon Oct 26 18:15:40 2009  - INFO: Renaming the old LVs on the target node
+  Mon Oct 26 18:15:40 2009  - INFO: Renaming the new LVs on the target node
+  Mon Oct 26 18:15:40 2009  - INFO: Adding new mirror component on node2
+  Mon Oct 26 18:15:41 2009 STEP 5/6 Sync devices
+  Mon Oct 26 18:15:41 2009  - INFO: Waiting for instance instance1 to sync disks.
+  Mon Oct 26 18:15:41 2009  - INFO: - device disk/0: 12.40% done, 9 estimated seconds remaining
+  Mon Oct 26 18:15:50 2009  - INFO: Instance instance1's disks are in sync.
+  Mon Oct 26 18:15:50 2009 STEP 6/6 Removing old storage
+  Mon Oct 26 18:15:50 2009  - INFO: Remove logical volumes for disk/0
+  Mon Oct 26 18:15:52 2009 Replacing disk(s) 0 for instance2
+  Mon Oct 26 18:15:52 2009 STEP 1/6 Check device existence
+  …
+  Mon Oct 26 18:16:01 2009 STEP 6/6 Removing old storage
+  Mon Oct 26 18:16:01 2009  - INFO: Remove logical volumes for disk/0
+  Mon Oct 26 18:16:02 2009 Replacing disk(s) 0 for instance3
+  Mon Oct 26 18:16:02 2009 STEP 1/6 Check device existence
+  …
+  Mon Oct 26 18:16:09 2009 STEP 6/6 Removing old storage
+  Mon Oct 26 18:16:09 2009  - INFO: Remove logical volumes for disk/0
+  Mon Oct 26 18:16:10 2009 Replacing disk(s) 0 for instance4
+  Mon Oct 26 18:16:10 2009 STEP 1/6 Check device existence
+  …
+  Mon Oct 26 18:16:18 2009 STEP 6/6 Removing old storage
+  Mon Oct 26 18:16:18 2009  - INFO: Remove logical volumes for disk/0
+  node1#
+
+As this point, all instances should be healthy again.
+
+.. note:: Ganeti 2.0 doesn't have the ``-a`` option to replace-disks, so
+   for it you have to run the loop twice, once over primary instances
+   with argument ``-p`` and once secondary instances with argument
+   ``-s``, but otherwise the operations are similar::
+
+     node1# gnt-instance replace-disks -p instance1
+     …
+     node1# for i in instance{2..4}; do gnt-instance replace-disks -s $i; done
+
+Common cluster problems
+-----------------------
+
+There are a number of small issues that might appear on a cluster that
+can be solved easily as long as the issue is properly identified. For
+this exercise we will consider the case of node3, which was broken
+previously and re-added to the cluster without reinstallation. Running
+cluster verify on the cluster reports::
+
+  node1# gnt-cluster verify
+  Mon Oct 26 18:30:08 2009 * Verifying global settings
+  Mon Oct 26 18:30:08 2009 * Gathering data (3 nodes)
+  Mon Oct 26 18:30:10 2009 * Verifying node status
+  Mon Oct 26 18:30:10 2009   - ERROR: node node3: unallocated drbd minor 0 is in use
+  Mon Oct 26 18:30:10 2009   - ERROR: node node3: unallocated drbd minor 1 is in use
+  Mon Oct 26 18:30:10 2009 * Verifying instance status
+  Mon Oct 26 18:30:10 2009   - ERROR: instance instance4: instance should not run on node node3
+  Mon Oct 26 18:30:10 2009 * Verifying orphan volumes
+  Mon Oct 26 18:30:10 2009   - ERROR: node node3: volume 22459cf8-117d-4bea-a1aa-791667d07800.disk0_data is unknown
+  Mon Oct 26 18:30:10 2009   - ERROR: node node3: volume 1aaf4716-e57f-4101-a8d6-03af5da9dc50.disk0_data is unknown
+  Mon Oct 26 18:30:10 2009   - ERROR: node node3: volume 1aaf4716-e57f-4101-a8d6-03af5da9dc50.disk0_meta is unknown
+  Mon Oct 26 18:30:10 2009   - ERROR: node node3: volume 22459cf8-117d-4bea-a1aa-791667d07800.disk0_meta is unknown
+  Mon Oct 26 18:30:10 2009 * Verifying remaining instances
+  Mon Oct 26 18:30:10 2009 * Verifying N+1 Memory redundancy
+  Mon Oct 26 18:30:10 2009 * Other Notes
+  Mon Oct 26 18:30:10 2009 * Hooks Results
+  node1#
+
+Instance status
++++++++++++++++
+
+As you can see, *instance4* has a copy running on node3, because we
+forced the failover when node3 failed. This case is dangerous as the
+instance will have the same IP and MAC address, wreaking havok on the
+network environment and anyone who tries to use it.
+
+Ganeti doesn't directly handle this case. It is recommended to logon to
+node3 and run::
+
+  node3# xm destroy instance4
+
+Unallocated DRBD minors
++++++++++++++++++++++++
+
+There are still unallocated DRBD minors on node3. Again, these are not
+handled by Ganeti directly and need to be cleaned up via DRBD commands::
+
+  node3# drbdsetup /dev/drbd0 down
+  node3# drbdsetup /dev/drbd1 down
+  node3#
+
+Orphan volumes
+++++++++++++++
+
+At this point, the only remaining problem should be the so-called
+*orphan* volumes. This can happen also in the case of an aborted
+disk-replace, or similar situation where Ganeti was not able to recover
+automatically. Here you need to remove them manually via LVM commands::
+
+  node3# lvremove xenvg
+  Do you really want to remove active logical volume "22459cf8-117d-4bea-a1aa-791667d07800.disk0_data"? [y/n]: y
+    Logical volume "22459cf8-117d-4bea-a1aa-791667d07800.disk0_data" successfully removed
+  Do you really want to remove active logical volume "22459cf8-117d-4bea-a1aa-791667d07800.disk0_meta"? [y/n]: y
+    Logical volume "22459cf8-117d-4bea-a1aa-791667d07800.disk0_meta" successfully removed
+  Do you really want to remove active logical volume "1aaf4716-e57f-4101-a8d6-03af5da9dc50.disk0_data"? [y/n]: y
+    Logical volume "1aaf4716-e57f-4101-a8d6-03af5da9dc50.disk0_data" successfully removed
+  Do you really want to remove active logical volume "1aaf4716-e57f-4101-a8d6-03af5da9dc50.disk0_meta"? [y/n]: y
+    Logical volume "1aaf4716-e57f-4101-a8d6-03af5da9dc50.disk0_meta" successfully removed
+  node3#
+
+At this point cluster verify shouldn't complain anymore::
+
+  node1# gnt-cluster verify
+  Mon Oct 26 18:37:51 2009 * Verifying global settings
+  Mon Oct 26 18:37:51 2009 * Gathering data (3 nodes)
+  Mon Oct 26 18:37:53 2009 * Verifying node status
+  Mon Oct 26 18:37:53 2009 * Verifying instance status
+  Mon Oct 26 18:37:53 2009 * Verifying orphan volumes
+  Mon Oct 26 18:37:53 2009 * Verifying remaining instances
+  Mon Oct 26 18:37:53 2009 * Verifying N+1 Memory redundancy
+  Mon Oct 26 18:37:53 2009 * Other Notes
+  Mon Oct 26 18:37:53 2009 * Hooks Results
+  node1#
+
+N+1 errors
+++++++++++
+
+Since redundant instances in Ganeti have a primary/secondary model, it
+is needed to leave aside on each node enough memory so that if one of
+its peer node fails, all the secondary instances that have that node as
+primary can be relocated. More specifically, if instance2 has node1 as
+primary and node2 as secondary (and node1 and node2 do not have any
+other instances in this layout), then it means that node2 must have
+enough free memory so that if node1 fails, we can failover instance2
+without any other operations (for reducing the downtime window). Let's
+increase the memory of the current instances to 4G, and add three new
+instances, two on node2:node3 with 8GB of RAM and one on node1:node2,
+with 12GB of RAM (numbers chosen so that we run out of memory)::
+
+  node1# gnt-instance modify -B memory=4G instance1
+  Modified instance instance1
+   - be/memory -> 4096
+  Please don't forget that these parameters take effect only at the next start of the instance.
+  node1# gnt-instance modify …
+
+  node1# gnt-instance add -t drbd -n node2:node3 -s 512m -B memory=8G -o debootstrap instance5
+  …
+  node1# gnt-instance add -t drbd -n node2:node3 -s 512m -B memory=8G -o debootstrap instance6
+  …
+  node1# gnt-instance add -t drbd -n node1:node2 -s 512m -B memory=8G -o debootstrap instance7
+  node1# gnt-instance reboot --all
+  The reboot will operate on 7 instances.
+  Do you want to continue?
+  Affected instances:
+    instance1
+    instance2
+    instance3
+    instance4
+    instance5
+    instance6
+    instance7
+  y/[n]/?: y
+  Submitted jobs 677, 678, 679, 680, 681, 682, 683
+  Waiting for job 677 for instance1...
+  Waiting for job 678 for instance2...
+  Waiting for job 679 for instance3...
+  Waiting for job 680 for instance4...
+  Waiting for job 681 for instance5...
+  Waiting for job 682 for instance6...
+  Waiting for job 683 for instance7...
+  node1#
+
+We rebooted instances for the memory changes to have effect. Now the
+cluster looks like::
+
+  node1# gnt-node list
+  Node  DTotal DFree MTotal MNode MFree Pinst Sinst
+  node1   1.3T  1.3T  32.0G  1.0G  6.5G     4     1
+  node2   1.3T  1.3T  32.0G  1.0G 10.5G     3     4
+  node3   1.3T  1.3T  32.0G  1.0G 30.5G     0     2
+  node1# gnt-cluster verify
+  Mon Oct 26 18:59:36 2009 * Verifying global settings
+  Mon Oct 26 18:59:36 2009 * Gathering data (3 nodes)
+  Mon Oct 26 18:59:37 2009 * Verifying node status
+  Mon Oct 26 18:59:37 2009 * Verifying instance status
+  Mon Oct 26 18:59:37 2009 * Verifying orphan volumes
+  Mon Oct 26 18:59:37 2009 * Verifying remaining instances
+  Mon Oct 26 18:59:37 2009 * Verifying N+1 Memory redundancy
+  Mon Oct 26 18:59:37 2009   - ERROR: node node2: not enough memory on to accommodate failovers should peer node node1 fail
+  Mon Oct 26 18:59:37 2009 * Other Notes
+  Mon Oct 26 18:59:37 2009 * Hooks Results
+  node1#
+
+The cluster verify error above shows that if node1 fails, node2 will not
+have enough memory to failover all primary instances on node1 to it. To
+solve this, you have a number of options:
+
+- try to manually move instances around (but this can become complicated
+  for any non-trivial cluster)
+- try to reduce memory of some instances to accommodate the available
+  node memory
+- if you have the ganeti-htools package installed, you can run the
+  ``hbal`` tool which will try to compute an automated cluster solution
+  that complies with the N+1 rule
+
+Network issues
+++++++++++++++
+
+In case a node has problems with the network (usually the secondary
+network, as problems with the primary network will render the node
+unusable for ganeti commands), it will show up in cluster verify as::
+
+  node1# gnt-cluster verify
+  Mon Oct 26 19:07:19 2009 * Verifying global settings
+  Mon Oct 26 19:07:19 2009 * Gathering data (3 nodes)
+  Mon Oct 26 19:07:23 2009 * Verifying node status
+  Mon Oct 26 19:07:23 2009   - ERROR: node node1: tcp communication with node 'node3': failure using the secondary interface(s)
+  Mon Oct 26 19:07:23 2009   - ERROR: node node2: tcp communication with node 'node3': failure using the secondary interface(s)
+  Mon Oct 26 19:07:23 2009   - ERROR: node node3: tcp communication with node 'node1': failure using the secondary interface(s)
+  Mon Oct 26 19:07:23 2009   - ERROR: node node3: tcp communication with node 'node2': failure using the secondary interface(s)
+  Mon Oct 26 19:07:23 2009   - ERROR: node node3: tcp communication with node 'node3': failure using the secondary interface(s)
+  Mon Oct 26 19:07:23 2009 * Verifying instance status
+  Mon Oct 26 19:07:23 2009 * Verifying orphan volumes
+  Mon Oct 26 19:07:23 2009 * Verifying remaining instances
+  Mon Oct 26 19:07:23 2009 * Verifying N+1 Memory redundancy
+  Mon Oct 26 19:07:23 2009 * Other Notes
+  Mon Oct 26 19:07:23 2009 * Hooks Results
+  node1#
+
+This shows that both node1 and node2 have problems contacting node3 over
+the secondary network, and node3 has problems contacting them. From this
+output is can be deduced that since node1 and node2 can communicate
+between themselves, node3 is the one having problems, and you need to
+investigate its network settings/connection.
+
+Migration problems
+++++++++++++++++++
+
+Since live migration can sometimes fail and leave the instance in an
+inconsistent state, Ganeti provides a ``--cleanup`` argument to the
+migrate command that does:
+
+- check on which node the instance is actually running (has the
+  command failed before or after the actual migration?)
+- reconfigure the DRBD disks accordingly
+
+It is always safe to run this command as long as the instance has good
+data on its primary node (i.e. not showing as degraded). If so, you can
+simply run::
+
+  node1# gnt-instance migrate --cleanup instance1
+  Instance instance1 will be recovered from a failed migration. Note
+  that the migration procedure (including cleanup) is **experimental**
+  in this version. This might impact the instance if anything goes
+  wrong. Continue?
+  y/[n]/?: y
+  Mon Oct 26 19:13:49 2009 Migrating instance instance1
+  Mon Oct 26 19:13:49 2009 * checking where the instance actually runs (if this hangs, the hypervisor might be in a bad state)
+  Mon Oct 26 19:13:49 2009 * instance confirmed to be running on its primary node (node2)
+  Mon Oct 26 19:13:49 2009 * switching node node1 to secondary mode
+  Mon Oct 26 19:13:50 2009 * wait until resync is done
+  Mon Oct 26 19:13:50 2009 * changing into standalone mode
+  Mon Oct 26 19:13:50 2009 * changing disks into single-master mode
+  Mon Oct 26 19:13:50 2009 * wait until resync is done
+  Mon Oct 26 19:13:51 2009 * done
+  node1#
+
+In use disks at instance shutdown
++++++++++++++++++++++++++++++++++
+
+If you see something like the following when trying to shutdown or
+deactivate disks for an instance::
+
+  node1# gnt-instance shutdown instance1
+  Mon Oct 26 19:16:23 2009  - WARNING: Could not shutdown block device disk/0 on node node2: drbd0: can't shutdown drbd device: /dev/drbd0: State change failed: (-12) Device is held open by someone\n
+
+It most likely means something is holding open the underlying DRBD
+device. This can be bad if the instance is not running, as it might mean
+that there was concurrent access from both the node and the instance to
+the disks, but not always (e.g. you could only have had the partitions
+activated via ``kpartx``).
+
+To troubleshoot this issue you need to follow standard Linux practices,
+and pay attention to the hypervisor being used:
+
+- check if (in the above example) ``/dev/drbd0`` on node2 is being
+  mounted somewhere (``cat /proc/mounts``)
+- check if the device is not being used by device mapper itself:
+  ``dmsetup ls`` and look for entries of the form ``drbd0pX``, and if so
+  remove them with either ``kpartx -d`` or ``dmsetup remove``
+
+For Xen, check if it's not using the disks itself::
+
+  node1# xenstore-ls /local/domain/0/backend/vbd|grep -e "domain =" -e physical-device
+  domain = "instance2"
+  physical-device = "93:0"
+  domain = "instance3"
+  physical-device = "93:1"
+  domain = "instance4"
+  physical-device = "93:2"
+  node1#
+
+You can see in the above output that the node exports three disks, to
+three instances. The ``physical-device`` key is in major:minor format in
+hexadecimal, and 0x93 represents DRBD's major number. Thus we can see
+from the above that instance2 has /dev/drbd0, instance3 /dev/drbd1, and
+instance4 /dev/drbd2.
+
+.. vim: set textwidth=72 :
+.. Local Variables:
+.. mode: rst
+.. fill-column: 72
+.. End:

diff --git a/man/gnt-node.sgml b/man/gnt-node.sgml
index 6b8d9b1..44ab6c2 100644 (file)
--- a/man/gnt-node.sgml
+++ b/man/gnt-node.sgml
@@ -907,7 +907,7 @@ node5.example.com /dev/sdb1 698.6G 1.3G 697.4G
       </para>
 
       <para>
       </para>
 
       <para>

-        On LVM volume groups, <command>repair-volume</command> runs
+        On LVM volume groups, <command>repair-storage</command> runs

         <quote>vgreduce --removemissing</quote>.
       </para>
 
         <quote>vgreduce --removemissing</quote>.
       </para>