Synnefo » snf-ganeti

.. contents::

Introduction

------------

information store for helping with cluster administration, for example

by attaching owner information to each instance after it's created::

And then by listing each instance and its tags, this information could

be used for contacting the users of each instance.

While not directly visible by an end-user, it's useful to know that a

basic cluster operation (e.g. starting an instance) is represented

code). These OpCodes are executed as part of a *Job*. The OpCodes in a

single Job are processed serially by Ganeti, but different Jobs will be

processed (depending on resource availability) in parallel. They will

With the above parameters in mind, the command is::

The instance name must be resolvable (e.g. exist in DNS) and usually

points to an address in the same subnet as the cluster itself.

single disk of 50GB and the default memory size, having primary node

``node1`` and secondary node ``node3``, use the following command::

    instance1

There is a also a command for batch instance creation from a

irreversible and destroys all the contents of your instance. Use with

care::

.. _instance-startup-label:

Instances are automatically started at instance creation time. To

manually start one which is currently stopped you can run::

Ganeti will start an instance with up to its maximum instance memory. If

not enough memory is available Ganeti will use all the available memory

stopped state ``offline``. In this case, you will first have to

put it back to online mode by running::

The command to stop the running instance is::

If you want to shut the instance down more permanently, so that it

does not require dynamically allocated resources (memory and vcpus),

after shutting down an instance, execute the following::

.. warning:: Do not use the Xen or KVM commands directly to stop

   instances. If you run for example ``xm shutdown`` or ``xm destroy``

The command to see all the instances configured and their status is::

The command can return a custom set of information when using the ``-o``

option (as always, check the manpage for a detailed specification). Each

To get more detailed information about an instance, you can run::

which will give a multi-line block of information about the instance,

it's hardware resources (especially its disks and their redundancy

If you find that you need more memory on a node any instance can be

manually resized without downtime, with the command::

The same command can also be used to increase the memory available on an

instance, provided that enough free memory is available on its node, and

configuration, which then you can backup, or import into another

cluster. The way to export an instance is::

The target node can be any node in the cluster with enough space under

Importing an instance is similar to creating a new one, but additionally

one must specify the location of the snapshot. The command is::

By default, parameters will be read from the export information, but you

can of course pass them in via the command line - most of the options

then create a Ganeti instance in the usual way, except that instead of

passing the disk information you specify the current volumes::

This will take over the given logical volumes, rename them to the Ganeti

standard (UUID-based), and without installing the OS on them start

failed and it's not up anymore. Doing it is really easy, on the master

node you can just run::

That's it. After the command completes the secondary node is now the

primary, and vice-versa.

iallocator plugin. If you omit both, the default iallocator will be

used to determine the target node::

Live migrating an instance

~~~~~~~~~~~~~~~~~~~~~~~~~~

both its nodes are running fine, you can at migrate it over to its

secondary node, without downtime. On the master node you need to run::

The current load on the instance and its memory size will influence how

long the migration will take. In any case, for both KVM and Xen

iallocator plugin. If you omit both, the default iallocator will be

used to determine the target node::

Moving an instance (offline)

~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If an instance has not been create as mirrored, then the only way to

change its primary node is to execute the move command::

This has a few prerequisites:

is common that after a complete disk failure, any LVM command aborts

with an error similar to::

  /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error

  /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error

  Couldn't find all physical volumes for volume group xenvg.

Before restoring an instance's disks to healthy status, it's needed to

#. first, if the disk is completely gone and LVM commands exit with

   “Couldn't find device with uuid…” then you need to run the command::

#. after the above command, the LVM commands should be executing

   normally (warnings are normal, but the commands will not fail

#. if the failed disk is still visible in the output of the ``pvs``

   command, you need to deactivate it from allocations by running::

At this point, the volume group should be consistent and any bad

physical volumes should not longer be available for allocation.

For all three cases, the ``replace-disks`` operation can be used::

  # re-create disks on the primary node

  # re-create disks on the current secondary

  # change the secondary node, via manual specification

  # change the secondary node, via an iallocator script

  # since Ganeti 2.1: automatically fix the primary or secondary node

Since the process involves copying all data from the working node to the

target node, it will take a while, depending on the instance's disk

disks, after which a reinstall can be run, via the ``recreate-disks``

command::

Note that this will fail if the disks already exists.

modify`` command::

  # start with a non-redundant instance

  # later convert it to redundant

  # and convert it back

The conversion must be done while the instance is stopped, and

converting from plain to drbd template presents a small risk, especially

inconsistent. The correct way to access an instance's disks is to run

(on the master node, as usual) the command::

And then, *on the primary node of the instance*, access the device that

gets created. For example, you could mount the given disks, then edit

Note that with partitioned disks (as opposed to whole-disk filesystems),

you will need to use a tool like :manpage:`kpartx(8)`::

  # edit files under mnt as desired

After you've finished you can deactivate them with the deactivate-disks

command, which works in the same way::

Note that if any process started by you is still using the disks, the

above command will error out, and you **must** cleanup and ensure that

The command to access a running instance's console is::

Use the console normally and then type ``^]`` when done, to exit.

There is a wrapper command for rebooting instances::

By default, this does the equivalent of shutting down and then starting

the instance, but it accepts parameters to perform a soft-reboot (via

Should you have any problems with instance operating systems the command

to see a complete status for all your nodes is::

.. _instance-relocation-label:

steps::

  # instance is located on A, B

  # instance has moved from (A, B) to (A, C)

  # we now flip the primary/secondary nodes

  # instance lives on (C, A)

  # we can then change A to D via:

Which brings it into the final configuration of ``(C, D)``. Note that we

needed to do two replace-disks operation (two copies of the instance

It is at any time possible to extend the cluster with one more node, by

using the node add operation::

If the cluster has a replication network defined, then you need to pass

the ``-s REPLICATION_IP`` parameter to this option.

Ganeti configuration is broken, for example if it has been reinstalled

by mistake::

This will reinitialise the node as if it's been newly added, but while

keeping its existing configuration in the cluster (primary/secondary IP,

If you want to promote a different node to the master role (for whatever

reason), run on any other master-candidate node the command::

and the node you ran it on is now the new master. In case you try to run

this on a non master-candidate node, you will get an error telling you

The ``gnt-node modify`` command can be used to select a new role::

  # change to master candidate

  # change to drained status

  # change to offline status

  # change to regular mode (reset all flags)

Note that the cluster requires that at any point in time, a certain

number of nodes are master candidates, so changing from master candidate

commands (as seen in :ref:`instance-change-primary-label`) or the bulk

per-node versions; these are::

Note that the instance “move” command doesn't currently have a node

equivalent.

For the evacuation of secondary instances, a command called

:command:`gnt-node evacuate` is provided and its syntax is::

The first version will compute the new secondary for each instance in

turn using the given iallocator script, whereas the second one will

Once a node no longer has any instances (neither primary nor secondary),

it's easy to remove it from the cluster::

This will deconfigure the node, stop the ganeti daemons on it and leave

it hopefully like before it joined to the cluster.

logical volumes on a given node or on all nodes and their association to

instances via the ``volumes`` command::

  Node  PhysDev   VG    Name             Size Instance

  node1 /dev/sdb1 xenvg e61fbc97-….disk0 512M instance17

  node1 /dev/sdb1 xenvg ebd1a7d1-….disk0 512M instance19

First is listing the backend storage and their space situation::

  Node  Name        Size Used   Free

  node1 /dev/sda7 673.8G   0M 673.8G

  node1 /dev/sdb1 698.6G 1.5G 697.1G

The default is to list LVM physical volumes. It's also possible to list

the LVM volume groups::

  Node  Name  Size

  node1 xenvg 1.3T

  node2 xenvg 1.3T

Next is repairing storage units, which is currently only implemented for

volume groups and does the equivalent of ``vgreduce --removemissing``::

  Sun Oct 25 22:21:45 2009 Repairing storage unit 'xenvg' on node2 ...

Last is the modification of volume properties, which is (again) only

implemented for LVM physical volumes and allows toggling the

``allocatable`` value::

Use of the storage commands

~~~~~~~~~~~~~~~~~~~~~~~~~~~

One of the few commands that can be run on any node (not only the

master) is the ``getmaster`` command::

  node1.example.com

It is possible to query and change global cluster parameters via the

``info`` and ``modify`` commands::

  Cluster name: cluster.example.com

  Cluster UUID: 07805e6f-f0af-4310-95f1-572862ee939c

  Creation time: 2009-09-25 05:04:15

For detailed option list see the :manpage:`gnt-cluster(8)` man page.

The cluster version can be obtained via the ``version`` command::

  Software version: 2.1.0

  Internode protocol: 20

  Configuration format: 2010000

There are two commands provided for replicating files to all nodes of a

cluster and for running commands on all the nodes::

These are simple wrappers over scp/ssh and more advanced usage can be

obtained using :manpage:`dsh(1)` and similar commands. But they are

highlighting any issues. In normal operation, this command should return

no ``ERROR`` messages::

  Sun Oct 25 23:08:58 2009 * Verifying global settings

  Sun Oct 25 23:08:58 2009 * Gathering data (2 nodes)

  Sun Oct 25 23:09:00 2009 * Verifying node status

disks have the correct status based on the desired instance state

(up/down)::

Note that this command will show no output when disks are healthy.

recorded disk size and the actual disk size (disk size information is

needed for proper activation and growth of DRBD-based disks)::

  Sun Oct 25 23:13:16 2009  - INFO: Disk 0 of instance instance1 has mismatched size, correcting: recorded 512, actual 2048

  Sun Oct 25 23:13:17 2009  - WARNING: Invalid result from node node4, ignoring node results

configuration files, you can force an push of the master configuration

to all other nodes via the ``redist-conf`` command::

This command will be silent unless there are problems sending updates to

the other nodes.

``rename`` command. If only the IP has changed, you need to pass the

current name and Ganeti will realise its IP has changed::

  This will rename the cluster to 'cluster.example.com'. If

  you are connected over the network to the cluster name, the operation

  is very dangerous as the IP address will be removed from the node and

  the change may not go through. Continue?

  Failure: prerequisites not met for this operation:

  Neither the name nor the IP address of the cluster has changed

The job queue execution in Ganeti 2.0 and higher can be inspected,

suspended and resumed via the ``queue`` command::

  The drain flag is unset

  Failed to submit job for instance1: Job queue is drained, refusing job

  The drain flag is set

This is most useful if you have an active cluster and you need to

upgrade the Ganeti software, or simply restart the software on any node:

forgotten. Thus there are some commands for automated control of the

watcher: ``pause``, ``info`` and ``continue``::

  The watcher is not paused.

  The watcher is paused until Mon Oct 26 00:30:37 2009.

  The watcher is paused until Mon Oct 26 00:30:37 2009.

  2009-10-25 23:30:47,984:  pid=28867 ganeti-watcher:486 DEBUG Pause has been set, exiting

  The watcher is no longer paused.

  2009-10-25 23:31:04,789:  pid=28976 ganeti-watcher:345 DEBUG Archived 0 jobs, left 0

  2009-10-25 23:31:05,884:  pid=28976 ganeti-watcher:280 DEBUG Got data from cluster, writing instance status file

  2009-10-25 23:31:06,061:  pid=28976 ganeti-watcher:150 DEBUG Data didn't change, just touching status file

  The watcher is not paused.

The exact details of the argument to the ``pause`` command are available

in the manpage.

Tags can be added via ``add-tags``::

The above commands add three tags to an instance, to a node and to the

Tags can also be remove via a syntax very similar to the add one::

And listed via::

Global tag search

+++++++++++++++++

It is also possible to execute a global search on the all tags defined

in the cluster configuration, via a cluster command::

The parameter expected is a regular expression (see

:manpage:`regex(7)`). This will return all tags that match the search,

together with the object they are defined in (the names being show in a

hierarchical kind of way)::

  /cluster foo

  /instances/instance1 owner:bar

First is the job list command::

  17771 success INSTANCE_QUERY_DATA

  17773 success CLUSTER_VERIFY_DISKS

  17775 success CLUSTER_REPAIR_DISK_SIZES

More detailed information about a job can be found via the ``info``

command::

  Job ID: 17776

    Status: error

    Received:         2009-10-25 23:18:02.180569

job, similar to the log that one get from the ``gnt-`` commands, via the

watch command::

  JobID: 17818

  Output from job 17818 follows

  -----------------------------

  Mon Oct 26 00:22:48 2009  - INFO: Selected nodes for instance instance1 via iallocator dumb: node1, node2

  Mon Oct 26 00:23:03 2009 creating os for instance instance1 on node node1

  Mon Oct 26 00:23:03 2009 * running the instance OS create scripts...

  Mon Oct 26 00:23:13 2009 * starting instance...

This is useful if you need to follow a job's progress from multiple

terminals.

A job that has not yet started to run can be canceled::

But not one that has already started execution::

  Job 17805 is no longer waiting in the queue

There are two queues for jobs: the *current* and the *archive*

strategies, etc. In order to accomplish this, mark these nodes as

non-``vm_capable``::

The vm_capable status can be listed as usual via ``gnt-node list``::

  Node  VMCapable

  node1 Y

  node2 Y

As usual, the node modify operation can change this flag::

  Fri Jan  7 06:23:07 2011  - INFO: Demoting from master candidate

  Fri Jan  7 06:23:08 2011  - INFO: Promoted nodes to master candidate role: node4

  Modified node node3

And the node list operation will list this flag::

  Node  MasterCapable

  node1 Y

  node2 Y