1 Ganeti administrator's guide
2 ============================
4 Documents Ganeti version |version|
13 Ganeti is a virtualization cluster management software. You are expected
14 to be a system administrator familiar with your Linux distribution and
15 the Xen or KVM virtualization environments before using it.
17 The various components of Ganeti all have man pages and interactive
18 help. This manual though will help you getting familiar with the system
19 by explaining the most common operations, grouped by related use.
21 After a terminology glossary and a section on the prerequisites needed
22 to use this manual, the rest of this document is divided in sections
23 for the different targets that a command affects: instance, nodes, etc.
25 .. _terminology-label:
30 This section provides a small introduction to Ganeti terminology, which
31 might be useful when reading the rest of the document.
36 A set of machines (nodes) that cooperate to offer a coherent, highly
37 available virtualization service under a single administration domain.
42 A physical machine which is member of a cluster. Nodes are the basic
43 cluster infrastructure, and they don't need to be fault tolerant in
44 order to achieve high availability for instances.
46 Node can be added and removed (if they host no instances) at will from
47 the cluster. In a HA cluster and only with HA instances, the loss of any
48 single node will not cause disk data loss for any instance; of course,
49 a node crash will cause the crash of the its primary instances.
51 A node belonging to a cluster can be in one of the following roles at a
54 - *master* node, which is the node from which the cluster is controlled
55 - *master candidate* node, only nodes in this role have the full cluster
56 configuration and knowledge, and only master candidates can become the
58 - *regular* node, which is the state in which most nodes will be on
59 bigger clusters (>20 nodes)
60 - *drained* node, nodes in this state are functioning normally but the
61 cannot receive new instances; the intention is that nodes in this role
62 have some issue and they are being evacuated for hardware repairs
63 - *offline* node, in which there is a record in the cluster
64 configuration about the node, but the daemons on the master node will
65 not talk to this node; any instances declared as having an offline
66 node as either primary or secondary will be flagged as an error in the
67 cluster verify operation
69 Depending on the role, each node will run a set of daemons:
71 - the :command:`ganeti-noded` daemon, which control the manipulation of
72 this node's hardware resources; it runs on all nodes which are in a
74 - the :command:`ganeti-confd` daemon (Ganeti 2.1+) which runs on all
75 nodes, but is only functional on master candidate nodes; this daemon
76 can be disabled at configuration time if you don't need its
78 - the :command:`ganeti-rapi` daemon which runs on the master node and
79 offers an HTTP-based API for the cluster
80 - the :command:`ganeti-masterd` daemon which runs on the master node and
81 allows control of the cluster
83 Beside the node role, there are other node flags that influence its
86 - the *master_capable* flag denotes whether the node can ever become a
87 master candidate; setting this to 'no' means that auto-promotion will
88 never make this node a master candidate; this flag can be useful for a
89 remote node that only runs local instances, and having it become a
90 master is impractical due to networking or other constraints
91 - the *vm_capable* flag denotes whether the node can host instances or
92 not; for example, one might use a non-vm_capable node just as a master
93 candidate, for configuration backups; setting this flag to no
94 disallows placement of instances of this node, deactivates hypervisor
95 and related checks on it (e.g. bridge checks, LVM check, etc.), and
96 removes it from cluster capacity computations
102 A virtual machine which runs on a cluster. It can be a fault tolerant,
103 highly available entity.
105 An instance has various parameters, which are classified in three
106 categories: hypervisor related-parameters (called ``hvparams``), general
107 parameters (called ``beparams``) and per network-card parameters (called
108 ``nicparams``). All these parameters can be modified either at instance
109 level or via defaults at cluster level.
114 The are multiple options for the storage provided to an instance; while
115 the instance sees the same virtual drive in all cases, the node-level
116 configuration varies between them.
118 There are four disk templates you can choose from:
121 The instance has no disks. Only used for special purpose operating
122 systems or for testing.
125 The instance will use plain files as backend for its disks. No
126 redundancy is provided, and this is somewhat more difficult to
127 configure for high performance.
130 The instance will use LVM devices as backend for its disks. No
131 redundancy is provided.
134 .. note:: This is only valid for multi-node clusters using DRBD 8.0+
136 A mirror is set between the local node and a remote one, which must be
137 specified with the second value of the --node option. Use this option
138 to obtain a highly available instance that can be failed over to a
139 remote node should the primary one fail.
144 A framework for using external (user-provided) scripts to compute the
145 placement of instances on the cluster nodes. This eliminates the need to
146 manually specify nodes in instance add, instance moves, node evacuate,
149 In order for Ganeti to be able to use these scripts, they must be place
150 in the iallocator directory (usually ``lib/ganeti/iallocators`` under
151 the installation prefix, e.g. ``/usr/local``).
153 “Primary” and “secondary” concepts
154 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
156 An instance has a primary and depending on the disk configuration, might
157 also have a secondary node. The instance always runs on the primary node
158 and only uses its secondary node for disk replication.
160 Similarly, the term of primary and secondary instances when talking
161 about a node refers to the set of instances having the given node as
162 primary, respectively secondary.
167 Tags are short strings that can be attached to either to cluster itself,
168 or to nodes or instances. They are useful as a very simplistic
169 information store for helping with cluster administration, for example
170 by attaching owner information to each instance after it's created::
172 gnt-instance add … instance1
173 gnt-instance add-tags instance1 owner:user2
175 And then by listing each instance and its tags, this information could
176 be used for contacting the users of each instance.
181 While not directly visible by an end-user, it's useful to know that a
182 basic cluster operation (e.g. starting an instance) is represented
183 internall by Ganeti as an *OpCode* (abbreviation from operation
184 code). These OpCodes are executed as part of a *Job*. The OpCodes in a
185 single Job are processed serially by Ganeti, but different Jobs will be
186 processed (depending on resource availability) in parallel. They will
187 not be executed in the submission order, but depending on resource
188 availability, locks and (starting with Ganeti 2.3) priority. An earlier
189 job may have to wait for a lock while a newer job doesn't need any locks
190 and can be executed right away. Operations requiring a certain order
191 need to be submitted as a single job, or the client must submit one job
192 at a time and wait for it to finish before continuing.
194 For example, shutting down the entire cluster can be done by running the
195 command ``gnt-instance shutdown --all``, which will submit for each
196 instance a separate job containing the “shutdown instance” OpCode.
202 You need to have your Ganeti cluster installed and configured before you
203 try any of the commands in this document. Please follow the
204 :doc:`install` for instructions on how to do that.
212 The add operation might seem complex due to the many parameters it
213 accepts, but once you have understood the (few) required parameters and
214 the customisation capabilities you will see it is an easy operation.
216 The add operation requires at minimum five parameters:
218 - the OS for the instance
220 - the disk count and size
221 - the node specification or alternatively the iallocator to use
222 - and finally the instance name
224 The OS for the instance must be visible in the output of the command
225 ``gnt-os list`` and specifies which guest OS to install on the instance.
227 The disk template specifies what kind of storage to use as backend for
228 the (virtual) disks presented to the instance; note that for instances
229 with multiple virtual disks, they all must be of the same type.
231 The node(s) on which the instance will run can be given either manually,
232 via the ``-n`` option, or computed automatically by Ganeti, if you have
233 installed any iallocator script.
235 With the above parameters in mind, the command is::
238 -n TARGET_NODE:SECONDARY_NODE \
240 -t DISK_TEMPLATE -s DISK_SIZE \
243 The instance name must be resolvable (e.g. exist in DNS) and usually
244 points to an address in the same subnet as the cluster itself.
246 The above command has the minimum required options; other options you
247 can give include, among others:
249 - The memory size (``-B memory``)
251 - The number of virtual CPUs (``-B vcpus``)
253 - Arguments for the NICs of the instance; by default, a single-NIC
254 instance is created. The IP and/or bridge of the NIC can be changed
255 via ``--nic 0:ip=IP,bridge=BRIDGE``
257 See the manpage for gnt-instance for the detailed option list.
259 For example if you want to create an highly available instance, with a
260 single disk of 50GB and the default memory size, having primary node
261 ``node1`` and secondary node ``node3``, use the following command::
263 gnt-instance add -n node1:node3 -o debootstrap -t drbd \
266 There is a also a command for batch instance creation from a
267 specification file, see the ``batch-create`` operation in the
268 gnt-instance manual page.
270 Regular instance operations
271 +++++++++++++++++++++++++++
276 Removing an instance is even easier than creating one. This operation is
277 irreversible and destroys all the contents of your instance. Use with
280 gnt-instance remove INSTANCE_NAME
285 Instances are automatically started at instance creation time. To
286 manually start one which is currently stopped you can run::
288 gnt-instance startup INSTANCE_NAME
290 Note, that this will not work when an instance is in a permanently
291 stopped state ``offline``. In this case, you will first have to
292 put it back to online mode by running::
294 gnt-instance modify --online INSTANCE_NAME
296 The command to stop the running instance is::
298 gnt-instance shutdown INSTANCE_NAME
300 If you want to shut the instance down more permanently, so that it
301 does not require dynamically allocated resources (memory and vcpus),
302 after shutting down an instance, execute the following::
304 gnt-instance modify --ofline INSTANCE_NAME
306 .. warning:: Do not use the Xen or KVM commands directly to stop
307 instances. If you run for example ``xm shutdown`` or ``xm destroy``
308 on an instance Ganeti will automatically restart it (via
309 the :command:`ganeti-watcher` command which is launched via cron).
314 There are two ways to get information about instances: listing
315 instances, which does a tabular output containing a given set of fields
316 about each instance, and querying detailed information about a set of
319 The command to see all the instances configured and their status is::
323 The command can return a custom set of information when using the ``-o``
324 option (as always, check the manpage for a detailed specification). Each
325 instance will be represented on a line, thus making it easy to parse
326 this output via the usual shell utilities (grep, sed, etc.).
328 To get more detailed information about an instance, you can run::
330 gnt-instance info INSTANCE
332 which will give a multi-line block of information about the instance,
333 it's hardware resources (especially its disks and their redundancy
334 status), etc. This is harder to parse and is more expensive than the
335 list operation, but returns much more detailed information.
341 You can create a snapshot of an instance disk and its Ganeti
342 configuration, which then you can backup, or import into another
343 cluster. The way to export an instance is::
345 gnt-backup export -n TARGET_NODE INSTANCE_NAME
348 The target node can be any node in the cluster with enough space under
349 ``/srv/ganeti`` to hold the instance image. Use the ``--noshutdown``
350 option to snapshot an instance without rebooting it. Note that Ganeti
351 only keeps one snapshot for an instance - any previous snapshot of the
352 same instance existing cluster-wide under ``/srv/ganeti`` will be
353 removed by this operation: if you want to keep them, you need to move
354 them out of the Ganeti exports directory.
356 Importing an instance is similar to creating a new one, but additionally
357 one must specify the location of the snapshot. The command is::
359 gnt-backup import -n TARGET_NODE \
360 --src-node=NODE --src-dir=DIR INSTANCE_NAME
362 By default, parameters will be read from the export information, but you
363 can of course pass them in via the command line - most of the options
364 available for the command :command:`gnt-instance add` are supported here
367 Import of foreign instances
368 +++++++++++++++++++++++++++
370 There is a possibility to import a foreign instance whose disk data is
371 already stored as LVM volumes without going through copying it: the disk
374 For this, ensure that the original, non-managed instance is stopped,
375 then create a Ganeti instance in the usual way, except that instead of
376 passing the disk information you specify the current volumes::
378 gnt-instance add -t plain -n HOME_NODE ... \
379 --disk 0:adopt=lv_name[,vg=vg_name] INSTANCE_NAME
381 This will take over the given logical volumes, rename them to the Ganeti
382 standard (UUID-based), and without installing the OS on them start
383 directly the instance. If you configure the hypervisor similar to the
384 non-managed configuration that the instance had, the transition should
385 be seamless for the instance. For more than one disk, just pass another
386 disk parameter (e.g. ``--disk 1:adopt=...``).
388 Instance kernel selection
389 +++++++++++++++++++++++++
391 The kernel that instances uses to bootup can come either from the node,
392 or from instances themselves, depending on the setup.
397 With Xen PVM, there are three options.
399 First, you can use a kernel from the node, by setting the hypervisor
402 - ``kernel_path`` to a valid file on the node (and appropriately
404 - ``kernel_args`` optionally set to a valid Linux setting (e.g. ``ro``)
405 - ``root_path`` to a valid setting (e.g. ``/dev/xvda1``)
406 - ``bootloader_path`` and ``bootloader_args`` to empty
408 Alternatively, you can delegate the kernel management to instances, and
409 use either ``pvgrub`` or the deprecated ``pygrub``. For this, you must
410 install the kernels and initrds in the instance and create a valid GRUB
411 v1 configuration file.
413 For ``pvgrub`` (new in version 2.4.2), you need to set:
415 - ``kernel_path`` to point to the ``pvgrub`` loader present on the node
416 (e.g. ``/usr/lib/xen/boot/pv-grub-x86_32.gz``)
417 - ``kernel_args`` to the path to the GRUB config file, relative to the
418 instance (e.g. ``(hd0,0)/grub/menu.lst``)
419 - ``root_path`` **must** be empty
420 - ``bootloader_path`` and ``bootloader_args`` to empty
422 While ``pygrub`` is deprecated, here is how you can configure it:
424 - ``bootloader_path`` to the pygrub binary (e.g. ``/usr/bin/pygrub``)
425 - the other settings are not important
427 More information can be found in the Xen wiki pages for `pvgrub
428 <http://wiki.xensource.com/xenwiki/PvGrub>`_ and `pygrub
429 <http://wiki.xensource.com/xenwiki/PyGrub>`_.
434 For KVM also the kernel can be loaded either way.
436 For loading the kernels from the node, you need to set:
438 - ``kernel_path`` to a valid value
439 - ``initrd_path`` optionally set if you use an initrd
440 - ``kernel_args`` optionally set to a valid value (e.g. ``ro``)
442 If you want instead to have the instance boot from its disk (and execute
443 its bootloader), simply set the ``kernel_path`` parameter to an empty
444 string, and all the others will be ignored.
449 .. note:: This section only applies to multi-node clusters
451 .. _instance-change-primary-label:
453 Changing the primary node
454 +++++++++++++++++++++++++
456 There are three ways to exchange an instance's primary and secondary
457 nodes; the right one to choose depends on how the instance has been
458 created and the status of its current primary node. See
459 :ref:`rest-redundancy-label` for information on changing the secondary
460 node. Note that it's only possible to change the primary node to the
461 secondary and vice-versa; a direct change of the primary node with a
462 third node, while keeping the current secondary is not possible in a
463 single step, only via multiple operations as detailed in
464 :ref:`instance-relocation-label`.
466 Failing over an instance
467 ~~~~~~~~~~~~~~~~~~~~~~~~
469 If an instance is built in highly available mode you can at any time
470 fail it over to its secondary node, even if the primary has somehow
471 failed and it's not up anymore. Doing it is really easy, on the master
472 node you can just run::
474 gnt-instance failover INSTANCE_NAME
476 That's it. After the command completes the secondary node is now the
477 primary, and vice-versa.
479 Live migrating an instance
480 ~~~~~~~~~~~~~~~~~~~~~~~~~~
482 If an instance is built in highly available mode, it currently runs and
483 both its nodes are running fine, you can at migrate it over to its
484 secondary node, without downtime. On the master node you need to run::
486 gnt-instance migrate INSTANCE_NAME
488 The current load on the instance and its memory size will influence how
489 long the migration will take. In any case, for both KVM and Xen
490 hypervisors, the migration will be transparent to the instance.
492 Moving an instance (offline)
493 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
495 If an instance has not been create as mirrored, then the only way to
496 change its primary node is to execute the move command::
498 gnt-instance move -n NEW_NODE INSTANCE
500 This has a few prerequisites:
502 - the instance must be stopped
503 - its current primary node must be on-line and healthy
504 - the disks of the instance must not have any errors
506 Since this operation actually copies the data from the old node to the
507 new node, expect it to take proportional to the size of the instance's
508 disks and the speed of both the nodes' I/O system and their networking.
513 Disk failures are a common cause of errors in any server
514 deployment. Ganeti offers protection from single-node failure if your
515 instances were created in HA mode, and it also offers ways to restore
516 redundancy after a failure.
518 Preparing for disk operations
519 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
521 It is important to note that for Ganeti to be able to do any disk
522 operation, the Linux machines on top of which Ganeti must be consistent;
523 for LVM, this means that the LVM commands must not return failures; it
524 is common that after a complete disk failure, any LVM command aborts
525 with an error similar to::
528 /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
529 /dev/sdb1: read failed after 0 of 4096 at 750153695232: Input/output
531 /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
532 Couldn't find device with uuid
533 't30jmN-4Rcf-Fr5e-CURS-pawt-z0jU-m1TgeJ'.
534 Couldn't find all physical volumes for volume group xenvg.
536 Before restoring an instance's disks to healthy status, it's needed to
537 fix the volume group used by Ganeti so that we can actually create and
538 manage the logical volumes. This is usually done in a multi-step
541 #. first, if the disk is completely gone and LVM commands exit with
542 “Couldn't find device with uuid…” then you need to run the command::
544 vgreduce --removemissing VOLUME_GROUP
546 #. after the above command, the LVM commands should be executing
547 normally (warnings are normal, but the commands will not fail
550 #. if the failed disk is still visible in the output of the ``pvs``
551 command, you need to deactivate it from allocations by running::
555 At this point, the volume group should be consistent and any bad
556 physical volumes should not longer be available for allocation.
558 Note that since version 2.1 Ganeti provides some commands to automate
559 these two operations, see :ref:`storage-units-label`.
561 .. _rest-redundancy-label:
563 Restoring redundancy for DRBD-based instances
564 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
566 A DRBD instance has two nodes, and the storage on one of them has
567 failed. Depending on which node (primary or secondary) has failed, you
568 have three options at hand:
570 - if the storage on the primary node has failed, you need to re-create
572 - if the storage on the secondary node has failed, you can either
573 re-create the disks on it or change the secondary and recreate
574 redundancy on the new secondary node
576 Of course, at any point it's possible to force re-creation of disks even
577 though everything is already fine.
579 For all three cases, the ``replace-disks`` operation can be used::
581 # re-create disks on the primary node
582 gnt-instance replace-disks -p INSTANCE_NAME
583 # re-create disks on the current secondary
584 gnt-instance replace-disks -s INSTANCE_NAME
585 # change the secondary node, via manual specification
586 gnt-instance replace-disks -n NODE INSTANCE_NAME
587 # change the secondary node, via an iallocator script
588 gnt-instance replace-disks -I SCRIPT INSTANCE_NAME
589 # since Ganeti 2.1: automatically fix the primary or secondary node
590 gnt-instance replace-disks -a INSTANCE_NAME
592 Since the process involves copying all data from the working node to the
593 target node, it will take a while, depending on the instance's disk
594 size, node I/O system and network speed. But it is (baring any network
595 interruption) completely transparent for the instance.
597 Re-creating disks for non-redundant instances
598 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
600 .. versionadded:: 2.1
602 For non-redundant instances, there isn't a copy (except backups) to
603 re-create the disks. But it's possible to at-least re-create empty
604 disks, after which a reinstall can be run, via the ``recreate-disks``
607 gnt-instance recreate-disks INSTANCE
609 Note that this will fail if the disks already exists.
611 Conversion of an instance's disk type
612 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
614 It is possible to convert between a non-redundant instance of type
615 ``plain`` (LVM storage) and redundant ``drbd`` via the ``gnt-instance
618 # start with a non-redundant instance
619 gnt-instance add -t plain ... INSTANCE
621 # later convert it to redundant
622 gnt-instance stop INSTANCE
623 gnt-instance modify -t drbd -n NEW_SECONDARY INSTANCE
624 gnt-instance start INSTANCE
626 # and convert it back
627 gnt-instance stop INSTANCE
628 gnt-instance modify -t plain INSTANCE
629 gnt-instance start INSTANCE
631 The conversion must be done while the instance is stopped, and
632 converting from plain to drbd template presents a small risk, especially
633 if the instance has multiple disks and/or if one node fails during the
634 conversion procedure). As such, it's recommended (as always) to make
635 sure that downtime for manual recovery is acceptable and that the
636 instance has up-to-date backups.
641 Accessing an instance's disks
642 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
644 From an instance's primary node you can have access to its disks. Never
645 ever mount the underlying logical volume manually on a fault tolerant
646 instance, or will break replication and your data will be
647 inconsistent. The correct way to access an instance's disks is to run
648 (on the master node, as usual) the command::
650 gnt-instance activate-disks INSTANCE
652 And then, *on the primary node of the instance*, access the device that
653 gets created. For example, you could mount the given disks, then edit
654 files on the filesystem, etc.
656 Note that with partitioned disks (as opposed to whole-disk filesystems),
657 you will need to use a tool like :manpage:`kpartx(8)`::
659 node1# gnt-instance activate-disks instance1
662 node3# kpartx -l /dev/…
663 node3# kpartx -a /dev/…
664 node3# mount /dev/mapper/… /mnt/
665 # edit files under mnt as desired
667 node3# kpartx -d /dev/…
671 After you've finished you can deactivate them with the deactivate-disks
672 command, which works in the same way::
674 gnt-instance deactivate-disks INSTANCE
676 Note that if any process started by you is still using the disks, the
677 above command will error out, and you **must** cleanup and ensure that
678 the above command runs successfully before you start the instance,
679 otherwise the instance will suffer corruption.
681 Accessing an instance's console
682 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
684 The command to access a running instance's console is::
686 gnt-instance console INSTANCE_NAME
688 Use the console normally and then type ``^]`` when done, to exit.
690 Other instance operations
691 +++++++++++++++++++++++++
696 There is a wrapper command for rebooting instances::
698 gnt-instance reboot instance2
700 By default, this does the equivalent of shutting down and then starting
701 the instance, but it accepts parameters to perform a soft-reboot (via
702 the hypervisor), a hard reboot (hypervisor shutdown and then startup) or
703 a full one (the default, which also de-configures and then configures
704 again the disks of the instance).
706 Instance OS definitions debugging
707 +++++++++++++++++++++++++++++++++
709 Should you have any problems with instance operating systems the command
710 to see a complete status for all your nodes is::
714 .. _instance-relocation-label:
719 While it is not possible to move an instance from nodes ``(A, B)`` to
720 nodes ``(C, D)`` in a single move, it is possible to do so in a few
723 # instance is located on A, B
724 node1# gnt-instance replace -n nodeC instance1
725 # instance has moved from (A, B) to (A, C)
726 # we now flip the primary/secondary nodes
727 node1# gnt-instance migrate instance1
728 # instance lives on (C, A)
729 # we can then change A to D via:
730 node1# gnt-instance replace -n nodeD instance1
732 Which brings it into the final configuration of ``(C, D)``. Note that we
733 needed to do two replace-disks operation (two copies of the instance
734 disks), because we needed to get rid of both the original nodes (A and
740 There are much fewer node operations available than for instances, but
741 they are equivalently important for maintaining a healthy cluster.
746 It is at any time possible to extend the cluster with one more node, by
747 using the node add operation::
749 gnt-node add NEW_NODE
751 If the cluster has a replication network defined, then you need to pass
752 the ``-s REPLICATION_IP`` parameter to this option.
754 A variation of this command can be used to re-configure a node if its
755 Ganeti configuration is broken, for example if it has been reinstalled
758 gnt-node add --readd EXISTING_NODE
760 This will reinitialise the node as if it's been newly added, but while
761 keeping its existing configuration in the cluster (primary/secondary IP,
762 etc.), in other words you won't need to use ``-s`` here.
764 Changing the node role
765 ++++++++++++++++++++++
767 A node can be in different roles, as explained in the
768 :ref:`terminology-label` section. Promoting a node to the master role is
769 special, while the other roles are handled all via a single command.
771 Failing over the master node
772 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
774 If you want to promote a different node to the master role (for whatever
775 reason), run on any other master-candidate node the command::
777 gnt-cluster master-failover
779 and the node you ran it on is now the new master. In case you try to run
780 this on a non master-candidate node, you will get an error telling you
781 which nodes are valid.
783 Changing between the other roles
784 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
786 The ``gnt-node modify`` command can be used to select a new role::
788 # change to master candidate
789 gnt-node modify -C yes NODE
790 # change to drained status
791 gnt-node modify -D yes NODE
792 # change to offline status
793 gnt-node modify -O yes NODE
794 # change to regular mode (reset all flags)
795 gnt-node modify -O no -D no -C no NODE
797 Note that the cluster requires that at any point in time, a certain
798 number of nodes are master candidates, so changing from master candidate
799 to other roles might fail. It is recommended to either force the
800 operation (via the ``--force`` option) or first change the number of
801 master candidates in the cluster - see :ref:`cluster-config-label`.
806 There are two steps of moving instances off a node:
808 - moving the primary instances (actually converting them into secondary
810 - moving the secondary instances (including any instances converted in
813 Primary instance conversion
814 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
816 For this step, you can use either individual instance move
817 commands (as seen in :ref:`instance-change-primary-label`) or the bulk
818 per-node versions; these are::
820 gnt-node migrate NODE
821 gnt-node evacuate NODE
823 Note that the instance “move” command doesn't currently have a node
826 Both these commands, or the equivalent per-instance command, will make
827 this node the secondary node for the respective instances, whereas their
828 current secondary node will become primary. Note that it is not possible
829 to change in one step the primary node to another node as primary, while
830 keeping the same secondary node.
832 Secondary instance evacuation
833 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
835 For the evacuation of secondary instances, a command called
836 :command:`gnt-node evacuate` is provided and its syntax is::
838 gnt-node evacuate -I IALLOCATOR_SCRIPT NODE
839 gnt-node evacuate -n DESTINATION_NODE NODE
841 The first version will compute the new secondary for each instance in
842 turn using the given iallocator script, whereas the second one will
843 simply move all instances to DESTINATION_NODE.
848 Once a node no longer has any instances (neither primary nor secondary),
849 it's easy to remove it from the cluster::
851 gnt-node remove NODE_NAME
853 This will deconfigure the node, stop the ganeti daemons on it and leave
854 it hopefully like before it joined to the cluster.
859 When using LVM (either standalone or with DRBD), it can become tedious
860 to debug and fix it in case of errors. Furthermore, even file-based
861 storage can become complicated to handle manually on many hosts. Ganeti
862 provides a couple of commands to help with automation.
867 This is a command specific to LVM handling. It allows listing the
868 logical volumes on a given node or on all nodes and their association to
869 instances via the ``volumes`` command::
871 node1# gnt-node volumes
872 Node PhysDev VG Name Size Instance
873 node1 /dev/sdb1 xenvg e61fbc97-….disk0 512M instance17
874 node1 /dev/sdb1 xenvg ebd1a7d1-….disk0 512M instance19
875 node2 /dev/sdb1 xenvg 0af08a3d-….disk0 512M instance20
876 node2 /dev/sdb1 xenvg cc012285-….disk0 512M instance16
877 node2 /dev/sdb1 xenvg f0fac192-….disk0 512M instance18
879 The above command maps each logical volume to a volume group and
880 underlying physical volume and (possibly) to an instance.
882 .. _storage-units-label:
884 Generalized storage handling
885 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
887 .. versionadded:: 2.1
889 Starting with Ganeti 2.1, a new storage framework has been implemented
890 that tries to abstract the handling of the storage type the cluster
893 First is listing the backend storage and their space situation::
895 node1# gnt-node list-storage
896 Node Name Size Used Free
897 node1 /dev/sda7 673.8G 0M 673.8G
898 node1 /dev/sdb1 698.6G 1.5G 697.1G
899 node2 /dev/sda7 673.8G 0M 673.8G
900 node2 /dev/sdb1 698.6G 1.0G 697.6G
902 The default is to list LVM physical volumes. It's also possible to list
903 the LVM volume groups::
905 node1# gnt-node list-storage -t lvm-vg
910 Next is repairing storage units, which is currently only implemented for
911 volume groups and does the equivalent of ``vgreduce --removemissing``::
913 node1# gnt-node repair-storage node2 lvm-vg xenvg
914 Sun Oct 25 22:21:45 2009 Repairing storage unit 'xenvg' on node2 ...
916 Last is the modification of volume properties, which is (again) only
917 implemented for LVM physical volumes and allows toggling the
918 ``allocatable`` value::
920 node1# gnt-node modify-storage --allocatable=no node2 lvm-pv /dev/sdb1
922 Use of the storage commands
923 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
925 All these commands are needed when recovering a node from a disk
928 - first, we need to recover from complete LVM failure (due to missing
929 disk), by running the ``repair-storage`` command
930 - second, we need to change allocation on any partially-broken disk
931 (i.e. LVM still sees it, but it has bad blocks) by running
933 - then we can evacuate the instances as needed
939 Beside the cluster initialisation command (which is detailed in the
940 :doc:`install` document) and the master failover command which is
941 explained under node handling, there are a couple of other cluster
942 operations available.
944 .. _cluster-config-label:
949 One of the few commands that can be run on any node (not only the
950 master) is the ``getmaster`` command::
952 node2# gnt-cluster getmaster
956 It is possible to query and change global cluster parameters via the
957 ``info`` and ``modify`` commands::
959 node1# gnt-cluster info
960 Cluster name: cluster.example.com
961 Cluster UUID: 07805e6f-f0af-4310-95f1-572862ee939c
962 Creation time: 2009-09-25 05:04:15
963 Modification time: 2009-10-18 22:11:47
964 Master node: node1.example.com
965 Architecture (this node): 64bit (x86_64)
968 Default hypervisor: xen-pvm
969 Enabled hypervisors: xen-pvm
970 Hypervisor parameters:
975 - candidate pool size: 10
977 Default instance parameters:
981 Default nic parameters:
986 There various parameters above can be changed via the ``modify``
989 - the hypervisor parameters can be changed via ``modify -H
990 xen-pvm:root_path=…``, and so on for other hypervisors/key/values
991 - the "default instance parameters" are changeable via ``modify -B
992 parameter=value…`` syntax
993 - the cluster parameters are changeable via separate options to the
994 modify command (e.g. ``--candidate-pool-size``, etc.)
996 For detailed option list see the :manpage:`gnt-cluster(8)` man page.
998 The cluster version can be obtained via the ``version`` command::
999 node1# gnt-cluster version
1000 Software version: 2.1.0
1001 Internode protocol: 20
1002 Configuration format: 2010000
1006 This is not very useful except when debugging Ganeti.
1008 Global node commands
1009 ++++++++++++++++++++
1011 There are two commands provided for replicating files to all nodes of a
1012 cluster and for running commands on all the nodes::
1014 node1# gnt-cluster copyfile /path/to/file
1015 node1# gnt-cluster command ls -l /path/to/file
1017 These are simple wrappers over scp/ssh and more advanced usage can be
1018 obtained using :manpage:`dsh(1)` and similar commands. But they are
1019 useful to update an OS script from the master node, for example.
1021 Cluster verification
1022 ++++++++++++++++++++
1024 There are three commands that relate to global cluster checks. The first
1025 one is ``verify`` which gives an overview on the cluster state,
1026 highlighting any issues. In normal operation, this command should return
1027 no ``ERROR`` messages::
1029 node1# gnt-cluster verify
1030 Sun Oct 25 23:08:58 2009 * Verifying global settings
1031 Sun Oct 25 23:08:58 2009 * Gathering data (2 nodes)
1032 Sun Oct 25 23:09:00 2009 * Verifying node status
1033 Sun Oct 25 23:09:00 2009 * Verifying instance status
1034 Sun Oct 25 23:09:00 2009 * Verifying orphan volumes
1035 Sun Oct 25 23:09:00 2009 * Verifying remaining instances
1036 Sun Oct 25 23:09:00 2009 * Verifying N+1 Memory redundancy
1037 Sun Oct 25 23:09:00 2009 * Other Notes
1038 Sun Oct 25 23:09:00 2009 - NOTICE: 5 non-redundant instance(s) found.
1039 Sun Oct 25 23:09:00 2009 * Hooks Results
1041 The second command is ``verify-disks``, which checks that the instance's
1042 disks have the correct status based on the desired instance state
1045 node1# gnt-cluster verify-disks
1047 Note that this command will show no output when disks are healthy.
1049 The last command is used to repair any discrepancies in Ganeti's
1050 recorded disk size and the actual disk size (disk size information is
1051 needed for proper activation and growth of DRBD-based disks)::
1053 node1# gnt-cluster repair-disk-sizes
1054 Sun Oct 25 23:13:16 2009 - INFO: Disk 0 of instance instance1 has mismatched size, correcting: recorded 512, actual 2048
1055 Sun Oct 25 23:13:17 2009 - WARNING: Invalid result from node node4, ignoring node results
1057 The above shows one instance having wrong disk size, and a node which
1058 returned invalid data, and thus we ignored all primary instances of that
1061 Configuration redistribution
1062 ++++++++++++++++++++++++++++
1064 If the verify command complains about file mismatches between the master
1065 and other nodes, due to some node problems or if you manually modified
1066 configuration files, you can force an push of the master configuration
1067 to all other nodes via the ``redist-conf`` command::
1069 node1# gnt-cluster redist-conf
1072 This command will be silent unless there are problems sending updates to
1079 It is possible to rename a cluster, or to change its IP address, via the
1080 ``rename`` command. If only the IP has changed, you need to pass the
1081 current name and Ganeti will realise its IP has changed::
1083 node1# gnt-cluster rename cluster.example.com
1084 This will rename the cluster to 'cluster.example.com'. If
1085 you are connected over the network to the cluster name, the operation
1086 is very dangerous as the IP address will be removed from the node and
1087 the change may not go through. Continue?
1089 Failure: prerequisites not met for this operation:
1090 Neither the name nor the IP address of the cluster has changed
1092 In the above output, neither value has changed since the cluster
1093 initialisation so the operation is not completed.
1098 The job queue execution in Ganeti 2.0 and higher can be inspected,
1099 suspended and resumed via the ``queue`` command::
1101 node1~# gnt-cluster queue info
1102 The drain flag is unset
1103 node1~# gnt-cluster queue drain
1104 node1~# gnt-instance stop instance1
1105 Failed to submit job for instance1: Job queue is drained, refusing job
1106 node1~# gnt-cluster queue info
1107 The drain flag is set
1108 node1~# gnt-cluster queue undrain
1110 This is most useful if you have an active cluster and you need to
1111 upgrade the Ganeti software, or simply restart the software on any node:
1113 #. suspend the queue via ``queue drain``
1114 #. wait until there are no more running jobs via ``gnt-job list``
1115 #. restart the master or another node, or upgrade the software
1116 #. resume the queue via ``queue undrain``
1118 .. note:: this command only stores a local flag file, and if you
1119 failover the master, it will not have effect on the new master.
1125 The :manpage:`ganeti-watcher` is a program, usually scheduled via
1126 ``cron``, that takes care of cluster maintenance operations (restarting
1127 downed instances, activating down DRBD disks, etc.). However, during
1128 maintenance and troubleshooting, this can get in your way; disabling it
1129 via commenting out the cron job is not so good as this can be
1130 forgotten. Thus there are some commands for automated control of the
1131 watcher: ``pause``, ``info`` and ``continue``::
1133 node1~# gnt-cluster watcher info
1134 The watcher is not paused.
1135 node1~# gnt-cluster watcher pause 1h
1136 The watcher is paused until Mon Oct 26 00:30:37 2009.
1137 node1~# gnt-cluster watcher info
1138 The watcher is paused until Mon Oct 26 00:30:37 2009.
1139 node1~# ganeti-watcher -d
1140 2009-10-25 23:30:47,984: pid=28867 ganeti-watcher:486 DEBUG Pause has been set, exiting
1141 node1~# gnt-cluster watcher continue
1142 The watcher is no longer paused.
1143 node1~# ganeti-watcher -d
1144 2009-10-25 23:31:04,789: pid=28976 ganeti-watcher:345 DEBUG Archived 0 jobs, left 0
1145 2009-10-25 23:31:05,884: pid=28976 ganeti-watcher:280 DEBUG Got data from cluster, writing instance status file
1146 2009-10-25 23:31:06,061: pid=28976 ganeti-watcher:150 DEBUG Data didn't change, just touching status file
1147 node1~# gnt-cluster watcher info
1148 The watcher is not paused.
1151 The exact details of the argument to the ``pause`` command are available
1154 .. note:: this command only stores a local flag file, and if you
1155 failover the master, it will not have effect on the new master.
1157 Node auto-maintenance
1158 +++++++++++++++++++++
1160 If the cluster parameter ``maintain_node_health`` is enabled (see the
1161 manpage for :command:`gnt-cluster`, the init and modify subcommands),
1162 then the following will happen automatically:
1164 - the watcher will shutdown any instances running on offline nodes
1165 - the watcher will deactivate any DRBD devices on offline nodes
1167 In the future, more actions are planned, so only enable this parameter
1168 if the nodes are completely dedicated to Ganeti; otherwise it might be
1169 possible to lose data due to auto-maintenance actions.
1171 Removing a cluster entirely
1172 +++++++++++++++++++++++++++
1174 The usual method to cleanup a cluster is to run ``gnt-cluster destroy``
1175 however if the Ganeti installation is broken in any way then this will
1178 It is possible in such a case to cleanup manually most if not all traces
1179 of a cluster installation by following these steps on all of the nodes:
1181 1. Shutdown all instances. This depends on the virtualisation method
1182 used (Xen, KVM, etc.):
1184 - Xen: run ``xm list`` and ``xm destroy`` on all the non-Domain-0
1186 - KVM: kill all the KVM processes
1187 - chroot: kill all processes under the chroot mountpoints
1189 2. If using DRBD, shutdown all DRBD minors (which should by at this time
1190 no-longer in use by instances); on each node, run ``drbdsetup
1191 /dev/drbdN down`` for each active DRBD minor.
1193 3. If using LVM, cleanup the Ganeti volume group; if only Ganeti created
1194 logical volumes (and you are not sharing the volume group with the
1195 OS, for example), then simply running ``lvremove -f xenvg`` (replace
1196 'xenvg' with your volume group name) should do the required cleanup.
1198 4. If using file-based storage, remove recursively all files and
1199 directories under your file-storage directory: ``rm -rf
1200 /srv/ganeti/file-storage/*`` replacing the path with the correct path
1203 5. Stop the ganeti daemons (``/etc/init.d/ganeti stop``) and kill any
1204 that remain alive (``pgrep ganeti`` and ``pkill ganeti``).
1206 6. Remove the ganeti state directory (``rm -rf /var/lib/ganeti/*``),
1207 replacing the path with the correct path for your installation.
1209 On the master node, remove the cluster from the master-netdev (usually
1210 ``xen-br0`` for bridged mode, otherwise ``eth0`` or similar), by running
1211 ``ip a del $clusterip/32 dev xen-br0`` (use the correct cluster ip and
1212 network device name).
1214 At this point, the machines are ready for a cluster creation; in case
1215 you want to remove Ganeti completely, you need to also undo some of the
1216 SSH changes and log directories:
1218 - ``rm -rf /var/log/ganeti /srv/ganeti`` (replace with the correct
1220 - remove from ``/root/.ssh`` the keys that Ganeti added (check the
1221 ``authorized_keys`` and ``id_dsa`` files)
1222 - regenerate the host's SSH keys (check the OpenSSH startup scripts)
1225 Otherwise, if you plan to re-create the cluster, you can just go ahead
1226 and rerun ``gnt-cluster init``.
1231 The tags handling (addition, removal, listing) is similar for all the
1232 objects that support it (instances, nodes, and the cluster).
1237 Note that the set of characters present in a tag and the maximum tag
1238 length are restricted. Currently the maximum length is 128 characters,
1239 there can be at most 4096 tags per object, and the set of characters is
1240 comprised by alphanumeric characters and additionally ``.+*/:@-``.
1245 Tags can be added via ``add-tags``::
1247 gnt-instance add-tags INSTANCE a b c
1248 gnt-node add-tags INSTANCE a b c
1249 gnt-cluster add-tags a b c
1252 The above commands add three tags to an instance, to a node and to the
1253 cluster. Note that the cluster command only takes tags as arguments,
1254 whereas the node and instance commands first required the node and
1257 Tags can also be added from a file, via the ``--from=FILENAME``
1258 argument. The file is expected to contain one tag per line.
1260 Tags can also be remove via a syntax very similar to the add one::
1262 gnt-instance remove-tags INSTANCE a b c
1266 gnt-instance list-tags
1268 gnt-cluster list-tags
1273 It is also possible to execute a global search on the all tags defined
1274 in the cluster configuration, via a cluster command::
1276 gnt-cluster search-tags REGEXP
1278 The parameter expected is a regular expression (see
1279 :manpage:`regex(7)`). This will return all tags that match the search,
1280 together with the object they are defined in (the names being show in a
1281 hierarchical kind of way)::
1283 node1# gnt-cluster search-tags o
1285 /instances/instance1 owner:bar
1291 The various jobs submitted by the instance/node/cluster commands can be
1292 examined, canceled and archived by various invocations of the
1293 ``gnt-job`` command.
1295 First is the job list command::
1298 17771 success INSTANCE_QUERY_DATA
1299 17773 success CLUSTER_VERIFY_DISKS
1300 17775 success CLUSTER_REPAIR_DISK_SIZES
1301 17776 error CLUSTER_RENAME(cluster.example.com)
1302 17780 success CLUSTER_REDIST_CONF
1303 17792 success INSTANCE_REBOOT(instance1.example.com)
1305 More detailed information about a job can be found via the ``info``
1308 node1# gnt-job info 17776
1311 Received: 2009-10-25 23:18:02.180569
1312 Processing start: 2009-10-25 23:18:02.200335 (delta 0.019766s)
1313 Processing end: 2009-10-25 23:18:02.279743 (delta 0.079408s)
1314 Total processing time: 0.099174 seconds
1318 Processing start: 2009-10-25 23:18:02.200335
1319 Processing end: 2009-10-25 23:18:02.252282
1321 name: cluster.example.com
1324 [Neither the name nor the IP address of the cluster has changed]
1327 During the execution of a job, it's possible to follow the output of a
1328 job, similar to the log that one get from the ``gnt-`` commands, via the
1331 node1# gnt-instance add --submit … instance1
1333 node1# gnt-job watch 17818
1334 Output from job 17818 follows
1335 -----------------------------
1336 Mon Oct 26 00:22:48 2009 - INFO: Selected nodes for instance instance1 via iallocator dumb: node1, node2
1337 Mon Oct 26 00:22:49 2009 * creating instance disks...
1338 Mon Oct 26 00:22:52 2009 adding instance instance1 to cluster config
1339 Mon Oct 26 00:22:52 2009 - INFO: Waiting for instance instance1 to sync disks.
1341 Mon Oct 26 00:23:03 2009 creating os for instance instance1 on node node1
1342 Mon Oct 26 00:23:03 2009 * running the instance OS create scripts...
1343 Mon Oct 26 00:23:13 2009 * starting instance...
1346 This is useful if you need to follow a job's progress from multiple
1349 A job that has not yet started to run can be canceled::
1351 node1# gnt-job cancel 17810
1353 But not one that has already started execution::
1355 node1# gnt-job cancel 17805
1356 Job 17805 is no longer waiting in the queue
1358 There are two queues for jobs: the *current* and the *archive*
1359 queue. Jobs are initially submitted to the current queue, and they stay
1360 in that queue until they have finished execution (either successfully or
1361 not). At that point, they can be moved into the archive queue, and the
1362 ganeti-watcher script will do this automatically after 6 hours. The
1363 ganeti-cleaner script will remove the jobs from the archive directory
1366 Note that only jobs in the current queue can be viewed via the list and
1367 info commands; Ganeti itself doesn't examine the archive directory. If
1368 you need to see an older job, either move the file manually in the
1369 top-level queue directory, or look at its contents (it's a
1370 JSON-formatted file).
1372 Special Ganeti deployments
1373 --------------------------
1375 Since Ganeti 2.4, it is possible to extend the Ganeti deployment with
1376 two custom scenarios: Ganeti inside Ganeti and multi-site model.
1378 Running Ganeti under Ganeti
1379 +++++++++++++++++++++++++++
1381 It is sometimes useful to be able to use a Ganeti instance as a Ganeti
1382 node (part of another cluster, usually). One example scenario is two
1383 small clusters, where we want to have an additional master candidate
1384 that holds the cluster configuration and can be used for helping with
1385 the master voting process.
1387 However, these Ganeti instance should not host instances themselves, and
1388 should not be considered in the normal capacity planning, evacuation
1389 strategies, etc. In order to accomplish this, mark these nodes as
1390 non-``vm_capable``::
1392 node1# gnt-node modify --vm-capable=no node3
1394 The vm_capable status can be listed as usual via ``gnt-node list``::
1396 node1# gnt-node list -oname,vm_capable
1402 When this flag is set, the cluster will not do any operations that
1403 relate to instances on such nodes, e.g. hypervisor operations,
1404 disk-related operations, etc. Basically they will just keep the ssconf
1405 files, and if master candidates the full configuration.
1410 If Ganeti is deployed in multi-site model, with each site being a node
1411 group (so that instances are not relocated across the WAN by mistake),
1412 it is conceivable that either the WAN latency is high or that some sites
1413 have a lower reliability than others. In this case, it doesn't make
1414 sense to replicate the job information across all sites (or even outside
1415 of a “central” node group), so it should be possible to restrict which
1416 nodes can become master candidates via the auto-promotion algorithm.
1418 Ganeti 2.4 introduces for this purpose a new ``master_capable`` flag,
1419 which (when unset) prevents nodes from being marked as master
1420 candidates, either manually or automatically.
1422 As usual, the node modify operation can change this flag::
1424 node1# gnt-node modify --auto-promote --master-capable=no node3
1425 Fri Jan 7 06:23:07 2011 - INFO: Demoting from master candidate
1426 Fri Jan 7 06:23:08 2011 - INFO: Promoted nodes to master candidate role: node4
1428 - master_capable -> False
1429 - master_candidate -> False
1431 And the node list operation will list this flag::
1433 node1# gnt-node list -oname,master_capable node1 node2 node3
1439 Note that marking a node both not ``vm_capable`` and not
1440 ``master_capable`` makes the node practically unusable from Ganeti's
1441 point of view. Hence these two flags should be used probably in
1442 contrast: some nodes will be only master candidates (master_capable but
1443 not vm_capable), and other nodes will only hold instances (vm_capable
1444 but not master_capable).
1450 Beside the usual ``gnt-`` and ``ganeti-`` commands which are provided
1451 and installed in ``$prefix/sbin`` at install time, there are a couple of
1452 other tools installed which are used seldom but can be helpful in some
1458 The ``lvmstrap`` tool, introduced in :ref:`configure-lvm-label` section,
1459 has two modes of operation:
1461 - ``diskinfo`` shows the discovered disks on the system and their status
1462 - ``create`` takes all not-in-use disks and creates a volume group out
1465 .. warning:: The ``create`` argument to this command causes data-loss!
1470 The ``cfgupgrade`` tools is used to upgrade between major (and minor)
1471 Ganeti versions. Point-releases are usually transparent for the admin.
1473 More information about the upgrade procedure is listed on the wiki at
1474 http://code.google.com/p/ganeti/wiki/UpgradeNotes.
1476 There is also a script designed to upgrade from Ganeti 1.2 to 2.0,
1477 called ``cfgupgrade12``.
1482 .. note:: This command is not actively maintained; make sure you backup
1483 your configuration before using it
1485 This can be used as an alternative to direct editing of the
1486 main configuration file if Ganeti has a bug and prevents you, for
1487 example, from removing an instance or a node from the configuration
1495 .. warning:: This command will erase existing instances if given as
1498 This tool is used to exercise either the hardware of machines or
1499 alternatively the Ganeti software. It is safe to run on an existing
1500 cluster **as long as you don't pass it existing instance names**.
1502 The command will, by default, execute a comprehensive set of operations
1503 against a list of instances, these being:
1506 - disk replacement (for redundant instances)
1507 - failover and migration (for redundant instances)
1508 - move (for non-redundant instances)
1510 - add disks, remove disk
1511 - add NICs, remove NICs
1512 - export and then import
1516 - and finally removal of the test instances
1518 Executing all these operations will test that the hardware performs
1519 well: the creation, disk replace, disk add and disk growth will exercise
1520 the storage and network; the migrate command will test the memory of the
1521 systems. Depending on the passed options, it can also test that the
1522 instance OS definitions are executing properly the rename, import and
1528 This tool takes the Ganeti configuration and outputs a "sanitized"
1529 version, by randomizing or clearing:
1531 - DRBD secrets and cluster public key (always)
1532 - host names (optional)
1534 - OS names (optional)
1535 - LV names (optional, only useful for very old clusters which still have
1536 instances whose LVs are based on the instance name)
1538 By default, all optional items are activated except the LV name
1539 randomization. When passing ``--no-randomization``, which disables the
1540 optional items (i.e. just the DRBD secrets and cluster public keys are
1541 randomized), the resulting file can be used as a safety copy of the
1542 cluster config - while not trivial, the layout of the cluster can be
1543 recreated from it and if the instance disks have not been lost it
1544 permits recovery from the loss of all master candidates.
1549 See :doc:`separate documentation for move-instance <move-instance>`.
1551 .. TODO: document cluster-merge tool
1554 Other Ganeti projects
1555 ---------------------
1557 Below is a list (which might not be up-to-date) of additional projects
1558 that can be useful in a Ganeti deployment. They can be downloaded from
1559 the project site (http://code.google.com/p/ganeti/) and the repositories
1560 are also on the project git site (http://git.ganeti.org).
1565 The ``ganeti-nbma`` software is designed to allow instances to live on a
1566 separate, virtual network from the nodes, and in an environment where
1567 nodes are not guaranteed to be able to reach each other via multicasting
1568 or broadcasting. For more information see the README in the source
1574 Before Ganeti version 2.5, this was a standalone project; since that
1575 version it is integrated into the Ganeti codebase (see
1576 :doc:`install-quick` for instructions on how to enable it). If you run
1577 an older Ganeti version, you will have to download and build it
1580 For more information and installation instructions, see the README file
1581 in the source archive.
1583 .. vim: set textwidth=72 :