Statistics
| Branch: | Tag: | Revision:

root / doc / admin.rst @ bbf74a76

History | View | Annotate | Download (47.2 kB)

1
Ganeti administrator's guide
2
============================
3

    
4
Documents Ganeti version |version|
5

    
6
.. contents::
7

    
8
.. highlight:: text
9

    
10
Introduction
11
------------
12

    
13
Ganeti is a virtualization cluster management software. You are expected
14
to be a system administrator familiar with your Linux distribution and
15
the Xen or KVM virtualization environments before using it.
16

    
17
The various components of Ganeti all have man pages and interactive
18
help. This manual though will help you getting familiar with the system
19
by explaining the most common operations, grouped by related use.
20

    
21
After a terminology glossary and a section on the prerequisites needed
22
to use this manual, the rest of this document is divided in sections
23
for the different targets that a command affects: instance, nodes, etc.
24

    
25
.. _terminology-label:
26

    
27
Ganeti terminology
28
++++++++++++++++++
29

    
30
This section provides a small introduction to Ganeti terminology, which
31
might be useful when reading the rest of the document.
32

    
33
Cluster
34
~~~~~~~
35

    
36
A set of machines (nodes) that cooperate to offer a coherent, highly
37
available virtualization service under a single administration domain.
38

    
39
Node
40
~~~~
41

    
42
A physical machine which is member of a cluster.  Nodes are the basic
43
cluster infrastructure, and they don't need to be fault tolerant in
44
order to achieve high availability for instances.
45

    
46
Node can be added and removed (if they host no instances) at will from
47
the cluster. In a HA cluster and only with HA instances, the loss of any
48
single node will not cause disk data loss for any instance; of course,
49
a node crash will cause the crash of the its primary instances.
50

    
51
A node belonging to a cluster can be in one of the following roles at a
52
given time:
53

    
54
- *master* node, which is the node from which the cluster is controlled
55
- *master candidate* node, only nodes in this role have the full cluster
56
  configuration and knowledge, and only master candidates can become the
57
  master node
58
- *regular* node, which is the state in which most nodes will be on
59
  bigger clusters (>20 nodes)
60
- *drained* node, nodes in this state are functioning normally but the
61
  cannot receive new instances; the intention is that nodes in this role
62
  have some issue and they are being evacuated for hardware repairs
63
- *offline* node, in which there is a record in the cluster
64
  configuration about the node, but the daemons on the master node will
65
  not talk to this node; any instances declared as having an offline
66
  node as either primary or secondary will be flagged as an error in the
67
  cluster verify operation
68

    
69
Depending on the role, each node will run a set of daemons:
70

    
71
- the :command:`ganeti-noded` daemon, which control the manipulation of
72
  this node's hardware resources; it runs on all nodes which are in a
73
  cluster
74
- the :command:`ganeti-confd` daemon (Ganeti 2.1+) which runs on all
75
  nodes, but is only functional on master candidate nodes
76
- the :command:`ganeti-rapi` daemon which runs on the master node and
77
  offers an HTTP-based API for the cluster
78
- the :command:`ganeti-masterd` daemon which runs on the master node and
79
  allows control of the cluster
80

    
81
Instance
82
~~~~~~~~
83

    
84
A virtual machine which runs on a cluster. It can be a fault tolerant,
85
highly available entity.
86

    
87
An instance has various parameters, which are classified in three
88
categories: hypervisor related-parameters (called ``hvparams``), general
89
parameters (called ``beparams``) and per network-card parameters (called
90
``nicparams``). All these parameters can be modified either at instance
91
level or via defaults at cluster level.
92

    
93
Disk template
94
~~~~~~~~~~~~~
95

    
96
The are multiple options for the storage provided to an instance; while
97
the instance sees the same virtual drive in all cases, the node-level
98
configuration varies between them.
99

    
100
There are four disk templates you can choose from:
101

    
102
diskless
103
  The instance has no disks. Only used for special purpose operating
104
  systems or for testing.
105

    
106
file
107
  The instance will use plain files as backend for its disks. No
108
  redundancy is provided, and this is somewhat more difficult to
109
  configure for high performance.
110

    
111
plain
112
  The instance will use LVM devices as backend for its disks. No
113
  redundancy is provided.
114

    
115
drbd
116
  .. note:: This is only valid for multi-node clusters using DRBD 8.0+
117

    
118
  A mirror is set between the local node and a remote one, which must be
119
  specified with the second value of the --node option. Use this option
120
  to obtain a highly available instance that can be failed over to a
121
  remote node should the primary one fail.
122

    
123
IAllocator
124
~~~~~~~~~~
125

    
126
A framework for using external (user-provided) scripts to compute the
127
placement of instances on the cluster nodes. This eliminates the need to
128
manually specify nodes in instance add, instance moves, node evacuate,
129
etc.
130

    
131
In order for Ganeti to be able to use these scripts, they must be place
132
in the iallocator directory (usually ``lib/ganeti/iallocators`` under
133
the installation prefix, e.g. ``/usr/local``).
134

    
135
“Primary” and “secondary” concepts
136
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
137

    
138
An instance has a primary and depending on the disk configuration, might
139
also have a secondary node. The instance always runs on the primary node
140
and only uses its secondary node for disk replication.
141

    
142
Similarly, the term of primary and secondary instances when talking
143
about a node refers to the set of instances having the given node as
144
primary, respectively secondary.
145

    
146
Tags
147
~~~~
148

    
149
Tags are short strings that can be attached to either to cluster itself,
150
or to nodes or instances. They are useful as a very simplistic
151
information store for helping with cluster administration, for example
152
by attaching owner information to each instance after it's created::
153

    
154
  gnt-instance add … instance1
155
  gnt-instance add-tags instance1 owner:user2
156

    
157
And then by listing each instance and its tags, this information could
158
be used for contacting the users of each instance.
159

    
160
Jobs and OpCodes
161
~~~~~~~~~~~~~~~~
162

    
163
While not directly visible by an end-user, it's useful to know that a
164
basic cluster operation (e.g. starting an instance) is represented
165
internall by Ganeti as an *OpCode* (abbreviation from operation
166
code). These OpCodes are executed as part of a *Job*. The OpCodes in a
167
single Job are processed serially by Ganeti, but different Jobs will be
168
processed (depending on resource availability) in parallel.
169

    
170
For example, shutting down the entire cluster can be done by running the
171
command ``gnt-instance shutdown --all``, which will submit for each
172
instance a separate job containing the “shutdown instance” OpCode.
173

    
174

    
175
Prerequisites
176
+++++++++++++
177

    
178
You need to have your Ganeti cluster installed and configured before you
179
try any of the commands in this document. Please follow the
180
:doc:`install` for instructions on how to do that.
181

    
182
Instance management
183
-------------------
184

    
185
Adding an instance
186
++++++++++++++++++
187

    
188
The add operation might seem complex due to the many parameters it
189
accepts, but once you have understood the (few) required parameters and
190
the customisation capabilities you will see it is an easy operation.
191

    
192
The add operation requires at minimum five parameters:
193

    
194
- the OS for the instance
195
- the disk template
196
- the disk count and size
197
- the node specification or alternatively the iallocator to use
198
- and finally the instance name
199

    
200
The OS for the instance must be visible in the output of the command
201
``gnt-os list`` and specifies which guest OS to install on the instance.
202

    
203
The disk template specifies what kind of storage to use as backend for
204
the (virtual) disks presented to the instance; note that for instances
205
with multiple virtual disks, they all must be of the same type.
206

    
207
The node(s) on which the instance will run can be given either manually,
208
via the ``-n`` option, or computed automatically by Ganeti, if you have
209
installed any iallocator script.
210

    
211
With the above parameters in mind, the command is::
212

    
213
  gnt-instance add \
214
    -n TARGET_NODE:SECONDARY_NODE \
215
    -o OS_TYPE \
216
    -t DISK_TEMPLATE -s DISK_SIZE \
217
    INSTANCE_NAME
218

    
219
The instance name must be resolvable (e.g. exist in DNS) and usually
220
points to an address in the same subnet as the cluster itself.
221

    
222
The above command has the minimum required options; other options you
223
can give include, among others:
224

    
225
- The memory size (``-B memory``)
226

    
227
- The number of virtual CPUs (``-B vcpus``)
228

    
229
- Arguments for the NICs of the instance; by default, a single-NIC
230
  instance is created. The IP and/or bridge of the NIC can be changed
231
  via ``--nic 0:ip=IP,bridge=BRIDGE``
232

    
233
See the manpage for gnt-instance for the detailed option list.
234

    
235
For example if you want to create an highly available instance, with a
236
single disk of 50GB and the default memory size, having primary node
237
``node1`` and secondary node ``node3``, use the following command::
238

    
239
  gnt-instance add -n node1:node3 -o debootstrap -t drbd \
240
    instance1
241

    
242
There is a also a command for batch instance creation from a
243
specification file, see the ``batch-create`` operation in the
244
gnt-instance manual page.
245

    
246
Regular instance operations
247
+++++++++++++++++++++++++++
248

    
249
Removal
250
~~~~~~~
251

    
252
Removing an instance is even easier than creating one. This operation is
253
irreversible and destroys all the contents of your instance. Use with
254
care::
255

    
256
  gnt-instance remove INSTANCE_NAME
257

    
258
Startup/shutdown
259
~~~~~~~~~~~~~~~~
260

    
261
Instances are automatically started at instance creation time. To
262
manually start one which is currently stopped you can run::
263

    
264
  gnt-instance startup INSTANCE_NAME
265

    
266
While the command to stop one is::
267

    
268
  gnt-instance shutdown INSTANCE_NAME
269

    
270
.. warning:: Do not use the Xen or KVM commands directly to stop
271
   instances. If you run for example ``xm shutdown`` or ``xm destroy``
272
   on an instance Ganeti will automatically restart it (via the
273
   :command:`ganeti-watcher` command which is launched via cron).
274

    
275
Querying instances
276
~~~~~~~~~~~~~~~~~~
277

    
278
There are two ways to get information about instances: listing
279
instances, which does a tabular output containing a given set of fields
280
about each instance, and querying detailed information about a set of
281
instances.
282

    
283
The command to see all the instances configured and their status is::
284

    
285
  gnt-instance list
286

    
287
The command can return a custom set of information when using the ``-o``
288
option (as always, check the manpage for a detailed specification). Each
289
instance will be represented on a line, thus making it easy to parse
290
this output via the usual shell utilities (grep, sed, etc.).
291

    
292
To get more detailed information about an instance, you can run::
293

    
294
  gnt-instance info INSTANCE
295

    
296
which will give a multi-line block of information about the instance,
297
it's hardware resources (especially its disks and their redundancy
298
status), etc. This is harder to parse and is more expensive than the
299
list operation, but returns much more detailed information.
300

    
301

    
302
Export/Import
303
+++++++++++++
304

    
305
You can create a snapshot of an instance disk and its Ganeti
306
configuration, which then you can backup, or import into another
307
cluster. The way to export an instance is::
308

    
309
  gnt-backup export -n TARGET_NODE INSTANCE_NAME
310

    
311

    
312
The target node can be any node in the cluster with enough space under
313
``/srv/ganeti`` to hold the instance image. Use the ``--noshutdown``
314
option to snapshot an instance without rebooting it. Note that Ganeti
315
only keeps one snapshot for an instance - any previous snapshot of the
316
same instance existing cluster-wide under ``/srv/ganeti`` will be
317
removed by this operation: if you want to keep them, you need to move
318
them out of the Ganeti exports directory.
319

    
320
Importing an instance is similar to creating a new one, but additionally
321
one must specify the location of the snapshot. The command is::
322

    
323
  gnt-backup import -n TARGET_NODE -t DISK_TEMPLATE \
324
    --src-node=NODE --src-dir=DIR INSTANCE_NAME
325

    
326
Most of the options available for the command :command:`gnt-instance
327
add` are supported here too.
328

    
329
Import of foreign instances
330
+++++++++++++++++++++++++++
331

    
332
There is a possibility to import a foreign instance whose disk data is
333
already stored as LVM volumes without going through copying it: the disk
334
adoption mode.
335

    
336
For this, ensure that the original, non-managed instance is stopped,
337
then create a Ganeti instance in the usual way, except that instead of
338
passing the disk information you specify the current volumes::
339

    
340
  gnt-instance add -t plain -n HOME_NODE ... \
341
    --disk 0:adopt=lv_name INSTANCE_NAME
342

    
343
This will take over the given logical volumes, rename them to the Ganeti
344
standard (UUID-based), and without installing the OS on them start
345
directly the instance. If you configure the hypervisor similar to the
346
non-managed configuration that the instance had, the transition should
347
be seamless for the instance. For more than one disk, just pass another
348
disk parameter (e.g. ``--disk 1:adopt=...``).
349

    
350
Instance HA features
351
--------------------
352

    
353
.. note:: This section only applies to multi-node clusters
354

    
355
.. _instance-change-primary-label:
356

    
357
Changing the primary node
358
+++++++++++++++++++++++++
359

    
360
There are three ways to exchange an instance's primary and secondary
361
nodes; the right one to choose depends on how the instance has been
362
created and the status of its current primary node. See
363
:ref:`rest-redundancy-label` for information on changing the secondary
364
node. Note that it's only possible to change the primary node to the
365
secondary and vice-versa; a direct change of the primary node with a
366
third node, while keeping the current secondary is not possible in a
367
single step, only via multiple operations as detailed in
368
:ref:`instance-relocation-label`.
369

    
370
Failing over an instance
371
~~~~~~~~~~~~~~~~~~~~~~~~
372

    
373
If an instance is built in highly available mode you can at any time
374
fail it over to its secondary node, even if the primary has somehow
375
failed and it's not up anymore. Doing it is really easy, on the master
376
node you can just run::
377

    
378
  gnt-instance failover INSTANCE_NAME
379

    
380
That's it. After the command completes the secondary node is now the
381
primary, and vice-versa.
382

    
383
Live migrating an instance
384
~~~~~~~~~~~~~~~~~~~~~~~~~~
385

    
386
If an instance is built in highly available mode, it currently runs and
387
both its nodes are running fine, you can at migrate it over to its
388
secondary node, without downtime. On the master node you need to run::
389

    
390
  gnt-instance migrate INSTANCE_NAME
391

    
392
The current load on the instance and its memory size will influence how
393
long the migration will take. In any case, for both KVM and Xen
394
hypervisors, the migration will be transparent to the instance.
395

    
396
Moving an instance (offline)
397
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
398

    
399
If an instance has not been create as mirrored, then the only way to
400
change its primary node is to execute the move command::
401

    
402
  gnt-instance move -n NEW_NODE INSTANCE
403

    
404
This has a few prerequisites:
405

    
406
- the instance must be stopped
407
- its current primary node must be on-line and healthy
408
- the disks of the instance must not have any errors
409

    
410
Since this operation actually copies the data from the old node to the
411
new node, expect it to take proportional to the size of the instance's
412
disks and the speed of both the nodes' I/O system and their networking.
413

    
414
Disk operations
415
+++++++++++++++
416

    
417
Disk failures are a common cause of errors in any server
418
deployment. Ganeti offers protection from single-node failure if your
419
instances were created in HA mode, and it also offers ways to restore
420
redundancy after a failure.
421

    
422
Preparing for disk operations
423
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
424

    
425
It is important to note that for Ganeti to be able to do any disk
426
operation, the Linux machines on top of which Ganeti must be consistent;
427
for LVM, this means that the LVM commands must not return failures; it
428
is common that after a complete disk failure, any LVM command aborts
429
with an error similar to::
430

    
431
  # vgs
432
  /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
433
  /dev/sdb1: read failed after 0 of 4096 at 750153695232: Input/output
434
  error
435
  /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
436
  Couldn't find device with uuid
437
  't30jmN-4Rcf-Fr5e-CURS-pawt-z0jU-m1TgeJ'.
438
  Couldn't find all physical volumes for volume group xenvg.
439

    
440
Before restoring an instance's disks to healthy status, it's needed to
441
fix the volume group used by Ganeti so that we can actually create and
442
manage the logical volumes. This is usually done in a multi-step
443
process:
444

    
445
#. first, if the disk is completely gone and LVM commands exit with
446
   “Couldn't find device with uuid…” then you need to run the command::
447

    
448
    vgreduce --removemissing VOLUME_GROUP
449

    
450
#. after the above command, the LVM commands should be executing
451
   normally (warnings are normal, but the commands will not fail
452
   completely).
453

    
454
#. if the failed disk is still visible in the output of the ``pvs``
455
   command, you need to deactivate it from allocations by running::
456

    
457
    pvs -x n /dev/DISK
458

    
459
At this point, the volume group should be consistent and any bad
460
physical volumes should not longer be available for allocation.
461

    
462
Note that since version 2.1 Ganeti provides some commands to automate
463
these two operations, see :ref:`storage-units-label`.
464

    
465
.. _rest-redundancy-label:
466

    
467
Restoring redundancy for DRBD-based instances
468
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
469

    
470
A DRBD instance has two nodes, and the storage on one of them has
471
failed. Depending on which node (primary or secondary) has failed, you
472
have three options at hand:
473

    
474
- if the storage on the primary node has failed, you need to re-create
475
  the disks on it
476
- if the storage on the secondary node has failed, you can either
477
  re-create the disks on it or change the secondary and recreate
478
  redundancy on the new secondary node
479

    
480
Of course, at any point it's possible to force re-creation of disks even
481
though everything is already fine.
482

    
483
For all three cases, the ``replace-disks`` operation can be used::
484

    
485
  # re-create disks on the primary node
486
  gnt-instance replace-disks -p INSTANCE_NAME
487
  # re-create disks on the current secondary
488
  gnt-instance replace-disks -s INSTANCE_NAME
489
  # change the secondary node, via manual specification
490
  gnt-instance replace-disks -n NODE INSTANCE_NAME
491
  # change the secondary node, via an iallocator script
492
  gnt-instance replace-disks -I SCRIPT INSTANCE_NAME
493
  # since Ganeti 2.1: automatically fix the primary or secondary node
494
  gnt-instance replace-disks -a INSTANCE_NAME
495

    
496
Since the process involves copying all data from the working node to the
497
target node, it will take a while, depending on the instance's disk
498
size, node I/O system and network speed. But it is (baring any network
499
interruption) completely transparent for the instance.
500

    
501
Re-creating disks for non-redundant instances
502
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
503

    
504
.. versionadded:: 2.1
505

    
506
For non-redundant instances, there isn't a copy (except backups) to
507
re-create the disks. But it's possible to at-least re-create empty
508
disks, after which a reinstall can be run, via the ``recreate-disks``
509
command::
510

    
511
  gnt-instance recreate-disks INSTANCE
512

    
513
Note that this will fail if the disks already exists.
514

    
515
Conversion of an instance's disk type
516
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
517

    
518
It is possible to convert between a non-redundant instance of type
519
``plain`` (LVM storage) and redundant ``drbd`` via the ``gnt-instance
520
modify`` command::
521

    
522
  # start with a non-redundant instance
523
  gnt-instance add -t plain ... INSTANCE
524

    
525
  # later convert it to redundant
526
  gnt-instance stop INSTANCE
527
  gnt-instance modify -t drbd INSTANCE
528
  gnt-instance start INSTANCE
529

    
530
  # and convert it back
531
  gnt-instance stop INSTANCE
532
  gnt-instance modify -t plain INSTANCE
533
  gnt-instance start INSTANCE
534

    
535
The conversion must be done while the instance is stopped, and
536
converting from plain to drbd template presents a small risk, especially
537
if the instance has multiple disks and/or if one node fails during the
538
conversion procedure). As such, it's recommended (as always) to make
539
sure that downtime for manual recovery is acceptable and that the
540
instance has up-to-date backups.
541

    
542
Debugging instances
543
+++++++++++++++++++
544

    
545
Accessing an instance's disks
546
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
547

    
548
From an instance's primary node you can have access to its disks. Never
549
ever mount the underlying logical volume manually on a fault tolerant
550
instance, or will break replication and your data will be
551
inconsistent. The correct way to access an instance's disks is to run
552
(on the master node, as usual) the command::
553

    
554
  gnt-instance activate-disks INSTANCE
555

    
556
And then, *on the primary node of the instance*, access the device that
557
gets created. For example, you could mount the given disks, then edit
558
files on the filesystem, etc.
559

    
560
Note that with partitioned disks (as opposed to whole-disk filesystems),
561
you will need to use a tool like :manpage:`kpartx(8)`::
562

    
563
  node1# gnt-instance activate-disks instance1
564
565
  node1# ssh node3
566
  node3# kpartx -l /dev/…
567
  node3# kpartx -a /dev/…
568
  node3# mount /dev/mapper/… /mnt/
569
  # edit files under mnt as desired
570
  node3# umount /mnt/
571
  node3# kpartx -d /dev/…
572
  node3# exit
573
  node1#
574

    
575
After you've finished you can deactivate them with the deactivate-disks
576
command, which works in the same way::
577

    
578
  gnt-instance deactivate-disks INSTANCE
579

    
580
Note that if any process started by you is still using the disks, the
581
above command will error out, and you **must** cleanup and ensure that
582
the above command runs successfully before you start the instance,
583
otherwise the instance will suffer corruption.
584

    
585
Accessing an instance's console
586
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
587

    
588
The command to access a running instance's console is::
589

    
590
  gnt-instance console INSTANCE_NAME
591

    
592
Use the console normally and then type ``^]`` when done, to exit.
593

    
594
Other instance operations
595
+++++++++++++++++++++++++
596

    
597
Reboot
598
~~~~~~
599

    
600
There is a wrapper command for rebooting instances::
601

    
602
  gnt-instance reboot instance2
603

    
604
By default, this does the equivalent of shutting down and then starting
605
the instance, but it accepts parameters to perform a soft-reboot (via
606
the hypervisor), a hard reboot (hypervisor shutdown and then startup) or
607
a full one (the default, which also de-configures and then configures
608
again the disks of the instance).
609

    
610
Instance OS definitions debugging
611
+++++++++++++++++++++++++++++++++
612

    
613
Should you have any problems with instance operating systems the command
614
to see a complete status for all your nodes is::
615

    
616
   gnt-os diagnose
617

    
618
.. _instance-relocation-label:
619

    
620
Instance relocation
621
~~~~~~~~~~~~~~~~~~~
622

    
623
While it is not possible to move an instance from nodes ``(A, B)`` to
624
nodes ``(C, D)`` in a single move, it is possible to do so in a few
625
steps::
626

    
627
  # instance is located on A, B
628
  node1# gnt-instance replace -n nodeC instance1
629
  # instance has moved from (A, B) to (A, C)
630
  # we now flip the primary/secondary nodes
631
  node1# gnt-instance migrate instance1
632
  # instance lives on (C, A)
633
  # we can then change A to D via:
634
  node1# gnt-instance replace -n nodeD instance1
635

    
636
Which brings it into the final configuration of ``(C, D)``. Note that we
637
needed to do two replace-disks operation (two copies of the instance
638
disks), because we needed to get rid of both the original nodes (A and
639
B).
640

    
641
Node operations
642
---------------
643

    
644
There are much fewer node operations available than for instances, but
645
they are equivalently important for maintaining a healthy cluster.
646

    
647
Add/readd
648
+++++++++
649

    
650
It is at any time possible to extend the cluster with one more node, by
651
using the node add operation::
652

    
653
  gnt-node add NEW_NODE
654

    
655
If the cluster has a replication network defined, then you need to pass
656
the ``-s REPLICATION_IP`` parameter to this option.
657

    
658
A variation of this command can be used to re-configure a node if its
659
Ganeti configuration is broken, for example if it has been reinstalled
660
by mistake::
661

    
662
  gnt-node add --readd EXISTING_NODE
663

    
664
This will reinitialise the node as if it's been newly added, but while
665
keeping its existing configuration in the cluster (primary/secondary IP,
666
etc.), in other words you won't need to use ``-s`` here.
667

    
668
Changing the node role
669
++++++++++++++++++++++
670

    
671
A node can be in different roles, as explained in the
672
:ref:`terminology-label` section. Promoting a node to the master role is
673
special, while the other roles are handled all via a single command.
674

    
675
Failing over the master node
676
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
677

    
678
If you want to promote a different node to the master role (for whatever
679
reason), run on any other master-candidate node the command::
680

    
681
  gnt-cluster masterfailover
682

    
683
and the node you ran it on is now the new master. In case you try to run
684
this on a non master-candidate node, you will get an error telling you
685
which nodes are valid.
686

    
687
Changing between the other roles
688
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
689

    
690
The ``gnt-node modify`` command can be used to select a new role::
691

    
692
  # change to master candidate
693
  gnt-node modify -C yes NODE
694
  # change to drained status
695
  gnt-node modify -D yes NODE
696
  # change to offline status
697
  gnt-node modify -O yes NODE
698
  # change to regular mode (reset all flags)
699
  gnt-node modify -O no -D no -C no NODE
700

    
701
Note that the cluster requires that at any point in time, a certain
702
number of nodes are master candidates, so changing from master candidate
703
to other roles might fail. It is recommended to either force the
704
operation (via the ``--force`` option) or first change the number of
705
master candidates in the cluster - see :ref:`cluster-config-label`.
706

    
707
Evacuating nodes
708
++++++++++++++++
709

    
710
There are two steps of moving instances off a node:
711

    
712
- moving the primary instances (actually converting them into secondary
713
  instances)
714
- moving the secondary instances (including any instances converted in
715
  the step above)
716

    
717
Primary instance conversion
718
~~~~~~~~~~~~~~~~~~~~~~~~~~~
719

    
720
For this step, you can use either individual instance move
721
commands (as seen in :ref:`instance-change-primary-label`) or the bulk
722
per-node versions; these are::
723

    
724
  gnt-node migrate NODE
725
  gnt-node evacuate NODE
726

    
727
Note that the instance “move” command doesn't currently have a node
728
equivalent.
729

    
730
Both these commands, or the equivalent per-instance command, will make
731
this node the secondary node for the respective instances, whereas their
732
current secondary node will become primary. Note that it is not possible
733
to change in one step the primary node to another node as primary, while
734
keeping the same secondary node.
735

    
736
Secondary instance evacuation
737
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
738

    
739
For the evacuation of secondary instances, a command called
740
:command:`gnt-node evacuate` is provided and its syntax is::
741

    
742
  gnt-node evacuate -I IALLOCATOR_SCRIPT NODE
743
  gnt-node evacuate -n DESTINATION_NODE NODE
744

    
745
The first version will compute the new secondary for each instance in
746
turn using the given iallocator script, whereas the second one will
747
simply move all instances to DESTINATION_NODE.
748

    
749
Removal
750
+++++++
751

    
752
Once a node no longer has any instances (neither primary nor secondary),
753
it's easy to remove it from the cluster::
754

    
755
  gnt-node remove NODE_NAME
756

    
757
This will deconfigure the node, stop the ganeti daemons on it and leave
758
it hopefully like before it joined to the cluster.
759

    
760
Storage handling
761
++++++++++++++++
762

    
763
When using LVM (either standalone or with DRBD), it can become tedious
764
to debug and fix it in case of errors. Furthermore, even file-based
765
storage can become complicated to handle manually on many hosts. Ganeti
766
provides a couple of commands to help with automation.
767

    
768
Logical volumes
769
~~~~~~~~~~~~~~~
770

    
771
This is a command specific to LVM handling. It allows listing the
772
logical volumes on a given node or on all nodes and their association to
773
instances via the ``volumes`` command::
774

    
775
  node1# gnt-node volumes
776
  Node  PhysDev   VG    Name             Size Instance
777
  node1 /dev/sdb1 xenvg e61fbc97-….disk0 512M instance17
778
  node1 /dev/sdb1 xenvg ebd1a7d1-….disk0 512M instance19
779
  node2 /dev/sdb1 xenvg 0af08a3d-….disk0 512M instance20
780
  node2 /dev/sdb1 xenvg cc012285-….disk0 512M instance16
781
  node2 /dev/sdb1 xenvg f0fac192-….disk0 512M instance18
782

    
783
The above command maps each logical volume to a volume group and
784
underlying physical volume and (possibly) to an instance.
785

    
786
.. _storage-units-label:
787

    
788
Generalized storage handling
789
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
790

    
791
.. versionadded:: 2.1
792

    
793
Starting with Ganeti 2.1, a new storage framework has been implemented
794
that tries to abstract the handling of the storage type the cluster
795
uses.
796

    
797
First is listing the backend storage and their space situation::
798

    
799
  node1# gnt-node list-storage
800
  Node  Name        Size Used   Free
801
  node1 /dev/sda7 673.8G   0M 673.8G
802
  node1 /dev/sdb1 698.6G 1.5G 697.1G
803
  node2 /dev/sda7 673.8G   0M 673.8G
804
  node2 /dev/sdb1 698.6G 1.0G 697.6G
805

    
806
The default is to list LVM physical volumes. It's also possible to list
807
the LVM volume groups::
808

    
809
  node1# gnt-node list-storage -t lvm-vg
810
  Node  Name  Size
811
  node1 xenvg 1.3T
812
  node2 xenvg 1.3T
813

    
814
Next is repairing storage units, which is currently only implemented for
815
volume groups and does the equivalent of ``vgreduce --removemissing``::
816

    
817
  node1# gnt-node repair-storage node2 lvm-vg xenvg
818
  Sun Oct 25 22:21:45 2009 Repairing storage unit 'xenvg' on node2 ...
819

    
820
Last is the modification of volume properties, which is (again) only
821
implemented for LVM physical volumes and allows toggling the
822
``allocatable`` value::
823

    
824
  node1# gnt-node modify-storage --allocatable=no node2 lvm-pv /dev/sdb1
825

    
826
Use of the storage commands
827
~~~~~~~~~~~~~~~~~~~~~~~~~~~
828

    
829
All these commands are needed when recovering a node from a disk
830
failure:
831

    
832
- first, we need to recover from complete LVM failure (due to missing
833
  disk), by running the ``repair-storage`` command
834
- second, we need to change allocation on any partially-broken disk
835
  (i.e. LVM still sees it, but it has bad blocks) by running
836
  ``modify-storage``
837
- then we can evacuate the instances as needed
838

    
839

    
840
Cluster operations
841
------------------
842

    
843
Beside the cluster initialisation command (which is detailed in the
844
:doc:`install` document) and the master failover command which is
845
explained under node handling, there are a couple of other cluster
846
operations available.
847

    
848
.. _cluster-config-label:
849

    
850
Standard operations
851
+++++++++++++++++++
852

    
853
One of the few commands that can be run on any node (not only the
854
master) is the ``getmaster`` command::
855

    
856
  node2# gnt-cluster getmaster
857
  node1.example.com
858
  node2#
859

    
860
It is possible to query and change global cluster parameters via the
861
``info`` and ``modify`` commands::
862

    
863
  node1# gnt-cluster info
864
  Cluster name: cluster.example.com
865
  Cluster UUID: 07805e6f-f0af-4310-95f1-572862ee939c
866
  Creation time: 2009-09-25 05:04:15
867
  Modification time: 2009-10-18 22:11:47
868
  Master node: node1.example.com
869
  Architecture (this node): 64bit (x86_64)
870
871
  Tags: foo
872
  Default hypervisor: xen-pvm
873
  Enabled hypervisors: xen-pvm
874
  Hypervisor parameters:
875
    - xen-pvm:
876
        root_path: /dev/sda1
877
878
  Cluster parameters:
879
    - candidate pool size: 10
880
881
  Default instance parameters:
882
    - default:
883
        memory: 128
884
885
  Default nic parameters:
886
    - default:
887
        link: xen-br0
888
889

    
890
There various parameters above can be changed via the ``modify``
891
commands as follows:
892

    
893
- the hypervisor parameters can be changed via ``modify -H
894
  xen-pvm:root_path=…``, and so on for other hypervisors/key/values
895
- the "default instance parameters" are changeable via ``modify -B
896
  parameter=value…`` syntax
897
- the cluster parameters are changeable via separate options to the
898
  modify command (e.g. ``--candidate-pool-size``, etc.)
899

    
900
For detailed option list see the :manpage:`gnt-cluster(8)` man page.
901

    
902
The cluster version can be obtained via the ``version`` command::
903
  node1# gnt-cluster version
904
  Software version: 2.1.0
905
  Internode protocol: 20
906
  Configuration format: 2010000
907
  OS api version: 15
908
  Export interface: 0
909

    
910
This is not very useful except when debugging Ganeti.
911

    
912
Global node commands
913
++++++++++++++++++++
914

    
915
There are two commands provided for replicating files to all nodes of a
916
cluster and for running commands on all the nodes::
917

    
918
  node1# gnt-cluster copyfile /path/to/file
919
  node1# gnt-cluster command ls -l /path/to/file
920

    
921
These are simple wrappers over scp/ssh and more advanced usage can be
922
obtained using :manpage:`dsh(1)` and similar commands. But they are
923
useful to update an OS script from the master node, for example.
924

    
925
Cluster verification
926
++++++++++++++++++++
927

    
928
There are three commands that relate to global cluster checks. The first
929
one is ``verify`` which gives an overview on the cluster state,
930
highlighting any issues. In normal operation, this command should return
931
no ``ERROR`` messages::
932

    
933
  node1# gnt-cluster verify
934
  Sun Oct 25 23:08:58 2009 * Verifying global settings
935
  Sun Oct 25 23:08:58 2009 * Gathering data (2 nodes)
936
  Sun Oct 25 23:09:00 2009 * Verifying node status
937
  Sun Oct 25 23:09:00 2009 * Verifying instance status
938
  Sun Oct 25 23:09:00 2009 * Verifying orphan volumes
939
  Sun Oct 25 23:09:00 2009 * Verifying remaining instances
940
  Sun Oct 25 23:09:00 2009 * Verifying N+1 Memory redundancy
941
  Sun Oct 25 23:09:00 2009 * Other Notes
942
  Sun Oct 25 23:09:00 2009   - NOTICE: 5 non-redundant instance(s) found.
943
  Sun Oct 25 23:09:00 2009 * Hooks Results
944

    
945
The second command is ``verify-disks``, which checks that the instance's
946
disks have the correct status based on the desired instance state
947
(up/down)::
948

    
949
  node1# gnt-cluster verify-disks
950

    
951
Note that this command will show no output when disks are healthy.
952

    
953
The last command is used to repair any discrepancies in Ganeti's
954
recorded disk size and the actual disk size (disk size information is
955
needed for proper activation and growth of DRBD-based disks)::
956

    
957
  node1# gnt-cluster repair-disk-sizes
958
  Sun Oct 25 23:13:16 2009  - INFO: Disk 0 of instance instance1 has mismatched size, correcting: recorded 512, actual 2048
959
  Sun Oct 25 23:13:17 2009  - WARNING: Invalid result from node node4, ignoring node results
960

    
961
The above shows one instance having wrong disk size, and a node which
962
returned invalid data, and thus we ignored all primary instances of that
963
node.
964

    
965
Configuration redistribution
966
++++++++++++++++++++++++++++
967

    
968
If the verify command complains about file mismatches between the master
969
and other nodes, due to some node problems or if you manually modified
970
configuration files, you can force an push of the master configuration
971
to all other nodes via the ``redist-conf`` command::
972

    
973
  node1# gnt-cluster redist-conf
974
  node1#
975

    
976
This command will be silent unless there are problems sending updates to
977
the other nodes.
978

    
979

    
980
Cluster renaming
981
++++++++++++++++
982

    
983
It is possible to rename a cluster, or to change its IP address, via the
984
``rename`` command. If only the IP has changed, you need to pass the
985
current name and Ganeti will realise its IP has changed::
986

    
987
  node1# gnt-cluster rename cluster.example.com
988
  This will rename the cluster to 'cluster.example.com'. If
989
  you are connected over the network to the cluster name, the operation
990
  is very dangerous as the IP address will be removed from the node and
991
  the change may not go through. Continue?
992
  y/[n]/?: y
993
  Failure: prerequisites not met for this operation:
994
  Neither the name nor the IP address of the cluster has changed
995

    
996
In the above output, neither value has changed since the cluster
997
initialisation so the operation is not completed.
998

    
999
Queue operations
1000
++++++++++++++++
1001

    
1002
The job queue execution in Ganeti 2.0 and higher can be inspected,
1003
suspended and resumed via the ``queue`` command::
1004

    
1005
  node1~# gnt-cluster queue info
1006
  The drain flag is unset
1007
  node1~# gnt-cluster queue drain
1008
  node1~# gnt-instance stop instance1
1009
  Failed to submit job for instance1: Job queue is drained, refusing job
1010
  node1~# gnt-cluster queue info
1011
  The drain flag is set
1012
  node1~# gnt-cluster queue undrain
1013

    
1014
This is most useful if you have an active cluster and you need to
1015
upgrade the Ganeti software, or simply restart the software on any node:
1016

    
1017
#. suspend the queue via ``queue drain``
1018
#. wait until there are no more running jobs via ``gnt-job list``
1019
#. restart the master or another node, or upgrade the software
1020
#. resume the queue via ``queue undrain``
1021

    
1022
.. note:: this command only stores a local flag file, and if you
1023
   failover the master, it will not have effect on the new master.
1024

    
1025

    
1026
Watcher control
1027
+++++++++++++++
1028

    
1029
The :manpage:`ganeti-watcher` is a program, usually scheduled via
1030
``cron``, that takes care of cluster maintenance operations (restarting
1031
downed instances, activating down DRBD disks, etc.). However, during
1032
maintenance and troubleshooting, this can get in your way; disabling it
1033
via commenting out the cron job is not so good as this can be
1034
forgotten. Thus there are some commands for automated control of the
1035
watcher: ``pause``, ``info`` and ``continue``::
1036

    
1037
  node1~# gnt-cluster watcher info
1038
  The watcher is not paused.
1039
  node1~# gnt-cluster watcher pause 1h
1040
  The watcher is paused until Mon Oct 26 00:30:37 2009.
1041
  node1~# gnt-cluster watcher info
1042
  The watcher is paused until Mon Oct 26 00:30:37 2009.
1043
  node1~# ganeti-watcher -d
1044
  2009-10-25 23:30:47,984:  pid=28867 ganeti-watcher:486 DEBUG Pause has been set, exiting
1045
  node1~# gnt-cluster watcher continue
1046
  The watcher is no longer paused.
1047
  node1~# ganeti-watcher -d
1048
  2009-10-25 23:31:04,789:  pid=28976 ganeti-watcher:345 DEBUG Archived 0 jobs, left 0
1049
  2009-10-25 23:31:05,884:  pid=28976 ganeti-watcher:280 DEBUG Got data from cluster, writing instance status file
1050
  2009-10-25 23:31:06,061:  pid=28976 ganeti-watcher:150 DEBUG Data didn't change, just touching status file
1051
  node1~# gnt-cluster watcher info
1052
  The watcher is not paused.
1053
  node1~#
1054

    
1055
The exact details of the argument to the ``pause`` command are available
1056
in the manpage.
1057

    
1058
.. note:: this command only stores a local flag file, and if you
1059
   failover the master, it will not have effect on the new master.
1060

    
1061
Removing a cluster entirely
1062
+++++++++++++++++++++++++++
1063

    
1064
The usual method to cleanup a cluster is to run ``gnt-cluster destroy``
1065
however if the Ganeti installation is broken in any way then this will
1066
not run.
1067

    
1068
It is possible in such a case to cleanup manually most if not all traces
1069
of a cluster installation by following these steps on all of the nodes:
1070

    
1071
1. Shutdown all instances. This depends on the virtualisation method
1072
   used (Xen, KVM, etc.):
1073

    
1074
  - Xen: run ``xm list`` and ``xm destroy`` on all the non-Domain-0
1075
    instances
1076
  - KVM: kill all the KVM processes
1077
  - chroot: kill all processes under the chroot mountpoints
1078

    
1079
2. If using DRBD, shutdown all DRBD minors (which should by at this time
1080
   no-longer in use by instances); on each node, run ``drbdsetup
1081
   /dev/drbdN down`` for each active DRBD minor.
1082

    
1083
3. If using LVM, cleanup the Ganeti volume group; if only Ganeti created
1084
   logical volumes (and you are not sharing the volume group with the
1085
   OS, for example), then simply running ``lvremove -f xenvg`` (replace
1086
   'xenvg' with your volume group name) should do the required cleanup.
1087

    
1088
4. If using file-based storage, remove recursively all files and
1089
   directories under your file-storage directory: ``rm -rf
1090
   /srv/ganeti/file-storage/*`` replacing the path with the correct path
1091
   for your cluster.
1092

    
1093
5. Stop the ganeti daemons (``/etc/init.d/ganeti stop``) and kill any
1094
   that remain alive (``pgrep ganeti`` and ``pkill ganeti``).
1095

    
1096
6. Remove the ganeti state directory (``rm -rf /var/lib/ganeti/*``),
1097
   replacing the path with the correct path for your installation.
1098

    
1099
On the master node, remove the cluster from the master-netdev (usually
1100
``xen-br0`` for bridged mode, otherwise ``eth0`` or similar), by running
1101
``ip a del $clusterip/32 dev xen-br0`` (use the correct cluster ip and
1102
network device name).
1103

    
1104
At this point, the machines are ready for a cluster creation; in case
1105
you want to remove Ganeti completely, you need to also undo some of the
1106
SSH changes and log directories:
1107

    
1108
- ``rm -rf /var/log/ganeti /srv/ganeti`` (replace with the correct
1109
  paths)
1110
- remove from ``/root/.ssh`` the keys that Ganeti added (check the
1111
  ``authorized_keys`` and ``id_dsa`` files)
1112
- regenerate the host's SSH keys (check the OpenSSH startup scripts)
1113
- uninstall Ganeti
1114

    
1115
Otherwise, if you plan to re-create the cluster, you can just go ahead
1116
and rerun ``gnt-cluster init``.
1117

    
1118
Tags handling
1119
-------------
1120

    
1121
The tags handling (addition, removal, listing) is similar for all the
1122
objects that support it (instances, nodes, and the cluster).
1123

    
1124
Limitations
1125
+++++++++++
1126

    
1127
Note that the set of characters present in a tag and the maximum tag
1128
length are restricted. Currently the maximum length is 128 characters,
1129
there can be at most 4096 tags per object, and the set of characters is
1130
comprised by alphanumeric characters and additionally ``.+*/:-``.
1131

    
1132
Operations
1133
++++++++++
1134

    
1135
Tags can be added via ``add-tags``::
1136

    
1137
  gnt-instance add-tags INSTANCE a b c
1138
  gnt-node add-tags INSTANCE a b c
1139
  gnt-cluster add-tags a b c
1140

    
1141

    
1142
The above commands add three tags to an instance, to a node and to the
1143
cluster. Note that the cluster command only takes tags as arguments,
1144
whereas the node and instance commands first required the node and
1145
instance name.
1146

    
1147
Tags can also be added from a file, via the ``--from=FILENAME``
1148
argument. The file is expected to contain one tag per line.
1149

    
1150
Tags can also be remove via a syntax very similar to the add one::
1151

    
1152
  gnt-instance remove-tags INSTANCE a b c
1153

    
1154
And listed via::
1155

    
1156
  gnt-instance list-tags
1157
  gnt-node list-tags
1158
  gnt-cluster list-tags
1159

    
1160
Global tag search
1161
+++++++++++++++++
1162

    
1163
It is also possible to execute a global search on the all tags defined
1164
in the cluster configuration, via a cluster command::
1165

    
1166
  gnt-cluster search-tags REGEXP
1167

    
1168
The parameter expected is a regular expression (see
1169
:manpage:`regex(7)`). This will return all tags that match the search,
1170
together with the object they are defined in (the names being show in a
1171
hierarchical kind of way)::
1172

    
1173
  node1# gnt-cluster search-tags o
1174
  /cluster foo
1175
  /instances/instance1 owner:bar
1176

    
1177

    
1178
Job operations
1179
--------------
1180

    
1181
The various jobs submitted by the instance/node/cluster commands can be
1182
examined, canceled and archived by various invocations of the
1183
``gnt-job`` command.
1184

    
1185
First is the job list command::
1186

    
1187
  node1# gnt-job list
1188
  17771 success INSTANCE_QUERY_DATA
1189
  17773 success CLUSTER_VERIFY_DISKS
1190
  17775 success CLUSTER_REPAIR_DISK_SIZES
1191
  17776 error   CLUSTER_RENAME(cluster.example.com)
1192
  17780 success CLUSTER_REDIST_CONF
1193
  17792 success INSTANCE_REBOOT(instance1.example.com)
1194

    
1195
More detailed information about a job can be found via the ``info``
1196
command::
1197

    
1198
  node1# gnt-job info 17776
1199
  Job ID: 17776
1200
    Status: error
1201
    Received:         2009-10-25 23:18:02.180569
1202
    Processing start: 2009-10-25 23:18:02.200335 (delta 0.019766s)
1203
    Processing end:   2009-10-25 23:18:02.279743 (delta 0.079408s)
1204
    Total processing time: 0.099174 seconds
1205
    Opcodes:
1206
      OP_CLUSTER_RENAME
1207
        Status: error
1208
        Processing start: 2009-10-25 23:18:02.200335
1209
        Processing end:   2009-10-25 23:18:02.252282
1210
        Input fields:
1211
          name: cluster.example.com
1212
        Result:
1213
          OpPrereqError
1214
          [Neither the name nor the IP address of the cluster has changed]
1215
        Execution log:
1216

    
1217
During the execution of a job, it's possible to follow the output of a
1218
job, similar to the log that one get from the ``gnt-`` commands, via the
1219
watch command::
1220

    
1221
  node1# gnt-instance add --submit … instance1
1222
  JobID: 17818
1223
  node1# gnt-job watch 17818
1224
  Output from job 17818 follows
1225
  -----------------------------
1226
  Mon Oct 26 00:22:48 2009  - INFO: Selected nodes for instance instance1 via iallocator dumb: node1, node2
1227
  Mon Oct 26 00:22:49 2009 * creating instance disks...
1228
  Mon Oct 26 00:22:52 2009 adding instance instance1 to cluster config
1229
  Mon Oct 26 00:22:52 2009  - INFO: Waiting for instance instance1 to sync disks.
1230
1231
  Mon Oct 26 00:23:03 2009 creating os for instance xen-devi-18.fra.corp.google.com on node mpgntac4.fra.corp.google.com
1232
  Mon Oct 26 00:23:03 2009 * running the instance OS create scripts...
1233
  Mon Oct 26 00:23:13 2009 * starting instance...
1234
  node1#
1235

    
1236
This is useful if you need to follow a job's progress from multiple
1237
terminals.
1238

    
1239
A job that has not yet started to run can be canceled::
1240

    
1241
  node1# gnt-job cancel 17810
1242

    
1243
But not one that has already started execution::
1244

    
1245
  node1# gnt-job cancel 17805
1246
  Job 17805 is no longer waiting in the queue
1247

    
1248
There are two queues for jobs: the *current* and the *archive*
1249
queue. Jobs are initially submitted to the current queue, and they stay
1250
in that queue until they have finished execution (either successfully or
1251
not). At that point, they can be moved into the archive queue, and the
1252
ganeti-watcher script will do this automatically after 6 hours. The
1253
ganeti-cleaner script will remove the jobs from the archive directory
1254
after three weeks.
1255

    
1256
Note that only jobs in the current queue can be viewed via the list and
1257
info commands; Ganeti itself doesn't examine the archive directory. If
1258
you need to see an older job, either move the file manually in the
1259
top-level queue directory, or look at its contents (it's a
1260
JSON-formatted file).
1261

    
1262
Ganeti tools
1263
------------
1264

    
1265
Beside the usual ``gnt-`` and ``ganeti-`` commands which are provided
1266
and installed in ``$prefix/sbin`` at install time, there are a couple of
1267
other tools installed which are used seldom but can be helpful in some
1268
cases.
1269

    
1270
lvmstrap
1271
++++++++
1272

    
1273
The ``lvmstrap`` tool, introduced in :ref:`configure-lvm-label` section,
1274
has two modes of operation:
1275

    
1276
- ``diskinfo`` shows the discovered disks on the system and their status
1277
- ``create`` takes all not-in-use disks and creates a volume group out
1278
  of them
1279

    
1280
.. warning:: The ``create`` argument to this command causes data-loss!
1281

    
1282
cfgupgrade
1283
++++++++++
1284

    
1285
The ``cfgupgrade`` tools is used to upgrade between major (and minor)
1286
Ganeti versions. Point-releases are usually transparent for the admin.
1287

    
1288
More information about the upgrade procedure is listed on the wiki at
1289
http://code.google.com/p/ganeti/wiki/UpgradeNotes.
1290

    
1291
cfgshell
1292
++++++++
1293

    
1294
.. note:: This command is not actively maintained; make sure you backup
1295
   your configuration before using it
1296

    
1297
This can be used as an alternative to direct editing of the
1298
main configuration file if Ganeti has a bug and prevents you, for
1299
example, from removing an instance or a node from the configuration
1300
file.
1301

    
1302
.. _burnin-label:
1303

    
1304
burnin
1305
++++++
1306

    
1307
.. warning:: This command will erase existing instances if given as
1308
   arguments!
1309

    
1310
This tool is used to exercise either the hardware of machines or
1311
alternatively the Ganeti software. It is safe to run on an existing
1312
cluster **as long as you don't pass it existing instance names**.
1313

    
1314
The command will, by default, execute a comprehensive set of operations
1315
against a list of instances, these being:
1316

    
1317
- creation
1318
- disk replacement (for redundant instances)
1319
- failover and migration (for redundant instances)
1320
- move (for non-redundant instances)
1321
- disk growth
1322
- add disks, remove disk
1323
- add NICs, remove NICs
1324
- export and then import
1325
- rename
1326
- reboot
1327
- shutdown/startup
1328
- and finally removal of the test instances
1329

    
1330
Executing all these operations will test that the hardware performs
1331
well: the creation, disk replace, disk add and disk growth will exercise
1332
the storage and network; the migrate command will test the memory of the
1333
systems. Depending on the passed options, it can also test that the
1334
instance OS definitions are executing properly the rename, import and
1335
export operations.
1336

    
1337
Other Ganeti projects
1338
---------------------
1339

    
1340
There are two other Ganeti-related projects that can be useful in a
1341
Ganeti deployment. These can be downloaded from the project site
1342
(http://code.google.com/p/ganeti/) and the repositories are also on the
1343
project git site (http://git.ganeti.org).
1344

    
1345
NBMA tools
1346
++++++++++
1347

    
1348
The ``ganeti-nbma`` software is designed to allow instances to live on a
1349
separate, virtual network from the nodes, and in an environment where
1350
nodes are not guaranteed to be able to reach each other via multicasting
1351
or broadcasting. For more information see the README in the source
1352
archive.
1353

    
1354
ganeti-htools
1355
+++++++++++++
1356

    
1357
The ``ganeti-htools`` software consists of a set of tools:
1358

    
1359
- ``hail``: an advanced iallocator script compared to Ganeti's builtin
1360
  one
1361
- ``hbal``: a tool for rebalancing the cluster, i.e. moving instances
1362
  around in order to better use the resources on the nodes
1363
- ``hspace``: a tool for estimating the available capacity of a cluster,
1364
  so that capacity planning can be done efficiently
1365

    
1366
For more information and installation instructions, see the README file
1367
in the source archive.
1368

    
1369
.. vim: set textwidth=72 :
1370
.. Local Variables:
1371
.. mode: rst
1372
.. fill-column: 72
1373
.. End: