Statistics
| Branch: | Tag: | Revision:

root / doc / admin.rst @ bced76fd

History | View | Annotate | Download (66.8 kB)

1
Ganeti administrator's guide
2
============================
3

    
4
Documents Ganeti version |version|
5

    
6
.. contents::
7

    
8
.. highlight:: shell-example
9

    
10
Introduction
11
------------
12

    
13
Ganeti is a virtualization cluster management software. You are expected
14
to be a system administrator familiar with your Linux distribution and
15
the Xen or KVM virtualization environments before using it.
16

    
17
The various components of Ganeti all have man pages and interactive
18
help. This manual though will help you getting familiar with the system
19
by explaining the most common operations, grouped by related use.
20

    
21
After a terminology glossary and a section on the prerequisites needed
22
to use this manual, the rest of this document is divided in sections
23
for the different targets that a command affects: instance, nodes, etc.
24

    
25
.. _terminology-label:
26

    
27
Ganeti terminology
28
++++++++++++++++++
29

    
30
This section provides a small introduction to Ganeti terminology, which
31
might be useful when reading the rest of the document.
32

    
33
Cluster
34
~~~~~~~
35

    
36
A set of machines (nodes) that cooperate to offer a coherent, highly
37
available virtualization service under a single administration domain.
38

    
39
Node
40
~~~~
41

    
42
A physical machine which is member of a cluster.  Nodes are the basic
43
cluster infrastructure, and they don't need to be fault tolerant in
44
order to achieve high availability for instances.
45

    
46
Node can be added and removed (if they host no instances) at will from
47
the cluster. In a HA cluster and only with HA instances, the loss of any
48
single node will not cause disk data loss for any instance; of course,
49
a node crash will cause the crash of its primary instances.
50

    
51
A node belonging to a cluster can be in one of the following roles at a
52
given time:
53

    
54
- *master* node, which is the node from which the cluster is controlled
55
- *master candidate* node, only nodes in this role have the full cluster
56
  configuration and knowledge, and only master candidates can become the
57
  master node
58
- *regular* node, which is the state in which most nodes will be on
59
  bigger clusters (>20 nodes)
60
- *drained* node, nodes in this state are functioning normally but the
61
  cannot receive new instances; the intention is that nodes in this role
62
  have some issue and they are being evacuated for hardware repairs
63
- *offline* node, in which there is a record in the cluster
64
  configuration about the node, but the daemons on the master node will
65
  not talk to this node; any instances declared as having an offline
66
  node as either primary or secondary will be flagged as an error in the
67
  cluster verify operation
68

    
69
Depending on the role, each node will run a set of daemons:
70

    
71
- the :command:`ganeti-noded` daemon, which controls the manipulation of
72
  this node's hardware resources; it runs on all nodes which are in a
73
  cluster
74
- the :command:`ganeti-confd` daemon (Ganeti 2.1+) which runs on all
75
  nodes, but is only functional on master candidate nodes; this daemon
76
  can be disabled at configuration time if you don't need its
77
  functionality
78
- the :command:`ganeti-rapi` daemon which runs on the master node and
79
  offers an HTTP-based API for the cluster
80
- the :command:`ganeti-masterd` daemon which runs on the master node and
81
  allows control of the cluster
82

    
83
Beside the node role, there are other node flags that influence its
84
behaviour:
85

    
86
- the *master_capable* flag denotes whether the node can ever become a
87
  master candidate; setting this to 'no' means that auto-promotion will
88
  never make this node a master candidate; this flag can be useful for a
89
  remote node that only runs local instances, and having it become a
90
  master is impractical due to networking or other constraints
91
- the *vm_capable* flag denotes whether the node can host instances or
92
  not; for example, one might use a non-vm_capable node just as a master
93
  candidate, for configuration backups; setting this flag to no
94
  disallows placement of instances of this node, deactivates hypervisor
95
  and related checks on it (e.g. bridge checks, LVM check, etc.), and
96
  removes it from cluster capacity computations
97

    
98

    
99
Instance
100
~~~~~~~~
101

    
102
A virtual machine which runs on a cluster. It can be a fault tolerant,
103
highly available entity.
104

    
105
An instance has various parameters, which are classified in three
106
categories: hypervisor related-parameters (called ``hvparams``), general
107
parameters (called ``beparams``) and per network-card parameters (called
108
``nicparams``). All these parameters can be modified either at instance
109
level or via defaults at cluster level.
110

    
111
Disk template
112
~~~~~~~~~~~~~
113

    
114
The are multiple options for the storage provided to an instance; while
115
the instance sees the same virtual drive in all cases, the node-level
116
configuration varies between them.
117

    
118
There are five disk templates you can choose from:
119

    
120
diskless
121
  The instance has no disks. Only used for special purpose operating
122
  systems or for testing.
123

    
124
file
125
  The instance will use plain files as backend for its disks. No
126
  redundancy is provided, and this is somewhat more difficult to
127
  configure for high performance. Note that for security reasons the
128
  file storage directory must be listed under
129
  ``/etc/ganeti/file-storage-paths``, and that file is not copied
130
  automatically to all nodes by Ganeti. The format of that file is a
131
  newline-separated list of directories.
132

    
133
sharedfile
134
  The instance will use plain files as backend, but Ganeti assumes that
135
  those files will be available and in sync automatically on all nodes.
136
  This allows live migration and failover of instances using this
137
  method. As for ``file`` the file storage directory must be listed under
138
  ``/etc/ganeti/file-storage-paths`` or ganeti will refuse to create
139
  instances under it.
140

    
141
plain
142
  The instance will use LVM devices as backend for its disks. No
143
  redundancy is provided.
144

    
145
drbd
146
  .. note:: This is only valid for multi-node clusters using DRBD 8.0+
147

    
148
  A mirror is set between the local node and a remote one, which must be
149
  specified with the second value of the --node option. Use this option
150
  to obtain a highly available instance that can be failed over to a
151
  remote node should the primary one fail.
152

    
153
  .. note:: Ganeti does not support DRBD stacked devices:
154
     DRBD stacked setup is not fully symmetric and as such it is
155
     not working with live migration.
156

    
157
rbd
158
  The instance will use Volumes inside a RADOS cluster as backend for its
159
  disks. It will access them using the RADOS block device (RBD).
160

    
161
ext
162
  The instance will use an external storage provider. See
163
  :manpage:`ganeti-extstorage-interface(7)` for how to implement one.
164

    
165

    
166
IAllocator
167
~~~~~~~~~~
168

    
169
A framework for using external (user-provided) scripts to compute the
170
placement of instances on the cluster nodes. This eliminates the need to
171
manually specify nodes in instance add, instance moves, node evacuate,
172
etc.
173

    
174
In order for Ganeti to be able to use these scripts, they must be place
175
in the iallocator directory (usually ``lib/ganeti/iallocators`` under
176
the installation prefix, e.g. ``/usr/local``).
177

    
178
“Primary” and “secondary” concepts
179
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
180

    
181
An instance has a primary and depending on the disk configuration, might
182
also have a secondary node. The instance always runs on the primary node
183
and only uses its secondary node for disk replication.
184

    
185
Similarly, the term of primary and secondary instances when talking
186
about a node refers to the set of instances having the given node as
187
primary, respectively secondary.
188

    
189
Tags
190
~~~~
191

    
192
Tags are short strings that can be attached to either to cluster itself,
193
or to nodes or instances. They are useful as a very simplistic
194
information store for helping with cluster administration, for example
195
by attaching owner information to each instance after it's created::
196

    
197
  $ gnt-instance add … %instance1%
198
  $ gnt-instance add-tags %instance1% %owner:user2%
199

    
200
And then by listing each instance and its tags, this information could
201
be used for contacting the users of each instance.
202

    
203
Jobs and OpCodes
204
~~~~~~~~~~~~~~~~
205

    
206
While not directly visible by an end-user, it's useful to know that a
207
basic cluster operation (e.g. starting an instance) is represented
208
internally by Ganeti as an *OpCode* (abbreviation from operation
209
code). These OpCodes are executed as part of a *Job*. The OpCodes in a
210
single Job are processed serially by Ganeti, but different Jobs will be
211
processed (depending on resource availability) in parallel. They will
212
not be executed in the submission order, but depending on resource
213
availability, locks and (starting with Ganeti 2.3) priority. An earlier
214
job may have to wait for a lock while a newer job doesn't need any locks
215
and can be executed right away. Operations requiring a certain order
216
need to be submitted as a single job, or the client must submit one job
217
at a time and wait for it to finish before continuing.
218

    
219
For example, shutting down the entire cluster can be done by running the
220
command ``gnt-instance shutdown --all``, which will submit for each
221
instance a separate job containing the “shutdown instance” OpCode.
222

    
223

    
224
Prerequisites
225
+++++++++++++
226

    
227
You need to have your Ganeti cluster installed and configured before you
228
try any of the commands in this document. Please follow the
229
:doc:`install` for instructions on how to do that.
230

    
231
Instance management
232
-------------------
233

    
234
Adding an instance
235
++++++++++++++++++
236

    
237
The add operation might seem complex due to the many parameters it
238
accepts, but once you have understood the (few) required parameters and
239
the customisation capabilities you will see it is an easy operation.
240

    
241
The add operation requires at minimum five parameters:
242

    
243
- the OS for the instance
244
- the disk template
245
- the disk count and size
246
- the node specification or alternatively the iallocator to use
247
- and finally the instance name
248

    
249
The OS for the instance must be visible in the output of the command
250
``gnt-os list`` and specifies which guest OS to install on the instance.
251

    
252
The disk template specifies what kind of storage to use as backend for
253
the (virtual) disks presented to the instance; note that for instances
254
with multiple virtual disks, they all must be of the same type.
255

    
256
The node(s) on which the instance will run can be given either manually,
257
via the ``-n`` option, or computed automatically by Ganeti, if you have
258
installed any iallocator script.
259

    
260
With the above parameters in mind, the command is::
261

    
262
  $ gnt-instance add \
263
    -n %TARGET_NODE%:%SECONDARY_NODE% \
264
    -o %OS_TYPE% \
265
    -t %DISK_TEMPLATE% -s %DISK_SIZE% \
266
    %INSTANCE_NAME%
267

    
268
The instance name must be resolvable (e.g. exist in DNS) and usually
269
points to an address in the same subnet as the cluster itself.
270

    
271
The above command has the minimum required options; other options you
272
can give include, among others:
273

    
274
- The maximum/minimum memory size (``-B maxmem``, ``-B minmem``)
275
  (``-B memory`` can be used to specify only one size)
276

    
277
- The number of virtual CPUs (``-B vcpus``)
278

    
279
- Arguments for the NICs of the instance; by default, a single-NIC
280
  instance is created. The IP and/or bridge of the NIC can be changed
281
  via ``--net 0:ip=IP,link=BRIDGE``
282

    
283
See :manpage:`ganeti-instance(8)` for the detailed option list.
284

    
285
For example if you want to create an highly available instance, with a
286
single disk of 50GB and the default memory size, having primary node
287
``node1`` and secondary node ``node3``, use the following command::
288

    
289
  $ gnt-instance add -n node1:node3 -o debootstrap -t drbd -s 50G \
290
    instance1
291

    
292
There is a also a command for batch instance creation from a
293
specification file, see the ``batch-create`` operation in the
294
gnt-instance manual page.
295

    
296
Regular instance operations
297
+++++++++++++++++++++++++++
298

    
299
Removal
300
~~~~~~~
301

    
302
Removing an instance is even easier than creating one. This operation is
303
irreversible and destroys all the contents of your instance. Use with
304
care::
305

    
306
  $ gnt-instance remove %INSTANCE_NAME%
307

    
308
.. _instance-startup-label:
309

    
310
Startup/shutdown
311
~~~~~~~~~~~~~~~~
312

    
313
Instances are automatically started at instance creation time. To
314
manually start one which is currently stopped you can run::
315

    
316
  $ gnt-instance startup %INSTANCE_NAME%
317

    
318
Ganeti will start an instance with up to its maximum instance memory. If
319
not enough memory is available Ganeti will use all the available memory
320
down to the instance minimum memory. If not even that amount of memory
321
is free Ganeti will refuse to start the instance.
322

    
323
Note, that this will not work when an instance is in a permanently
324
stopped state ``offline``. In this case, you will first have to
325
put it back to online mode by running::
326

    
327
  $ gnt-instance modify --online %INSTANCE_NAME%
328

    
329
The command to stop the running instance is::
330

    
331
  $ gnt-instance shutdown %INSTANCE_NAME%
332

    
333
If you want to shut the instance down more permanently, so that it
334
does not require dynamically allocated resources (memory and vcpus),
335
after shutting down an instance, execute the following::
336

    
337
  $ gnt-instance modify --offline %INSTANCE_NAME%
338

    
339
.. warning:: Do not use the Xen or KVM commands directly to stop
340
   instances. If you run for example ``xm shutdown`` or ``xm destroy``
341
   on an instance Ganeti will automatically restart it (via
342
   the :command:`ganeti-watcher(8)` command which is launched via cron).
343

    
344
Querying instances
345
~~~~~~~~~~~~~~~~~~
346

    
347
There are two ways to get information about instances: listing
348
instances, which does a tabular output containing a given set of fields
349
about each instance, and querying detailed information about a set of
350
instances.
351

    
352
The command to see all the instances configured and their status is::
353

    
354
  $ gnt-instance list
355

    
356
The command can return a custom set of information when using the ``-o``
357
option (as always, check the manpage for a detailed specification). Each
358
instance will be represented on a line, thus making it easy to parse
359
this output via the usual shell utilities (grep, sed, etc.).
360

    
361
To get more detailed information about an instance, you can run::
362

    
363
  $ gnt-instance info %INSTANCE%
364

    
365
which will give a multi-line block of information about the instance,
366
it's hardware resources (especially its disks and their redundancy
367
status), etc. This is harder to parse and is more expensive than the
368
list operation, but returns much more detailed information.
369

    
370
Changing an instance's runtime memory
371
+++++++++++++++++++++++++++++++++++++
372

    
373
Ganeti will always make sure an instance has a value between its maximum
374
and its minimum memory available as runtime memory. As of version 2.6
375
Ganeti will only choose a size different than the maximum size when
376
starting up, failing over, or migrating an instance on a node with less
377
than the maximum memory available. It won't resize other instances in
378
order to free up space for an instance.
379

    
380
If you find that you need more memory on a node any instance can be
381
manually resized without downtime, with the command::
382

    
383
  $ gnt-instance modify -m %SIZE% %INSTANCE_NAME%
384

    
385
The same command can also be used to increase the memory available on an
386
instance, provided that enough free memory is available on its node, and
387
the specified size is not larger than the maximum memory size the
388
instance had when it was first booted (an instance will be unable to see
389
new memory above the maximum that was specified to the hypervisor at its
390
boot time, if it needs to grow further a reboot becomes necessary).
391

    
392
Export/Import
393
+++++++++++++
394

    
395
You can create a snapshot of an instance disk and its Ganeti
396
configuration, which then you can backup, or import into another
397
cluster. The way to export an instance is::
398

    
399
  $ gnt-backup export -n %TARGET_NODE% %INSTANCE_NAME%
400

    
401

    
402
The target node can be any node in the cluster with enough space under
403
``/srv/ganeti`` to hold the instance image. Use the ``--noshutdown``
404
option to snapshot an instance without rebooting it. Note that Ganeti
405
only keeps one snapshot for an instance - any previous snapshot of the
406
same instance existing cluster-wide under ``/srv/ganeti`` will be
407
removed by this operation: if you want to keep them, you need to move
408
them out of the Ganeti exports directory.
409

    
410
Importing an instance is similar to creating a new one, but additionally
411
one must specify the location of the snapshot. The command is::
412

    
413
  $ gnt-backup import -n %TARGET_NODE% \
414
    --src-node=%NODE% --src-dir=%DIR% %INSTANCE_NAME%
415

    
416
By default, parameters will be read from the export information, but you
417
can of course pass them in via the command line - most of the options
418
available for the command :command:`gnt-instance add` are supported here
419
too.
420

    
421
Import of foreign instances
422
+++++++++++++++++++++++++++
423

    
424
There is a possibility to import a foreign instance whose disk data is
425
already stored as LVM volumes without going through copying it: the disk
426
adoption mode.
427

    
428
For this, ensure that the original, non-managed instance is stopped,
429
then create a Ganeti instance in the usual way, except that instead of
430
passing the disk information you specify the current volumes::
431

    
432
  $ gnt-instance add -t plain -n %HOME_NODE% ... \
433
    --disk 0:adopt=%lv_name%[,vg=%vg_name%] %INSTANCE_NAME%
434

    
435
This will take over the given logical volumes, rename them to the Ganeti
436
standard (UUID-based), and without installing the OS on them start
437
directly the instance. If you configure the hypervisor similar to the
438
non-managed configuration that the instance had, the transition should
439
be seamless for the instance. For more than one disk, just pass another
440
disk parameter (e.g. ``--disk 1:adopt=...``).
441

    
442
Instance kernel selection
443
+++++++++++++++++++++++++
444

    
445
The kernel that instances uses to bootup can come either from the node,
446
or from instances themselves, depending on the setup.
447

    
448
Xen-PVM
449
~~~~~~~
450

    
451
With Xen PVM, there are three options.
452

    
453
First, you can use a kernel from the node, by setting the hypervisor
454
parameters as such:
455

    
456
- ``kernel_path`` to a valid file on the node (and appropriately
457
  ``initrd_path``)
458
- ``kernel_args`` optionally set to a valid Linux setting (e.g. ``ro``)
459
- ``root_path`` to a valid setting (e.g. ``/dev/xvda1``)
460
- ``bootloader_path`` and ``bootloader_args`` to empty
461

    
462
Alternatively, you can delegate the kernel management to instances, and
463
use either ``pvgrub`` or the deprecated ``pygrub``. For this, you must
464
install the kernels and initrds in the instance and create a valid GRUB
465
v1 configuration file.
466

    
467
For ``pvgrub`` (new in version 2.4.2), you need to set:
468

    
469
- ``kernel_path`` to point to the ``pvgrub`` loader present on the node
470
  (e.g. ``/usr/lib/xen/boot/pv-grub-x86_32.gz``)
471
- ``kernel_args`` to the path to the GRUB config file, relative to the
472
  instance (e.g. ``(hd0,0)/grub/menu.lst``)
473
- ``root_path`` **must** be empty
474
- ``bootloader_path`` and ``bootloader_args`` to empty
475

    
476
While ``pygrub`` is deprecated, here is how you can configure it:
477

    
478
- ``bootloader_path`` to the pygrub binary (e.g. ``/usr/bin/pygrub``)
479
- the other settings are not important
480

    
481
More information can be found in the Xen wiki pages for `pvgrub
482
<http://wiki.xensource.com/xenwiki/PvGrub>`_ and `pygrub
483
<http://wiki.xensource.com/xenwiki/PyGrub>`_.
484

    
485
KVM
486
~~~
487

    
488
For KVM also the kernel can be loaded either way.
489

    
490
For loading the kernels from the node, you need to set:
491

    
492
- ``kernel_path`` to a valid value
493
- ``initrd_path`` optionally set if you use an initrd
494
- ``kernel_args`` optionally set to a valid value (e.g. ``ro``)
495

    
496
If you want instead to have the instance boot from its disk (and execute
497
its bootloader), simply set the ``kernel_path`` parameter to an empty
498
string, and all the others will be ignored.
499

    
500
Instance HA features
501
--------------------
502

    
503
.. note:: This section only applies to multi-node clusters
504

    
505
.. _instance-change-primary-label:
506

    
507
Changing the primary node
508
+++++++++++++++++++++++++
509

    
510
There are three ways to exchange an instance's primary and secondary
511
nodes; the right one to choose depends on how the instance has been
512
created and the status of its current primary node. See
513
:ref:`rest-redundancy-label` for information on changing the secondary
514
node. Note that it's only possible to change the primary node to the
515
secondary and vice-versa; a direct change of the primary node with a
516
third node, while keeping the current secondary is not possible in a
517
single step, only via multiple operations as detailed in
518
:ref:`instance-relocation-label`.
519

    
520
Failing over an instance
521
~~~~~~~~~~~~~~~~~~~~~~~~
522

    
523
If an instance is built in highly available mode you can at any time
524
fail it over to its secondary node, even if the primary has somehow
525
failed and it's not up anymore. Doing it is really easy, on the master
526
node you can just run::
527

    
528
  $ gnt-instance failover %INSTANCE_NAME%
529

    
530
That's it. After the command completes the secondary node is now the
531
primary, and vice-versa.
532

    
533
The instance will be started with an amount of memory between its
534
``maxmem`` and its ``minmem`` value, depending on the free memory on its
535
target node, or the operation will fail if that's not possible. See
536
:ref:`instance-startup-label` for details.
537

    
538
If the instance's disk template is of type rbd, then you can specify
539
the target node (which can be any node) explicitly, or specify an
540
iallocator plugin. If you omit both, the default iallocator will be
541
used to determine the target node::
542

    
543
  $ gnt-instance failover -n %TARGET_NODE% %INSTANCE_NAME%
544

    
545
Live migrating an instance
546
~~~~~~~~~~~~~~~~~~~~~~~~~~
547

    
548
If an instance is built in highly available mode, it currently runs and
549
both its nodes are running fine, you can migrate it over to its
550
secondary node, without downtime. On the master node you need to run::
551

    
552
  $ gnt-instance migrate %INSTANCE_NAME%
553

    
554
The current load on the instance and its memory size will influence how
555
long the migration will take. In any case, for both KVM and Xen
556
hypervisors, the migration will be transparent to the instance.
557

    
558
If the destination node has less memory than the instance's current
559
runtime memory, but at least the instance's minimum memory available
560
Ganeti will automatically reduce the instance runtime memory before
561
migrating it, unless the ``--no-runtime-changes`` option is passed, in
562
which case the target node should have at least the instance's current
563
runtime memory free.
564

    
565
If the instance's disk template is of type rbd, then you can specify
566
the target node (which can be any node) explicitly, or specify an
567
iallocator plugin. If you omit both, the default iallocator will be
568
used to determine the target node::
569

    
570
   $ gnt-instance migrate -n %TARGET_NODE% %INSTANCE_NAME%
571

    
572
Moving an instance (offline)
573
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
574

    
575
If an instance has not been create as mirrored, then the only way to
576
change its primary node is to execute the move command::
577

    
578
  $ gnt-instance move -n %NEW_NODE% %INSTANCE%
579

    
580
This has a few prerequisites:
581

    
582
- the instance must be stopped
583
- its current primary node must be on-line and healthy
584
- the disks of the instance must not have any errors
585

    
586
Since this operation actually copies the data from the old node to the
587
new node, expect it to take proportional to the size of the instance's
588
disks and the speed of both the nodes' I/O system and their networking.
589

    
590
Disk operations
591
+++++++++++++++
592

    
593
Disk failures are a common cause of errors in any server
594
deployment. Ganeti offers protection from single-node failure if your
595
instances were created in HA mode, and it also offers ways to restore
596
redundancy after a failure.
597

    
598
Preparing for disk operations
599
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
600

    
601
It is important to note that for Ganeti to be able to do any disk
602
operation, the Linux machines on top of which Ganeti runs must be
603
consistent; for LVM, this means that the LVM commands must not return
604
failures; it is common that after a complete disk failure, any LVM
605
command aborts with an error similar to::
606

    
607
  $ vgs
608
  /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
609
  /dev/sdb1: read failed after 0 of 4096 at 750153695232: Input/output error
610
  /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
611
  Couldn't find device with uuid 't30jmN-4Rcf-Fr5e-CURS-pawt-z0jU-m1TgeJ'.
612
  Couldn't find all physical volumes for volume group xenvg.
613

    
614
Before restoring an instance's disks to healthy status, it's needed to
615
fix the volume group used by Ganeti so that we can actually create and
616
manage the logical volumes. This is usually done in a multi-step
617
process:
618

    
619
#. first, if the disk is completely gone and LVM commands exit with
620
   “Couldn't find device with uuid…” then you need to run the command::
621

    
622
    $ vgreduce --removemissing %VOLUME_GROUP%
623

    
624
#. after the above command, the LVM commands should be executing
625
   normally (warnings are normal, but the commands will not fail
626
   completely).
627

    
628
#. if the failed disk is still visible in the output of the ``pvs``
629
   command, you need to deactivate it from allocations by running::
630

    
631
    $ pvs -x n /dev/%DISK%
632

    
633
At this point, the volume group should be consistent and any bad
634
physical volumes should not longer be available for allocation.
635

    
636
Note that since version 2.1 Ganeti provides some commands to automate
637
these two operations, see :ref:`storage-units-label`.
638

    
639
.. _rest-redundancy-label:
640

    
641
Restoring redundancy for DRBD-based instances
642
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
643

    
644
A DRBD instance has two nodes, and the storage on one of them has
645
failed. Depending on which node (primary or secondary) has failed, you
646
have three options at hand:
647

    
648
- if the storage on the primary node has failed, you need to re-create
649
  the disks on it
650
- if the storage on the secondary node has failed, you can either
651
  re-create the disks on it or change the secondary and recreate
652
  redundancy on the new secondary node
653

    
654
Of course, at any point it's possible to force re-creation of disks even
655
though everything is already fine.
656

    
657
For all three cases, the ``replace-disks`` operation can be used::
658

    
659
  # re-create disks on the primary node
660
  $ gnt-instance replace-disks -p %INSTANCE_NAME%
661
  # re-create disks on the current secondary
662
  $ gnt-instance replace-disks -s %INSTANCE_NAME%
663
  # change the secondary node, via manual specification
664
  $ gnt-instance replace-disks -n %NODE% %INSTANCE_NAME%
665
  # change the secondary node, via an iallocator script
666
  $ gnt-instance replace-disks -I %SCRIPT% %INSTANCE_NAME%
667
  # since Ganeti 2.1: automatically fix the primary or secondary node
668
  $ gnt-instance replace-disks -a %INSTANCE_NAME%
669

    
670
Since the process involves copying all data from the working node to the
671
target node, it will take a while, depending on the instance's disk
672
size, node I/O system and network speed. But it is (barring any network
673
interruption) completely transparent for the instance.
674

    
675
Re-creating disks for non-redundant instances
676
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
677

    
678
.. versionadded:: 2.1
679

    
680
For non-redundant instances, there isn't a copy (except backups) to
681
re-create the disks. But it's possible to at-least re-create empty
682
disks, after which a reinstall can be run, via the ``recreate-disks``
683
command::
684

    
685
  $ gnt-instance recreate-disks %INSTANCE%
686

    
687
Note that this will fail if the disks already exists. The instance can
688
be assigned to new nodes automatically by specifying an iallocator
689
through the ``--iallocator`` option.
690

    
691
Conversion of an instance's disk type
692
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
693

    
694
It is possible to convert between a non-redundant instance of type
695
``plain`` (LVM storage) and redundant ``drbd`` via the ``gnt-instance
696
modify`` command::
697

    
698
  # start with a non-redundant instance
699
  $ gnt-instance add -t plain ... %INSTANCE%
700

    
701
  # later convert it to redundant
702
  $ gnt-instance stop %INSTANCE%
703
  $ gnt-instance modify -t drbd -n %NEW_SECONDARY% %INSTANCE%
704
  $ gnt-instance start %INSTANCE%
705

    
706
  # and convert it back
707
  $ gnt-instance stop %INSTANCE%
708
  $ gnt-instance modify -t plain %INSTANCE%
709
  $ gnt-instance start %INSTANCE%
710

    
711
The conversion must be done while the instance is stopped, and
712
converting from plain to drbd template presents a small risk, especially
713
if the instance has multiple disks and/or if one node fails during the
714
conversion procedure). As such, it's recommended (as always) to make
715
sure that downtime for manual recovery is acceptable and that the
716
instance has up-to-date backups.
717

    
718
Debugging instances
719
+++++++++++++++++++
720

    
721
Accessing an instance's disks
722
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
723

    
724
From an instance's primary node you can have access to its disks. Never
725
ever mount the underlying logical volume manually on a fault tolerant
726
instance, or will break replication and your data will be
727
inconsistent. The correct way to access an instance's disks is to run
728
(on the master node, as usual) the command::
729

    
730
  $ gnt-instance activate-disks %INSTANCE%
731

    
732
And then, *on the primary node of the instance*, access the device that
733
gets created. For example, you could mount the given disks, then edit
734
files on the filesystem, etc.
735

    
736
Note that with partitioned disks (as opposed to whole-disk filesystems),
737
you will need to use a tool like :manpage:`kpartx(8)`::
738

    
739
  # on node1
740
  $ gnt-instance activate-disks %instance1%
741
  node3:disk/0:…
742
  $ ssh node3
743
  # on node 3
744
  $ kpartx -l /dev/…
745
  $ kpartx -a /dev/…
746
  $ mount /dev/mapper/… /mnt/
747
  # edit files under mnt as desired
748
  $ umount /mnt/
749
  $ kpartx -d /dev/…
750
  $ exit
751
  # back to node 1
752

    
753
After you've finished you can deactivate them with the deactivate-disks
754
command, which works in the same way::
755

    
756
  $ gnt-instance deactivate-disks %INSTANCE%
757

    
758
Note that if any process started by you is still using the disks, the
759
above command will error out, and you **must** cleanup and ensure that
760
the above command runs successfully before you start the instance,
761
otherwise the instance will suffer corruption.
762

    
763
Accessing an instance's console
764
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
765

    
766
The command to access a running instance's console is::
767

    
768
  $ gnt-instance console %INSTANCE_NAME%
769

    
770
Use the console normally and then type ``^]`` when done, to exit.
771

    
772
Other instance operations
773
+++++++++++++++++++++++++
774

    
775
Reboot
776
~~~~~~
777

    
778
There is a wrapper command for rebooting instances::
779

    
780
  $ gnt-instance reboot %instance2%
781

    
782
By default, this does the equivalent of shutting down and then starting
783
the instance, but it accepts parameters to perform a soft-reboot (via
784
the hypervisor), a hard reboot (hypervisor shutdown and then startup) or
785
a full one (the default, which also de-configures and then configures
786
again the disks of the instance).
787

    
788
Instance OS definitions debugging
789
+++++++++++++++++++++++++++++++++
790

    
791
Should you have any problems with instance operating systems the command
792
to see a complete status for all your nodes is::
793

    
794
   $ gnt-os diagnose
795

    
796
.. _instance-relocation-label:
797

    
798
Instance relocation
799
~~~~~~~~~~~~~~~~~~~
800

    
801
While it is not possible to move an instance from nodes ``(A, B)`` to
802
nodes ``(C, D)`` in a single move, it is possible to do so in a few
803
steps::
804

    
805
  # instance is located on A, B
806
  $ gnt-instance replace-disks -n %nodeC% %instance1%
807
  # instance has moved from (A, B) to (A, C)
808
  # we now flip the primary/secondary nodes
809
  $ gnt-instance migrate %instance1%
810
  # instance lives on (C, A)
811
  # we can then change A to D via:
812
  $ gnt-instance replace-disks -n %nodeD% %instance1%
813

    
814
Which brings it into the final configuration of ``(C, D)``. Note that we
815
needed to do two replace-disks operation (two copies of the instance
816
disks), because we needed to get rid of both the original nodes (A and
817
B).
818

    
819
Node operations
820
---------------
821

    
822
There are much fewer node operations available than for instances, but
823
they are equivalently important for maintaining a healthy cluster.
824

    
825
Add/readd
826
+++++++++
827

    
828
It is at any time possible to extend the cluster with one more node, by
829
using the node add operation::
830

    
831
  $ gnt-node add %NEW_NODE%
832

    
833
If the cluster has a replication network defined, then you need to pass
834
the ``-s REPLICATION_IP`` parameter to this option.
835

    
836
A variation of this command can be used to re-configure a node if its
837
Ganeti configuration is broken, for example if it has been reinstalled
838
by mistake::
839

    
840
  $ gnt-node add --readd %EXISTING_NODE%
841

    
842
This will reinitialise the node as if it's been newly added, but while
843
keeping its existing configuration in the cluster (primary/secondary IP,
844
etc.), in other words you won't need to use ``-s`` here.
845

    
846
Changing the node role
847
++++++++++++++++++++++
848

    
849
A node can be in different roles, as explained in the
850
:ref:`terminology-label` section. Promoting a node to the master role is
851
special, while the other roles are handled all via a single command.
852

    
853
Failing over the master node
854
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
855

    
856
If you want to promote a different node to the master role (for whatever
857
reason), run on any other master-candidate node the command::
858

    
859
  $ gnt-cluster master-failover
860

    
861
and the node you ran it on is now the new master. In case you try to run
862
this on a non master-candidate node, you will get an error telling you
863
which nodes are valid.
864

    
865
Changing between the other roles
866
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
867

    
868
The ``gnt-node modify`` command can be used to select a new role::
869

    
870
  # change to master candidate
871
  $ gnt-node modify -C yes %NODE%
872
  # change to drained status
873
  $ gnt-node modify -D yes %NODE%
874
  # change to offline status
875
  $ gnt-node modify -O yes %NODE%
876
  # change to regular mode (reset all flags)
877
  $ gnt-node modify -O no -D no -C no %NODE%
878

    
879
Note that the cluster requires that at any point in time, a certain
880
number of nodes are master candidates, so changing from master candidate
881
to other roles might fail. It is recommended to either force the
882
operation (via the ``--force`` option) or first change the number of
883
master candidates in the cluster - see :ref:`cluster-config-label`.
884

    
885
Evacuating nodes
886
++++++++++++++++
887

    
888
There are two steps of moving instances off a node:
889

    
890
- moving the primary instances (actually converting them into secondary
891
  instances)
892
- moving the secondary instances (including any instances converted in
893
  the step above)
894

    
895
Primary instance conversion
896
~~~~~~~~~~~~~~~~~~~~~~~~~~~
897

    
898
For this step, you can use either individual instance move
899
commands (as seen in :ref:`instance-change-primary-label`) or the bulk
900
per-node versions; these are::
901

    
902
  $ gnt-node migrate %NODE%
903
  $ gnt-node evacuate -s %NODE%
904

    
905
Note that the instance “move” command doesn't currently have a node
906
equivalent.
907

    
908
Both these commands, or the equivalent per-instance command, will make
909
this node the secondary node for the respective instances, whereas their
910
current secondary node will become primary. Note that it is not possible
911
to change in one step the primary node to another node as primary, while
912
keeping the same secondary node.
913

    
914
Secondary instance evacuation
915
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
916

    
917
For the evacuation of secondary instances, a command called
918
:command:`gnt-node evacuate` is provided and its syntax is::
919

    
920
  $ gnt-node evacuate -I %IALLOCATOR_SCRIPT% %NODE%
921
  $ gnt-node evacuate -n %DESTINATION_NODE% %NODE%
922

    
923
The first version will compute the new secondary for each instance in
924
turn using the given iallocator script, whereas the second one will
925
simply move all instances to DESTINATION_NODE.
926

    
927
Removal
928
+++++++
929

    
930
Once a node no longer has any instances (neither primary nor secondary),
931
it's easy to remove it from the cluster::
932

    
933
  $ gnt-node remove %NODE_NAME%
934

    
935
This will deconfigure the node, stop the ganeti daemons on it and leave
936
it hopefully like before it joined to the cluster.
937

    
938
Replication network changes
939
+++++++++++++++++++++++++++
940

    
941
The :command:`gnt-node modify -s` command can be used to change the
942
secondary IP of a node. This operation can only be performed if:
943

    
944
- No instance is active on the target node
945
- The new target IP is reachable from the master's secondary IP
946

    
947
Also this operation will not allow to change a node from single-homed
948
(same primary and secondary ip) to multi-homed (separate replication
949
network) or vice versa, unless:
950

    
951
- The target node is the master node and `--force` is passed.
952
- The target cluster is single-homed and the new primary ip is a change
953
  to single homed for a particular node.
954
- The target cluster is multi-homed and the new primary ip is a change
955
  to multi homed for a particular node.
956

    
957
For example to do a single-homed to multi-homed conversion::
958

    
959
  $ gnt-node modify --force -s %SECONDARY_IP% %MASTER_NAME%
960
  $ gnt-node modify -s %SECONDARY_IP% %NODE1_NAME%
961
  $ gnt-node modify -s %SECONDARY_IP% %NODE2_NAME%
962
  $ gnt-node modify -s %SECONDARY_IP% %NODE3_NAME%
963
  ...
964

    
965
The same commands can be used for multi-homed to single-homed except the
966
secondary IPs should be the same as the primaries for each node, for
967
that case.
968

    
969
Storage handling
970
++++++++++++++++
971

    
972
When using LVM (either standalone or with DRBD), it can become tedious
973
to debug and fix it in case of errors. Furthermore, even file-based
974
storage can become complicated to handle manually on many hosts. Ganeti
975
provides a couple of commands to help with automation.
976

    
977
Logical volumes
978
~~~~~~~~~~~~~~~
979

    
980
This is a command specific to LVM handling. It allows listing the
981
logical volumes on a given node or on all nodes and their association to
982
instances via the ``volumes`` command::
983

    
984
  $ gnt-node volumes
985
  Node  PhysDev   VG    Name             Size Instance
986
  node1 /dev/sdb1 xenvg e61fbc97-….disk0 512M instance17
987
  node1 /dev/sdb1 xenvg ebd1a7d1-….disk0 512M instance19
988
  node2 /dev/sdb1 xenvg 0af08a3d-….disk0 512M instance20
989
  node2 /dev/sdb1 xenvg cc012285-….disk0 512M instance16
990
  node2 /dev/sdb1 xenvg f0fac192-….disk0 512M instance18
991

    
992
The above command maps each logical volume to a volume group and
993
underlying physical volume and (possibly) to an instance.
994

    
995
.. _storage-units-label:
996

    
997
Generalized storage handling
998
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
999

    
1000
.. versionadded:: 2.1
1001

    
1002
Starting with Ganeti 2.1, a new storage framework has been implemented
1003
that tries to abstract the handling of the storage type the cluster
1004
uses.
1005

    
1006
First is listing the backend storage and their space situation::
1007

    
1008
  $ gnt-node list-storage
1009
  Node  Name        Size Used   Free
1010
  node1 /dev/sda7 673.8G   0M 673.8G
1011
  node1 /dev/sdb1 698.6G 1.5G 697.1G
1012
  node2 /dev/sda7 673.8G   0M 673.8G
1013
  node2 /dev/sdb1 698.6G 1.0G 697.6G
1014

    
1015
The default is to list LVM physical volumes. It's also possible to list
1016
the LVM volume groups::
1017

    
1018
  $ gnt-node list-storage -t lvm-vg
1019
  Node  Name  Size
1020
  node1 xenvg 1.3T
1021
  node2 xenvg 1.3T
1022

    
1023
Next is repairing storage units, which is currently only implemented for
1024
volume groups and does the equivalent of ``vgreduce --removemissing``::
1025

    
1026
  $ gnt-node repair-storage %node2% lvm-vg xenvg
1027
  Sun Oct 25 22:21:45 2009 Repairing storage unit 'xenvg' on node2 ...
1028

    
1029
Last is the modification of volume properties, which is (again) only
1030
implemented for LVM physical volumes and allows toggling the
1031
``allocatable`` value::
1032

    
1033
  $ gnt-node modify-storage --allocatable=no %node2% lvm-pv /dev/%sdb1%
1034

    
1035
Use of the storage commands
1036
~~~~~~~~~~~~~~~~~~~~~~~~~~~
1037

    
1038
All these commands are needed when recovering a node from a disk
1039
failure:
1040

    
1041
- first, we need to recover from complete LVM failure (due to missing
1042
  disk), by running the ``repair-storage`` command
1043
- second, we need to change allocation on any partially-broken disk
1044
  (i.e. LVM still sees it, but it has bad blocks) by running
1045
  ``modify-storage``
1046
- then we can evacuate the instances as needed
1047

    
1048

    
1049
Cluster operations
1050
------------------
1051

    
1052
Beside the cluster initialisation command (which is detailed in the
1053
:doc:`install` document) and the master failover command which is
1054
explained under node handling, there are a couple of other cluster
1055
operations available.
1056

    
1057
.. _cluster-config-label:
1058

    
1059
Standard operations
1060
+++++++++++++++++++
1061

    
1062
One of the few commands that can be run on any node (not only the
1063
master) is the ``getmaster`` command::
1064

    
1065
  # on node2
1066
  $ gnt-cluster getmaster
1067
  node1.example.com
1068

    
1069
It is possible to query and change global cluster parameters via the
1070
``info`` and ``modify`` commands::
1071

    
1072
  $ gnt-cluster info
1073
  Cluster name: cluster.example.com
1074
  Cluster UUID: 07805e6f-f0af-4310-95f1-572862ee939c
1075
  Creation time: 2009-09-25 05:04:15
1076
  Modification time: 2009-10-18 22:11:47
1077
  Master node: node1.example.com
1078
  Architecture (this node): 64bit (x86_64)
1079
1080
  Tags: foo
1081
  Default hypervisor: xen-pvm
1082
  Enabled hypervisors: xen-pvm
1083
  Hypervisor parameters:
1084
    - xen-pvm:
1085
        root_path: /dev/sda1
1086
1087
  Cluster parameters:
1088
    - candidate pool size: 10
1089
1090
  Default instance parameters:
1091
    - default:
1092
        memory: 128
1093
1094
  Default nic parameters:
1095
    - default:
1096
        link: xen-br0
1097
1098

    
1099
There various parameters above can be changed via the ``modify``
1100
commands as follows:
1101

    
1102
- the hypervisor parameters can be changed via ``modify -H
1103
  xen-pvm:root_path=…``, and so on for other hypervisors/key/values
1104
- the "default instance parameters" are changeable via ``modify -B
1105
  parameter=value…`` syntax
1106
- the cluster parameters are changeable via separate options to the
1107
  modify command (e.g. ``--candidate-pool-size``, etc.)
1108

    
1109
For detailed option list see the :manpage:`gnt-cluster(8)` man page.
1110

    
1111
The cluster version can be obtained via the ``version`` command::
1112
  $ gnt-cluster version
1113
  Software version: 2.1.0
1114
  Internode protocol: 20
1115
  Configuration format: 2010000
1116
  OS api version: 15
1117
  Export interface: 0
1118

    
1119
This is not very useful except when debugging Ganeti.
1120

    
1121
Global node commands
1122
++++++++++++++++++++
1123

    
1124
There are two commands provided for replicating files to all nodes of a
1125
cluster and for running commands on all the nodes::
1126

    
1127
  $ gnt-cluster copyfile %/path/to/file%
1128
  $ gnt-cluster command %ls -l /path/to/file%
1129

    
1130
These are simple wrappers over scp/ssh and more advanced usage can be
1131
obtained using :manpage:`dsh(1)` and similar commands. But they are
1132
useful to update an OS script from the master node, for example.
1133

    
1134
Cluster verification
1135
++++++++++++++++++++
1136

    
1137
There are three commands that relate to global cluster checks. The first
1138
one is ``verify`` which gives an overview on the cluster state,
1139
highlighting any issues. In normal operation, this command should return
1140
no ``ERROR`` messages::
1141

    
1142
  $ gnt-cluster verify
1143
  Sun Oct 25 23:08:58 2009 * Verifying global settings
1144
  Sun Oct 25 23:08:58 2009 * Gathering data (2 nodes)
1145
  Sun Oct 25 23:09:00 2009 * Verifying node status
1146
  Sun Oct 25 23:09:00 2009 * Verifying instance status
1147
  Sun Oct 25 23:09:00 2009 * Verifying orphan volumes
1148
  Sun Oct 25 23:09:00 2009 * Verifying remaining instances
1149
  Sun Oct 25 23:09:00 2009 * Verifying N+1 Memory redundancy
1150
  Sun Oct 25 23:09:00 2009 * Other Notes
1151
  Sun Oct 25 23:09:00 2009   - NOTICE: 5 non-redundant instance(s) found.
1152
  Sun Oct 25 23:09:00 2009 * Hooks Results
1153

    
1154
The second command is ``verify-disks``, which checks that the instance's
1155
disks have the correct status based on the desired instance state
1156
(up/down)::
1157

    
1158
  $ gnt-cluster verify-disks
1159

    
1160
Note that this command will show no output when disks are healthy.
1161

    
1162
The last command is used to repair any discrepancies in Ganeti's
1163
recorded disk size and the actual disk size (disk size information is
1164
needed for proper activation and growth of DRBD-based disks)::
1165

    
1166
  $ gnt-cluster repair-disk-sizes
1167
  Sun Oct 25 23:13:16 2009  - INFO: Disk 0 of instance instance1 has mismatched size, correcting: recorded 512, actual 2048
1168
  Sun Oct 25 23:13:17 2009  - WARNING: Invalid result from node node4, ignoring node results
1169

    
1170
The above shows one instance having wrong disk size, and a node which
1171
returned invalid data, and thus we ignored all primary instances of that
1172
node.
1173

    
1174
Configuration redistribution
1175
++++++++++++++++++++++++++++
1176

    
1177
If the verify command complains about file mismatches between the master
1178
and other nodes, due to some node problems or if you manually modified
1179
configuration files, you can force an push of the master configuration
1180
to all other nodes via the ``redist-conf`` command::
1181

    
1182
  $ gnt-cluster redist-conf
1183

    
1184
This command will be silent unless there are problems sending updates to
1185
the other nodes.
1186

    
1187

    
1188
Cluster renaming
1189
++++++++++++++++
1190

    
1191
It is possible to rename a cluster, or to change its IP address, via the
1192
``rename`` command. If only the IP has changed, you need to pass the
1193
current name and Ganeti will realise its IP has changed::
1194

    
1195
  $ gnt-cluster rename %cluster.example.com%
1196
  This will rename the cluster to 'cluster.example.com'. If
1197
  you are connected over the network to the cluster name, the operation
1198
  is very dangerous as the IP address will be removed from the node and
1199
  the change may not go through. Continue?
1200
  y/[n]/?: %y%
1201
  Failure: prerequisites not met for this operation:
1202
  Neither the name nor the IP address of the cluster has changed
1203

    
1204
In the above output, neither value has changed since the cluster
1205
initialisation so the operation is not completed.
1206

    
1207
Queue operations
1208
++++++++++++++++
1209

    
1210
The job queue execution in Ganeti 2.0 and higher can be inspected,
1211
suspended and resumed via the ``queue`` command::
1212

    
1213
  $ gnt-cluster queue info
1214
  The drain flag is unset
1215
  $ gnt-cluster queue drain
1216
  $ gnt-instance stop %instance1%
1217
  Failed to submit job for instance1: Job queue is drained, refusing job
1218
  $ gnt-cluster queue info
1219
  The drain flag is set
1220
  $ gnt-cluster queue undrain
1221

    
1222
This is most useful if you have an active cluster and you need to
1223
upgrade the Ganeti software, or simply restart the software on any node:
1224

    
1225
#. suspend the queue via ``queue drain``
1226
#. wait until there are no more running jobs via ``gnt-job list``
1227
#. restart the master or another node, or upgrade the software
1228
#. resume the queue via ``queue undrain``
1229

    
1230
.. note:: this command only stores a local flag file, and if you
1231
   failover the master, it will not have effect on the new master.
1232

    
1233

    
1234
Watcher control
1235
+++++++++++++++
1236

    
1237
The :manpage:`ganeti-watcher(8)` is a program, usually scheduled via
1238
``cron``, that takes care of cluster maintenance operations (restarting
1239
downed instances, activating down DRBD disks, etc.). However, during
1240
maintenance and troubleshooting, this can get in your way; disabling it
1241
via commenting out the cron job is not so good as this can be
1242
forgotten. Thus there are some commands for automated control of the
1243
watcher: ``pause``, ``info`` and ``continue``::
1244

    
1245
  $ gnt-cluster watcher info
1246
  The watcher is not paused.
1247
  $ gnt-cluster watcher pause %1h%
1248
  The watcher is paused until Mon Oct 26 00:30:37 2009.
1249
  $ gnt-cluster watcher info
1250
  The watcher is paused until Mon Oct 26 00:30:37 2009.
1251
  $ ganeti-watcher -d
1252
  2009-10-25 23:30:47,984:  pid=28867 ganeti-watcher:486 DEBUG Pause has been set, exiting
1253
  $ gnt-cluster watcher continue
1254
  The watcher is no longer paused.
1255
  $ ganeti-watcher -d
1256
  2009-10-25 23:31:04,789:  pid=28976 ganeti-watcher:345 DEBUG Archived 0 jobs, left 0
1257
  2009-10-25 23:31:05,884:  pid=28976 ganeti-watcher:280 DEBUG Got data from cluster, writing instance status file
1258
  2009-10-25 23:31:06,061:  pid=28976 ganeti-watcher:150 DEBUG Data didn't change, just touching status file
1259
  $ gnt-cluster watcher info
1260
  The watcher is not paused.
1261

    
1262
The exact details of the argument to the ``pause`` command are available
1263
in the manpage.
1264

    
1265
.. note:: this command only stores a local flag file, and if you
1266
   failover the master, it will not have effect on the new master.
1267

    
1268
Node auto-maintenance
1269
+++++++++++++++++++++
1270

    
1271
If the cluster parameter ``maintain_node_health`` is enabled (see the
1272
manpage for :command:`gnt-cluster`, the init and modify subcommands),
1273
then the following will happen automatically:
1274

    
1275
- the watcher will shutdown any instances running on offline nodes
1276
- the watcher will deactivate any DRBD devices on offline nodes
1277

    
1278
In the future, more actions are planned, so only enable this parameter
1279
if the nodes are completely dedicated to Ganeti; otherwise it might be
1280
possible to lose data due to auto-maintenance actions.
1281

    
1282
Removing a cluster entirely
1283
+++++++++++++++++++++++++++
1284

    
1285
The usual method to cleanup a cluster is to run ``gnt-cluster destroy``
1286
however if the Ganeti installation is broken in any way then this will
1287
not run.
1288

    
1289
It is possible in such a case to cleanup manually most if not all traces
1290
of a cluster installation by following these steps on all of the nodes:
1291

    
1292
1. Shutdown all instances. This depends on the virtualisation method
1293
   used (Xen, KVM, etc.):
1294

    
1295
  - Xen: run ``xm list`` and ``xm destroy`` on all the non-Domain-0
1296
    instances
1297
  - KVM: kill all the KVM processes
1298
  - chroot: kill all processes under the chroot mountpoints
1299

    
1300
2. If using DRBD, shutdown all DRBD minors (which should by at this time
1301
   no-longer in use by instances); on each node, run ``drbdsetup
1302
   /dev/drbdN down`` for each active DRBD minor.
1303

    
1304
3. If using LVM, cleanup the Ganeti volume group; if only Ganeti created
1305
   logical volumes (and you are not sharing the volume group with the
1306
   OS, for example), then simply running ``lvremove -f xenvg`` (replace
1307
   'xenvg' with your volume group name) should do the required cleanup.
1308

    
1309
4. If using file-based storage, remove recursively all files and
1310
   directories under your file-storage directory: ``rm -rf
1311
   /srv/ganeti/file-storage/*`` replacing the path with the correct path
1312
   for your cluster.
1313

    
1314
5. Stop the ganeti daemons (``/etc/init.d/ganeti stop``) and kill any
1315
   that remain alive (``pgrep ganeti`` and ``pkill ganeti``).
1316

    
1317
6. Remove the ganeti state directory (``rm -rf /var/lib/ganeti/*``),
1318
   replacing the path with the correct path for your installation.
1319

    
1320
7. If using RBD, run ``rbd unmap /dev/rbdN`` to unmap the RBD disks.
1321
   Then remove the RBD disk images used by Ganeti, identified by their
1322
   UUIDs (``rbd rm uuid.rbd.diskN``).
1323

    
1324
On the master node, remove the cluster from the master-netdev (usually
1325
``xen-br0`` for bridged mode, otherwise ``eth0`` or similar), by running
1326
``ip a del $clusterip/32 dev xen-br0`` (use the correct cluster ip and
1327
network device name).
1328

    
1329
At this point, the machines are ready for a cluster creation; in case
1330
you want to remove Ganeti completely, you need to also undo some of the
1331
SSH changes and log directories:
1332

    
1333
- ``rm -rf /var/log/ganeti /srv/ganeti`` (replace with the correct
1334
  paths)
1335
- remove from ``/root/.ssh`` the keys that Ganeti added (check the
1336
  ``authorized_keys`` and ``id_dsa`` files)
1337
- regenerate the host's SSH keys (check the OpenSSH startup scripts)
1338
- uninstall Ganeti
1339

    
1340
Otherwise, if you plan to re-create the cluster, you can just go ahead
1341
and rerun ``gnt-cluster init``.
1342

    
1343
Replacing the SSH and SSL keys
1344
++++++++++++++++++++++++++++++
1345

    
1346
Ganeti uses both SSL and SSH keys, and actively modifies the SSH keys on
1347
the nodes.  As result, in order to replace these keys, a few extra steps
1348
need to be followed: :doc:`cluster-keys-replacement`
1349

    
1350
Monitoring the cluster
1351
----------------------
1352

    
1353
Starting with Ganeti 2.8, a monitoring daemon is available, providing
1354
information about the status and the performance of the system.
1355

    
1356
The monitoring daemon runs on every node, listening on TCP port 1815. Each
1357
instance of the daemon provides information related to the node it is running
1358
on.
1359

    
1360
.. include:: monitoring-query-format.rst
1361

    
1362
Tags handling
1363
-------------
1364

    
1365
The tags handling (addition, removal, listing) is similar for all the
1366
objects that support it (instances, nodes, and the cluster).
1367

    
1368
Limitations
1369
+++++++++++
1370

    
1371
Note that the set of characters present in a tag and the maximum tag
1372
length are restricted. Currently the maximum length is 128 characters,
1373
there can be at most 4096 tags per object, and the set of characters is
1374
comprised by alphanumeric characters and additionally ``.+*/:@-``.
1375

    
1376
Operations
1377
++++++++++
1378

    
1379
Tags can be added via ``add-tags``::
1380

    
1381
  $ gnt-instance add-tags %INSTANCE% %a% %b% %c%
1382
  $ gnt-node add-tags %INSTANCE% %a% %b% %c%
1383
  $ gnt-cluster add-tags %a% %b% %c%
1384

    
1385

    
1386
The above commands add three tags to an instance, to a node and to the
1387
cluster. Note that the cluster command only takes tags as arguments,
1388
whereas the node and instance commands first required the node and
1389
instance name.
1390

    
1391
Tags can also be added from a file, via the ``--from=FILENAME``
1392
argument. The file is expected to contain one tag per line.
1393

    
1394
Tags can also be remove via a syntax very similar to the add one::
1395

    
1396
  $ gnt-instance remove-tags %INSTANCE% %a% %b% %c%
1397

    
1398
And listed via::
1399

    
1400
  $ gnt-instance list-tags
1401
  $ gnt-node list-tags
1402
  $ gnt-cluster list-tags
1403

    
1404
Global tag search
1405
+++++++++++++++++
1406

    
1407
It is also possible to execute a global search on the all tags defined
1408
in the cluster configuration, via a cluster command::
1409

    
1410
  $ gnt-cluster search-tags %REGEXP%
1411

    
1412
The parameter expected is a regular expression (see
1413
:manpage:`regex(7)`). This will return all tags that match the search,
1414
together with the object they are defined in (the names being show in a
1415
hierarchical kind of way)::
1416

    
1417
  $ gnt-cluster search-tags %o%
1418
  /cluster foo
1419
  /instances/instance1 owner:bar
1420

    
1421
Autorepair
1422
----------
1423

    
1424
The tool ``harep`` can be used to automatically fix some problems that are
1425
present in the cluster.
1426

    
1427
It is mainly meant to be regularly and automatically executed
1428
as a cron job. This is quite evident by considering that, when executed, it does
1429
not immediately fix all the issues of the instances of the cluster, but it
1430
cycles the instances through a series of states, one at every ``harep``
1431
execution. Every state performs a step towards the resolution of the problem.
1432
This process goes on until the instance is brought back to the healthy state,
1433
or the tool realizes that it is not able to fix the instance, and
1434
therefore marks it as in failure state.
1435

    
1436
Allowing harep to act on the cluster
1437
++++++++++++++++++++++++++++++++++++
1438

    
1439
By default, ``harep`` checks the status of the cluster but it is not allowed to
1440
perform any modification. Modification must be explicitly allowed by an
1441
appropriate use of tags. Tagging can be applied at various levels, and can
1442
enable different kinds of autorepair, as hereafter described.
1443

    
1444
All the tags that authorize ``harep`` to perform modifications follow this
1445
syntax::
1446

    
1447
  ganeti:watcher:autorepair:<type>
1448

    
1449
where ``<type>`` indicates the kind of intervention that can be performed. Every
1450
possible value of ``<type>`` includes at least all the authorization of the
1451
previous one, plus its own. The possible values, in increasing order of
1452
severity, are:
1453

    
1454
- ``fix-storage`` allows a disk replacement or another operation that
1455
  fixes the instance backend storage without affecting the instance
1456
  itself. This can for example recover from a broken drbd secondary, but
1457
  risks data loss if something is wrong on the primary but the secondary
1458
  was somehow recoverable.
1459
- ``migrate`` allows an instance migration. This can recover from a
1460
  drained primary, but can cause an instance crash in some cases (bugs).
1461
- ``failover`` allows instance reboot on the secondary. This can recover
1462
  from an offline primary, but the instance will lose its running state.
1463
- ``reinstall`` allows disks to be recreated and an instance to be
1464
  reinstalled. This can recover from primary&secondary both being
1465
  offline, or from an offline primary in the case of non-redundant
1466
  instances. It causes data loss.
1467

    
1468
These autorepair tags can be applied to a cluster, a nodegroup or an instance,
1469
and will act where they are applied and to everything in the entities sub-tree
1470
(e.g. a tag applied to a nodegroup will apply to all the instances contained in
1471
that nodegroup, but not to the rest of the cluster).
1472

    
1473
If there are multiple ``ganeti:watcher:autorepair:<type>`` tags in an
1474
object (cluster, node group or instance), the least destructive tag
1475
takes precedence. When multiplicity happens across objects, the nearest
1476
tag wins. For example, if in a cluster with two instances, *I1* and
1477
*I2*, *I1* has ``failover``, and the cluster itself has both
1478
``fix-storage`` and ``reinstall``, *I1* will end up with ``failover``
1479
and *I2* with ``fix-storage``.
1480

    
1481
Limiting harep
1482
++++++++++++++
1483

    
1484
Sometimes it is useful to stop harep from performing its task temporarily,
1485
and it is useful to be able to do so without distrupting its configuration, that
1486
is, without removing the authorization tags. In order to do this, suspend tags
1487
are provided.
1488

    
1489
Suspend tags can be added to cluster, nodegroup or instances, and act on the
1490
entire entities sub-tree. No operation will be performed by ``harep`` on the
1491
instances protected by a suspend tag. Their syntax is as follows::
1492

    
1493
  ganeti:watcher:autorepair:suspend[:<timestamp>]
1494

    
1495
If there are multiple suspend tags in an object, the form without timestamp
1496
takes precedence (permanent suspension); or, if all object tags have a
1497
timestamp, the one with the highest timestamp.
1498

    
1499
Tags with a timestamp will be automatically removed when the time indicated by
1500
the timestamp is passed. Indefinite suspension tags have to be removed manually.
1501

    
1502
Result reporting
1503
++++++++++++++++
1504

    
1505
Harep will report about the result of its actions both through its CLI, and by
1506
adding tags to the instances it operated on. Such tags will follow the syntax
1507
hereby described::
1508

    
1509
  ganeti:watcher:autorepair:result:<type>:<id>:<timestamp>:<result>:<jobs>
1510

    
1511
If this tag is present a repair of type ``type`` has been performed on
1512
the instance and has been completed by ``timestamp``. The result is
1513
either ``success``, ``failure`` or ``enoperm``, and jobs is a
1514
*+*-separated list of jobs that were executed for this repair.
1515

    
1516
An ``enoperm`` result is an error state due to permission problems. It
1517
is returned when the repair cannot proceed because it would require to perform
1518
an operation that is not allowed by the ``ganeti:watcher:autorepair:<type>`` tag
1519
that is defining the instance autorepair permissions.
1520

    
1521
NB: if an instance repair ends up in a failure state, it will not be touched
1522
again by ``harep`` until it has been manually fixed by the system administrator
1523
and the ``ganeti:watcher:autorepair:result:failure:*`` tag has been manually
1524
removed.
1525

    
1526
Job operations
1527
--------------
1528

    
1529
The various jobs submitted by the instance/node/cluster commands can be
1530
examined, canceled and archived by various invocations of the
1531
``gnt-job`` command.
1532

    
1533
First is the job list command::
1534

    
1535
  $ gnt-job list
1536
  17771 success INSTANCE_QUERY_DATA
1537
  17773 success CLUSTER_VERIFY_DISKS
1538
  17775 success CLUSTER_REPAIR_DISK_SIZES
1539
  17776 error   CLUSTER_RENAME(cluster.example.com)
1540
  17780 success CLUSTER_REDIST_CONF
1541
  17792 success INSTANCE_REBOOT(instance1.example.com)
1542

    
1543
More detailed information about a job can be found via the ``info``
1544
command::
1545

    
1546
  $ gnt-job info %17776%
1547
  Job ID: 17776
1548
    Status: error
1549
    Received:         2009-10-25 23:18:02.180569
1550
    Processing start: 2009-10-25 23:18:02.200335 (delta 0.019766s)
1551
    Processing end:   2009-10-25 23:18:02.279743 (delta 0.079408s)
1552
    Total processing time: 0.099174 seconds
1553
    Opcodes:
1554
      OP_CLUSTER_RENAME
1555
        Status: error
1556
        Processing start: 2009-10-25 23:18:02.200335
1557
        Processing end:   2009-10-25 23:18:02.252282
1558
        Input fields:
1559
          name: cluster.example.com
1560
        Result:
1561
          OpPrereqError
1562
          [Neither the name nor the IP address of the cluster has changed]
1563
        Execution log:
1564

    
1565
During the execution of a job, it's possible to follow the output of a
1566
job, similar to the log that one get from the ``gnt-`` commands, via the
1567
watch command::
1568

    
1569
  $ gnt-instance add --submit … %instance1%
1570
  JobID: 17818
1571
  $ gnt-job watch %17818%
1572
  Output from job 17818 follows
1573
  -----------------------------
1574
  Mon Oct 26 00:22:48 2009  - INFO: Selected nodes for instance instance1 via iallocator dumb: node1, node2
1575
  Mon Oct 26 00:22:49 2009 * creating instance disks...
1576
  Mon Oct 26 00:22:52 2009 adding instance instance1 to cluster config
1577
  Mon Oct 26 00:22:52 2009  - INFO: Waiting for instance instance1 to sync disks.
1578
1579
  Mon Oct 26 00:23:03 2009 creating os for instance instance1 on node node1
1580
  Mon Oct 26 00:23:03 2009 * running the instance OS create scripts...
1581
  Mon Oct 26 00:23:13 2009 * starting instance...
1582
  $
1583

    
1584
This is useful if you need to follow a job's progress from multiple
1585
terminals.
1586

    
1587
A job that has not yet started to run can be canceled::
1588

    
1589
  $ gnt-job cancel %17810%
1590

    
1591
But not one that has already started execution::
1592

    
1593
  $ gnt-job cancel %17805%
1594
  Job 17805 is no longer waiting in the queue
1595

    
1596
There are two queues for jobs: the *current* and the *archive*
1597
queue. Jobs are initially submitted to the current queue, and they stay
1598
in that queue until they have finished execution (either successfully or
1599
not). At that point, they can be moved into the archive queue using e.g.
1600
``gnt-job autoarchive all``. The ``ganeti-watcher`` script will do this
1601
automatically 6 hours after a job is finished. The ``ganeti-cleaner``
1602
script will then remove archived the jobs from the archive directory
1603
after three weeks.
1604

    
1605
Note that ``gnt-job list`` only shows jobs in the current queue.
1606
Archived jobs can be viewed using ``gnt-job info <id>``.
1607

    
1608
Special Ganeti deployments
1609
--------------------------
1610

    
1611
Since Ganeti 2.4, it is possible to extend the Ganeti deployment with
1612
two custom scenarios: Ganeti inside Ganeti and multi-site model.
1613

    
1614
Running Ganeti under Ganeti
1615
+++++++++++++++++++++++++++
1616

    
1617
It is sometimes useful to be able to use a Ganeti instance as a Ganeti
1618
node (part of another cluster, usually). One example scenario is two
1619
small clusters, where we want to have an additional master candidate
1620
that holds the cluster configuration and can be used for helping with
1621
the master voting process.
1622

    
1623
However, these Ganeti instance should not host instances themselves, and
1624
should not be considered in the normal capacity planning, evacuation
1625
strategies, etc. In order to accomplish this, mark these nodes as
1626
non-``vm_capable``::
1627

    
1628
  $ gnt-node modify --vm-capable=no %node3%
1629

    
1630
The vm_capable status can be listed as usual via ``gnt-node list``::
1631

    
1632
  $ gnt-node list -oname,vm_capable
1633
  Node  VMCapable
1634
  node1 Y
1635
  node2 Y
1636
  node3 N
1637

    
1638
When this flag is set, the cluster will not do any operations that
1639
relate to instances on such nodes, e.g. hypervisor operations,
1640
disk-related operations, etc. Basically they will just keep the ssconf
1641
files, and if master candidates the full configuration.
1642

    
1643
Multi-site model
1644
++++++++++++++++
1645

    
1646
If Ganeti is deployed in multi-site model, with each site being a node
1647
group (so that instances are not relocated across the WAN by mistake),
1648
it is conceivable that either the WAN latency is high or that some sites
1649
have a lower reliability than others. In this case, it doesn't make
1650
sense to replicate the job information across all sites (or even outside
1651
of a “central” node group), so it should be possible to restrict which
1652
nodes can become master candidates via the auto-promotion algorithm.
1653

    
1654
Ganeti 2.4 introduces for this purpose a new ``master_capable`` flag,
1655
which (when unset) prevents nodes from being marked as master
1656
candidates, either manually or automatically.
1657

    
1658
As usual, the node modify operation can change this flag::
1659

    
1660
  $ gnt-node modify --auto-promote --master-capable=no %node3%
1661
  Fri Jan  7 06:23:07 2011  - INFO: Demoting from master candidate
1662
  Fri Jan  7 06:23:08 2011  - INFO: Promoted nodes to master candidate role: node4
1663
  Modified node node3
1664
   - master_capable -> False
1665
   - master_candidate -> False
1666

    
1667
And the node list operation will list this flag::
1668

    
1669
  $ gnt-node list -oname,master_capable %node1% %node2% %node3%
1670
  Node  MasterCapable
1671
  node1 Y
1672
  node2 Y
1673
  node3 N
1674

    
1675
Note that marking a node both not ``vm_capable`` and not
1676
``master_capable`` makes the node practically unusable from Ganeti's
1677
point of view. Hence these two flags should be used probably in
1678
contrast: some nodes will be only master candidates (master_capable but
1679
not vm_capable), and other nodes will only hold instances (vm_capable
1680
but not master_capable).
1681

    
1682

    
1683
Ganeti tools
1684
------------
1685

    
1686
Beside the usual ``gnt-`` and ``ganeti-`` commands which are provided
1687
and installed in ``$prefix/sbin`` at install time, there are a couple of
1688
other tools installed which are used seldom but can be helpful in some
1689
cases.
1690

    
1691
lvmstrap
1692
++++++++
1693

    
1694
The ``lvmstrap`` tool, introduced in :ref:`configure-lvm-label` section,
1695
has two modes of operation:
1696

    
1697
- ``diskinfo`` shows the discovered disks on the system and their status
1698
- ``create`` takes all not-in-use disks and creates a volume group out
1699
  of them
1700

    
1701
.. warning:: The ``create`` argument to this command causes data-loss!
1702

    
1703
cfgupgrade
1704
++++++++++
1705

    
1706
The ``cfgupgrade`` tools is used to upgrade between major (and minor)
1707
Ganeti versions, and to roll back. Point-releases are usually
1708
transparent for the admin.
1709

    
1710
More information about the upgrade procedure is listed on the wiki at
1711
http://code.google.com/p/ganeti/wiki/UpgradeNotes.
1712

    
1713
There is also a script designed to upgrade from Ganeti 1.2 to 2.0,
1714
called ``cfgupgrade12``.
1715

    
1716
cfgshell
1717
++++++++
1718

    
1719
.. note:: This command is not actively maintained; make sure you backup
1720
   your configuration before using it
1721

    
1722
This can be used as an alternative to direct editing of the
1723
main configuration file if Ganeti has a bug and prevents you, for
1724
example, from removing an instance or a node from the configuration
1725
file.
1726

    
1727
.. _burnin-label:
1728

    
1729
burnin
1730
++++++
1731

    
1732
.. warning:: This command will erase existing instances if given as
1733
   arguments!
1734

    
1735
This tool is used to exercise either the hardware of machines or
1736
alternatively the Ganeti software. It is safe to run on an existing
1737
cluster **as long as you don't pass it existing instance names**.
1738

    
1739
The command will, by default, execute a comprehensive set of operations
1740
against a list of instances, these being:
1741

    
1742
- creation
1743
- disk replacement (for redundant instances)
1744
- failover and migration (for redundant instances)
1745
- move (for non-redundant instances)
1746
- disk growth
1747
- add disks, remove disk
1748
- add NICs, remove NICs
1749
- export and then import
1750
- rename
1751
- reboot
1752
- shutdown/startup
1753
- and finally removal of the test instances
1754

    
1755
Executing all these operations will test that the hardware performs
1756
well: the creation, disk replace, disk add and disk growth will exercise
1757
the storage and network; the migrate command will test the memory of the
1758
systems. Depending on the passed options, it can also test that the
1759
instance OS definitions are executing properly the rename, import and
1760
export operations.
1761

    
1762
sanitize-config
1763
+++++++++++++++
1764

    
1765
This tool takes the Ganeti configuration and outputs a "sanitized"
1766
version, by randomizing or clearing:
1767

    
1768
- DRBD secrets and cluster public key (always)
1769
- host names (optional)
1770
- IPs (optional)
1771
- OS names (optional)
1772
- LV names (optional, only useful for very old clusters which still have
1773
  instances whose LVs are based on the instance name)
1774

    
1775
By default, all optional items are activated except the LV name
1776
randomization. When passing ``--no-randomization``, which disables the
1777
optional items (i.e. just the DRBD secrets and cluster public keys are
1778
randomized), the resulting file can be used as a safety copy of the
1779
cluster config - while not trivial, the layout of the cluster can be
1780
recreated from it and if the instance disks have not been lost it
1781
permits recovery from the loss of all master candidates.
1782

    
1783
move-instance
1784
+++++++++++++
1785

    
1786
See :doc:`separate documentation for move-instance <move-instance>`.
1787

    
1788
users-setup
1789
+++++++++++
1790

    
1791
Ganeti can either be run entirely as root, or with every daemon running as
1792
its own specific user (if the parameters ``--with-user-prefix`` and/or
1793
``--with-group-prefix`` have been specified at ``./configure``-time).
1794

    
1795
In case split users are activated, they are required to exist on the system,
1796
and they need to belong to the proper groups in order for the access
1797
permissions to files and programs to be correct.
1798

    
1799
The ``users-setup`` tool, when run, takes care of setting up the proper
1800
users and groups.
1801

    
1802
When invoked without parameters, the tool runs in interactive mode, showing the
1803
list of actions it will perform and asking for confirmation before proceeding.
1804

    
1805
Providing the ``--yes-do-it`` parameter to the tool prevents the confirmation
1806
from being asked, and the users and groups will be created immediately.
1807

    
1808
.. TODO: document cluster-merge tool
1809

    
1810

    
1811
Other Ganeti projects
1812
---------------------
1813

    
1814
Below is a list (which might not be up-to-date) of additional projects
1815
that can be useful in a Ganeti deployment. They can be downloaded from
1816
the project site (http://code.google.com/p/ganeti/) and the repositories
1817
are also on the project git site (http://git.ganeti.org).
1818

    
1819
NBMA tools
1820
++++++++++
1821

    
1822
The ``ganeti-nbma`` software is designed to allow instances to live on a
1823
separate, virtual network from the nodes, and in an environment where
1824
nodes are not guaranteed to be able to reach each other via multicasting
1825
or broadcasting. For more information see the README in the source
1826
archive.
1827

    
1828
ganeti-htools
1829
+++++++++++++
1830

    
1831
Before Ganeti version 2.5, this was a standalone project; since that
1832
version it is integrated into the Ganeti codebase (see
1833
:doc:`install-quick` for instructions on how to enable it). If you run
1834
an older Ganeti version, you will have to download and build it
1835
separately.
1836

    
1837
For more information and installation instructions, see the README file
1838
in the source archive.
1839

    
1840
.. vim: set textwidth=72 :
1841
.. Local Variables:
1842
.. mode: rst
1843
.. fill-column: 72
1844
.. End: