Statistics
| Branch: | Tag: | Revision:

root / doc / admin.rst @ fc3914e6

History | View | Annotate | Download (54.8 kB)

1
Ganeti administrator's guide
2
============================
3

    
4
Documents Ganeti version |version|
5

    
6
.. contents::
7

    
8
.. highlight:: text
9

    
10
Introduction
11
------------
12

    
13
Ganeti is a virtualization cluster management software. You are expected
14
to be a system administrator familiar with your Linux distribution and
15
the Xen or KVM virtualization environments before using it.
16

    
17
The various components of Ganeti all have man pages and interactive
18
help. This manual though will help you getting familiar with the system
19
by explaining the most common operations, grouped by related use.
20

    
21
After a terminology glossary and a section on the prerequisites needed
22
to use this manual, the rest of this document is divided in sections
23
for the different targets that a command affects: instance, nodes, etc.
24

    
25
.. _terminology-label:
26

    
27
Ganeti terminology
28
++++++++++++++++++
29

    
30
This section provides a small introduction to Ganeti terminology, which
31
might be useful when reading the rest of the document.
32

    
33
Cluster
34
~~~~~~~
35

    
36
A set of machines (nodes) that cooperate to offer a coherent, highly
37
available virtualization service under a single administration domain.
38

    
39
Node
40
~~~~
41

    
42
A physical machine which is member of a cluster.  Nodes are the basic
43
cluster infrastructure, and they don't need to be fault tolerant in
44
order to achieve high availability for instances.
45

    
46
Node can be added and removed (if they host no instances) at will from
47
the cluster. In a HA cluster and only with HA instances, the loss of any
48
single node will not cause disk data loss for any instance; of course,
49
a node crash will cause the crash of the its primary instances.
50

    
51
A node belonging to a cluster can be in one of the following roles at a
52
given time:
53

    
54
- *master* node, which is the node from which the cluster is controlled
55
- *master candidate* node, only nodes in this role have the full cluster
56
  configuration and knowledge, and only master candidates can become the
57
  master node
58
- *regular* node, which is the state in which most nodes will be on
59
  bigger clusters (>20 nodes)
60
- *drained* node, nodes in this state are functioning normally but the
61
  cannot receive new instances; the intention is that nodes in this role
62
  have some issue and they are being evacuated for hardware repairs
63
- *offline* node, in which there is a record in the cluster
64
  configuration about the node, but the daemons on the master node will
65
  not talk to this node; any instances declared as having an offline
66
  node as either primary or secondary will be flagged as an error in the
67
  cluster verify operation
68

    
69
Depending on the role, each node will run a set of daemons:
70

    
71
- the :command:`ganeti-noded` daemon, which control the manipulation of
72
  this node's hardware resources; it runs on all nodes which are in a
73
  cluster
74
- the :command:`ganeti-confd` daemon (Ganeti 2.1+) which runs on all
75
  nodes, but is only functional on master candidate nodes
76
- the :command:`ganeti-rapi` daemon which runs on the master node and
77
  offers an HTTP-based API for the cluster
78
- the :command:`ganeti-masterd` daemon which runs on the master node and
79
  allows control of the cluster
80

    
81
Beside the node role, there are other node flags that influence its
82
behaviour:
83

    
84
- the *master_capable* flag denotes whether the node can ever become a
85
  master candidate; setting this to 'no' means that auto-promotion will
86
  never make this node a master candidate; this flag can be useful for a
87
  remote node that only runs local instances, and having it become a
88
  master is impractical due to networking or other constraints
89
- the *vm_capable* flag denotes whether the node can host instances or
90
  not; for example, one might use a non-vm_capable node just as a master
91
  candidate, for configuration backups; setting this flag to no
92
  disallows placement of instances of this node, deactivates hypervisor
93
  and related checks on it (e.g. bridge checks, LVM check, etc.), and
94
  removes it from cluster capacity computations
95

    
96

    
97
Instance
98
~~~~~~~~
99

    
100
A virtual machine which runs on a cluster. It can be a fault tolerant,
101
highly available entity.
102

    
103
An instance has various parameters, which are classified in three
104
categories: hypervisor related-parameters (called ``hvparams``), general
105
parameters (called ``beparams``) and per network-card parameters (called
106
``nicparams``). All these parameters can be modified either at instance
107
level or via defaults at cluster level.
108

    
109
Disk template
110
~~~~~~~~~~~~~
111

    
112
The are multiple options for the storage provided to an instance; while
113
the instance sees the same virtual drive in all cases, the node-level
114
configuration varies between them.
115

    
116
There are four disk templates you can choose from:
117

    
118
diskless
119
  The instance has no disks. Only used for special purpose operating
120
  systems or for testing.
121

    
122
file
123
  The instance will use plain files as backend for its disks. No
124
  redundancy is provided, and this is somewhat more difficult to
125
  configure for high performance.
126

    
127
plain
128
  The instance will use LVM devices as backend for its disks. No
129
  redundancy is provided.
130

    
131
drbd
132
  .. note:: This is only valid for multi-node clusters using DRBD 8.0+
133

    
134
  A mirror is set between the local node and a remote one, which must be
135
  specified with the second value of the --node option. Use this option
136
  to obtain a highly available instance that can be failed over to a
137
  remote node should the primary one fail.
138

    
139
IAllocator
140
~~~~~~~~~~
141

    
142
A framework for using external (user-provided) scripts to compute the
143
placement of instances on the cluster nodes. This eliminates the need to
144
manually specify nodes in instance add, instance moves, node evacuate,
145
etc.
146

    
147
In order for Ganeti to be able to use these scripts, they must be place
148
in the iallocator directory (usually ``lib/ganeti/iallocators`` under
149
the installation prefix, e.g. ``/usr/local``).
150

    
151
“Primary” and “secondary” concepts
152
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
153

    
154
An instance has a primary and depending on the disk configuration, might
155
also have a secondary node. The instance always runs on the primary node
156
and only uses its secondary node for disk replication.
157

    
158
Similarly, the term of primary and secondary instances when talking
159
about a node refers to the set of instances having the given node as
160
primary, respectively secondary.
161

    
162
Tags
163
~~~~
164

    
165
Tags are short strings that can be attached to either to cluster itself,
166
or to nodes or instances. They are useful as a very simplistic
167
information store for helping with cluster administration, for example
168
by attaching owner information to each instance after it's created::
169

    
170
  gnt-instance add … instance1
171
  gnt-instance add-tags instance1 owner:user2
172

    
173
And then by listing each instance and its tags, this information could
174
be used for contacting the users of each instance.
175

    
176
Jobs and OpCodes
177
~~~~~~~~~~~~~~~~
178

    
179
While not directly visible by an end-user, it's useful to know that a
180
basic cluster operation (e.g. starting an instance) is represented
181
internall by Ganeti as an *OpCode* (abbreviation from operation
182
code). These OpCodes are executed as part of a *Job*. The OpCodes in a
183
single Job are processed serially by Ganeti, but different Jobs will be
184
processed (depending on resource availability) in parallel. They will
185
not be executed in the submission order, but depending on resource
186
availability, locks and (starting with Ganeti 2.3) priority. An earlier
187
job may have to wait for a lock while a newer job doesn't need any locks
188
and can be executed right away. Operations requiring a certain order
189
need to be submitted as a single job, or the client must submit one job
190
at a time and wait for it to finish before continuing.
191

    
192
For example, shutting down the entire cluster can be done by running the
193
command ``gnt-instance shutdown --all``, which will submit for each
194
instance a separate job containing the “shutdown instance” OpCode.
195

    
196

    
197
Prerequisites
198
+++++++++++++
199

    
200
You need to have your Ganeti cluster installed and configured before you
201
try any of the commands in this document. Please follow the
202
:doc:`install` for instructions on how to do that.
203

    
204
Instance management
205
-------------------
206

    
207
Adding an instance
208
++++++++++++++++++
209

    
210
The add operation might seem complex due to the many parameters it
211
accepts, but once you have understood the (few) required parameters and
212
the customisation capabilities you will see it is an easy operation.
213

    
214
The add operation requires at minimum five parameters:
215

    
216
- the OS for the instance
217
- the disk template
218
- the disk count and size
219
- the node specification or alternatively the iallocator to use
220
- and finally the instance name
221

    
222
The OS for the instance must be visible in the output of the command
223
``gnt-os list`` and specifies which guest OS to install on the instance.
224

    
225
The disk template specifies what kind of storage to use as backend for
226
the (virtual) disks presented to the instance; note that for instances
227
with multiple virtual disks, they all must be of the same type.
228

    
229
The node(s) on which the instance will run can be given either manually,
230
via the ``-n`` option, or computed automatically by Ganeti, if you have
231
installed any iallocator script.
232

    
233
With the above parameters in mind, the command is::
234

    
235
  gnt-instance add \
236
    -n TARGET_NODE:SECONDARY_NODE \
237
    -o OS_TYPE \
238
    -t DISK_TEMPLATE -s DISK_SIZE \
239
    INSTANCE_NAME
240

    
241
The instance name must be resolvable (e.g. exist in DNS) and usually
242
points to an address in the same subnet as the cluster itself.
243

    
244
The above command has the minimum required options; other options you
245
can give include, among others:
246

    
247
- The memory size (``-B memory``)
248

    
249
- The number of virtual CPUs (``-B vcpus``)
250

    
251
- Arguments for the NICs of the instance; by default, a single-NIC
252
  instance is created. The IP and/or bridge of the NIC can be changed
253
  via ``--nic 0:ip=IP,bridge=BRIDGE``
254

    
255
See the manpage for gnt-instance for the detailed option list.
256

    
257
For example if you want to create an highly available instance, with a
258
single disk of 50GB and the default memory size, having primary node
259
``node1`` and secondary node ``node3``, use the following command::
260

    
261
  gnt-instance add -n node1:node3 -o debootstrap -t drbd \
262
    instance1
263

    
264
There is a also a command for batch instance creation from a
265
specification file, see the ``batch-create`` operation in the
266
gnt-instance manual page.
267

    
268
Regular instance operations
269
+++++++++++++++++++++++++++
270

    
271
Removal
272
~~~~~~~
273

    
274
Removing an instance is even easier than creating one. This operation is
275
irreversible and destroys all the contents of your instance. Use with
276
care::
277

    
278
  gnt-instance remove INSTANCE_NAME
279

    
280
Startup/shutdown
281
~~~~~~~~~~~~~~~~
282

    
283
Instances are automatically started at instance creation time. To
284
manually start one which is currently stopped you can run::
285

    
286
  gnt-instance startup INSTANCE_NAME
287

    
288
While the command to stop one is::
289

    
290
  gnt-instance shutdown INSTANCE_NAME
291

    
292
.. warning:: Do not use the Xen or KVM commands directly to stop
293
   instances. If you run for example ``xm shutdown`` or ``xm destroy``
294
   on an instance Ganeti will automatically restart it (via the
295
   :command:`ganeti-watcher` command which is launched via cron).
296

    
297
Querying instances
298
~~~~~~~~~~~~~~~~~~
299

    
300
There are two ways to get information about instances: listing
301
instances, which does a tabular output containing a given set of fields
302
about each instance, and querying detailed information about a set of
303
instances.
304

    
305
The command to see all the instances configured and their status is::
306

    
307
  gnt-instance list
308

    
309
The command can return a custom set of information when using the ``-o``
310
option (as always, check the manpage for a detailed specification). Each
311
instance will be represented on a line, thus making it easy to parse
312
this output via the usual shell utilities (grep, sed, etc.).
313

    
314
To get more detailed information about an instance, you can run::
315

    
316
  gnt-instance info INSTANCE
317

    
318
which will give a multi-line block of information about the instance,
319
it's hardware resources (especially its disks and their redundancy
320
status), etc. This is harder to parse and is more expensive than the
321
list operation, but returns much more detailed information.
322

    
323

    
324
Export/Import
325
+++++++++++++
326

    
327
You can create a snapshot of an instance disk and its Ganeti
328
configuration, which then you can backup, or import into another
329
cluster. The way to export an instance is::
330

    
331
  gnt-backup export -n TARGET_NODE INSTANCE_NAME
332

    
333

    
334
The target node can be any node in the cluster with enough space under
335
``/srv/ganeti`` to hold the instance image. Use the ``--noshutdown``
336
option to snapshot an instance without rebooting it. Note that Ganeti
337
only keeps one snapshot for an instance - any previous snapshot of the
338
same instance existing cluster-wide under ``/srv/ganeti`` will be
339
removed by this operation: if you want to keep them, you need to move
340
them out of the Ganeti exports directory.
341

    
342
Importing an instance is similar to creating a new one, but additionally
343
one must specify the location of the snapshot. The command is::
344

    
345
  gnt-backup import -n TARGET_NODE \
346
    --src-node=NODE --src-dir=DIR INSTANCE_NAME
347

    
348
By default, parameters will be read from the export information, but you
349
can of course pass them in via the command line - most of the options
350
available for the command :command:`gnt-instance add` are supported here
351
too.
352

    
353
Import of foreign instances
354
+++++++++++++++++++++++++++
355

    
356
There is a possibility to import a foreign instance whose disk data is
357
already stored as LVM volumes without going through copying it: the disk
358
adoption mode.
359

    
360
For this, ensure that the original, non-managed instance is stopped,
361
then create a Ganeti instance in the usual way, except that instead of
362
passing the disk information you specify the current volumes::
363

    
364
  gnt-instance add -t plain -n HOME_NODE ... \
365
    --disk 0:adopt=lv_name[,vg=vg_name] INSTANCE_NAME
366

    
367
This will take over the given logical volumes, rename them to the Ganeti
368
standard (UUID-based), and without installing the OS on them start
369
directly the instance. If you configure the hypervisor similar to the
370
non-managed configuration that the instance had, the transition should
371
be seamless for the instance. For more than one disk, just pass another
372
disk parameter (e.g. ``--disk 1:adopt=...``).
373

    
374
Instance kernel selection
375
+++++++++++++++++++++++++
376

    
377
The kernel that instances uses to bootup can come either from the node,
378
or from instances themselves, depending on the setup.
379

    
380
Xen-PVM
381
~~~~~~~
382

    
383
With Xen PVM, there are three options.
384

    
385
First, you can use a kernel from the node, by setting the hypervisor
386
parameters as such:
387

    
388
- ``kernel_path`` to a valid file on the node (and appropriately
389
  ``initrd_path``)
390
- ``kernel_args`` optionally set to a valid Linux setting (e.g. ``ro``)
391
- ``root_path`` to a valid setting (e.g. ``/dev/xvda1``)
392
- ``bootloader_path`` and ``bootloader_args`` to empty
393

    
394
Alternatively, you can delete the kernel management to instances, and
395
use either ``pvgrub`` or the deprecated ``pygrub``. For this, you must
396
install the kernels and initrds in the instance, and create a valid grub
397
v1 configuration file.
398

    
399
For ``pvgrub`` (new in version 2.4.2), you need to set:
400

    
401
- ``kernel_path`` to point to the ``pvgrub`` loader present on the node
402
  (e.g. ``/usr/lib/xen/boot/pv-grub-x86_32.gz``)
403
- ``kernel_args`` to the path to the grub config file, relative to the
404
  instance (e.g. ``(hd0,0)/grub/menu.lst``)
405
- ``root_path`` **must** be empty
406
- ``bootloader_path`` and ``bootloader_args`` to empty
407

    
408
While ``pygrub`` is deprecated, here is how you can configure it:
409

    
410
- ``bootloader_path`` to the pygrub binary (e.g. ``/usr/bin/pygrub``)
411
- the other settings are not important
412

    
413
More information can be found in the Xen wiki pages for `pvgrub
414
<http://wiki.xensource.com/xenwiki/PvGrub>`_ and `pygrub
415
<http://wiki.xensource.com/xenwiki/PyGrub>`_.
416

    
417
KVM
418
~~~
419

    
420
For KVM also the kernel can be loaded either way.
421

    
422
For loading the kernels from the node, you need to set:
423

    
424
- ``kernel_path`` to a valid value
425
- ``initrd_path`` optionally set if you use an initrd
426
- ``kernel_args`` optionally set to a valid value (e.g. ``ro``)
427

    
428
If you want instead to have the instance boot from its disk (and execute
429
its bootloader), simply set the ``kernel_path`` parameter to an empty
430
string, and all the others will be ignored.
431

    
432
Instance HA features
433
--------------------
434

    
435
.. note:: This section only applies to multi-node clusters
436

    
437
.. _instance-change-primary-label:
438

    
439
Changing the primary node
440
+++++++++++++++++++++++++
441

    
442
There are three ways to exchange an instance's primary and secondary
443
nodes; the right one to choose depends on how the instance has been
444
created and the status of its current primary node. See
445
:ref:`rest-redundancy-label` for information on changing the secondary
446
node. Note that it's only possible to change the primary node to the
447
secondary and vice-versa; a direct change of the primary node with a
448
third node, while keeping the current secondary is not possible in a
449
single step, only via multiple operations as detailed in
450
:ref:`instance-relocation-label`.
451

    
452
Failing over an instance
453
~~~~~~~~~~~~~~~~~~~~~~~~
454

    
455
If an instance is built in highly available mode you can at any time
456
fail it over to its secondary node, even if the primary has somehow
457
failed and it's not up anymore. Doing it is really easy, on the master
458
node you can just run::
459

    
460
  gnt-instance failover INSTANCE_NAME
461

    
462
That's it. After the command completes the secondary node is now the
463
primary, and vice-versa.
464

    
465
Live migrating an instance
466
~~~~~~~~~~~~~~~~~~~~~~~~~~
467

    
468
If an instance is built in highly available mode, it currently runs and
469
both its nodes are running fine, you can at migrate it over to its
470
secondary node, without downtime. On the master node you need to run::
471

    
472
  gnt-instance migrate INSTANCE_NAME
473

    
474
The current load on the instance and its memory size will influence how
475
long the migration will take. In any case, for both KVM and Xen
476
hypervisors, the migration will be transparent to the instance.
477

    
478
Moving an instance (offline)
479
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
480

    
481
If an instance has not been create as mirrored, then the only way to
482
change its primary node is to execute the move command::
483

    
484
  gnt-instance move -n NEW_NODE INSTANCE
485

    
486
This has a few prerequisites:
487

    
488
- the instance must be stopped
489
- its current primary node must be on-line and healthy
490
- the disks of the instance must not have any errors
491

    
492
Since this operation actually copies the data from the old node to the
493
new node, expect it to take proportional to the size of the instance's
494
disks and the speed of both the nodes' I/O system and their networking.
495

    
496
Disk operations
497
+++++++++++++++
498

    
499
Disk failures are a common cause of errors in any server
500
deployment. Ganeti offers protection from single-node failure if your
501
instances were created in HA mode, and it also offers ways to restore
502
redundancy after a failure.
503

    
504
Preparing for disk operations
505
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
506

    
507
It is important to note that for Ganeti to be able to do any disk
508
operation, the Linux machines on top of which Ganeti must be consistent;
509
for LVM, this means that the LVM commands must not return failures; it
510
is common that after a complete disk failure, any LVM command aborts
511
with an error similar to::
512

    
513
  # vgs
514
  /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
515
  /dev/sdb1: read failed after 0 of 4096 at 750153695232: Input/output
516
  error
517
  /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
518
  Couldn't find device with uuid
519
  't30jmN-4Rcf-Fr5e-CURS-pawt-z0jU-m1TgeJ'.
520
  Couldn't find all physical volumes for volume group xenvg.
521

    
522
Before restoring an instance's disks to healthy status, it's needed to
523
fix the volume group used by Ganeti so that we can actually create and
524
manage the logical volumes. This is usually done in a multi-step
525
process:
526

    
527
#. first, if the disk is completely gone and LVM commands exit with
528
   “Couldn't find device with uuid…” then you need to run the command::
529

    
530
    vgreduce --removemissing VOLUME_GROUP
531

    
532
#. after the above command, the LVM commands should be executing
533
   normally (warnings are normal, but the commands will not fail
534
   completely).
535

    
536
#. if the failed disk is still visible in the output of the ``pvs``
537
   command, you need to deactivate it from allocations by running::
538

    
539
    pvs -x n /dev/DISK
540

    
541
At this point, the volume group should be consistent and any bad
542
physical volumes should not longer be available for allocation.
543

    
544
Note that since version 2.1 Ganeti provides some commands to automate
545
these two operations, see :ref:`storage-units-label`.
546

    
547
.. _rest-redundancy-label:
548

    
549
Restoring redundancy for DRBD-based instances
550
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
551

    
552
A DRBD instance has two nodes, and the storage on one of them has
553
failed. Depending on which node (primary or secondary) has failed, you
554
have three options at hand:
555

    
556
- if the storage on the primary node has failed, you need to re-create
557
  the disks on it
558
- if the storage on the secondary node has failed, you can either
559
  re-create the disks on it or change the secondary and recreate
560
  redundancy on the new secondary node
561

    
562
Of course, at any point it's possible to force re-creation of disks even
563
though everything is already fine.
564

    
565
For all three cases, the ``replace-disks`` operation can be used::
566

    
567
  # re-create disks on the primary node
568
  gnt-instance replace-disks -p INSTANCE_NAME
569
  # re-create disks on the current secondary
570
  gnt-instance replace-disks -s INSTANCE_NAME
571
  # change the secondary node, via manual specification
572
  gnt-instance replace-disks -n NODE INSTANCE_NAME
573
  # change the secondary node, via an iallocator script
574
  gnt-instance replace-disks -I SCRIPT INSTANCE_NAME
575
  # since Ganeti 2.1: automatically fix the primary or secondary node
576
  gnt-instance replace-disks -a INSTANCE_NAME
577

    
578
Since the process involves copying all data from the working node to the
579
target node, it will take a while, depending on the instance's disk
580
size, node I/O system and network speed. But it is (baring any network
581
interruption) completely transparent for the instance.
582

    
583
Re-creating disks for non-redundant instances
584
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
585

    
586
.. versionadded:: 2.1
587

    
588
For non-redundant instances, there isn't a copy (except backups) to
589
re-create the disks. But it's possible to at-least re-create empty
590
disks, after which a reinstall can be run, via the ``recreate-disks``
591
command::
592

    
593
  gnt-instance recreate-disks INSTANCE
594

    
595
Note that this will fail if the disks already exists.
596

    
597
Conversion of an instance's disk type
598
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
599

    
600
It is possible to convert between a non-redundant instance of type
601
``plain`` (LVM storage) and redundant ``drbd`` via the ``gnt-instance
602
modify`` command::
603

    
604
  # start with a non-redundant instance
605
  gnt-instance add -t plain ... INSTANCE
606

    
607
  # later convert it to redundant
608
  gnt-instance stop INSTANCE
609
  gnt-instance modify -t drbd -n NEW_SECONDARY INSTANCE
610
  gnt-instance start INSTANCE
611

    
612
  # and convert it back
613
  gnt-instance stop INSTANCE
614
  gnt-instance modify -t plain INSTANCE
615
  gnt-instance start INSTANCE
616

    
617
The conversion must be done while the instance is stopped, and
618
converting from plain to drbd template presents a small risk, especially
619
if the instance has multiple disks and/or if one node fails during the
620
conversion procedure). As such, it's recommended (as always) to make
621
sure that downtime for manual recovery is acceptable and that the
622
instance has up-to-date backups.
623

    
624
Debugging instances
625
+++++++++++++++++++
626

    
627
Accessing an instance's disks
628
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
629

    
630
From an instance's primary node you can have access to its disks. Never
631
ever mount the underlying logical volume manually on a fault tolerant
632
instance, or will break replication and your data will be
633
inconsistent. The correct way to access an instance's disks is to run
634
(on the master node, as usual) the command::
635

    
636
  gnt-instance activate-disks INSTANCE
637

    
638
And then, *on the primary node of the instance*, access the device that
639
gets created. For example, you could mount the given disks, then edit
640
files on the filesystem, etc.
641

    
642
Note that with partitioned disks (as opposed to whole-disk filesystems),
643
you will need to use a tool like :manpage:`kpartx(8)`::
644

    
645
  node1# gnt-instance activate-disks instance1
646
647
  node1# ssh node3
648
  node3# kpartx -l /dev/…
649
  node3# kpartx -a /dev/…
650
  node3# mount /dev/mapper/… /mnt/
651
  # edit files under mnt as desired
652
  node3# umount /mnt/
653
  node3# kpartx -d /dev/…
654
  node3# exit
655
  node1#
656

    
657
After you've finished you can deactivate them with the deactivate-disks
658
command, which works in the same way::
659

    
660
  gnt-instance deactivate-disks INSTANCE
661

    
662
Note that if any process started by you is still using the disks, the
663
above command will error out, and you **must** cleanup and ensure that
664
the above command runs successfully before you start the instance,
665
otherwise the instance will suffer corruption.
666

    
667
Accessing an instance's console
668
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
669

    
670
The command to access a running instance's console is::
671

    
672
  gnt-instance console INSTANCE_NAME
673

    
674
Use the console normally and then type ``^]`` when done, to exit.
675

    
676
Other instance operations
677
+++++++++++++++++++++++++
678

    
679
Reboot
680
~~~~~~
681

    
682
There is a wrapper command for rebooting instances::
683

    
684
  gnt-instance reboot instance2
685

    
686
By default, this does the equivalent of shutting down and then starting
687
the instance, but it accepts parameters to perform a soft-reboot (via
688
the hypervisor), a hard reboot (hypervisor shutdown and then startup) or
689
a full one (the default, which also de-configures and then configures
690
again the disks of the instance).
691

    
692
Instance OS definitions debugging
693
+++++++++++++++++++++++++++++++++
694

    
695
Should you have any problems with instance operating systems the command
696
to see a complete status for all your nodes is::
697

    
698
   gnt-os diagnose
699

    
700
.. _instance-relocation-label:
701

    
702
Instance relocation
703
~~~~~~~~~~~~~~~~~~~
704

    
705
While it is not possible to move an instance from nodes ``(A, B)`` to
706
nodes ``(C, D)`` in a single move, it is possible to do so in a few
707
steps::
708

    
709
  # instance is located on A, B
710
  node1# gnt-instance replace -n nodeC instance1
711
  # instance has moved from (A, B) to (A, C)
712
  # we now flip the primary/secondary nodes
713
  node1# gnt-instance migrate instance1
714
  # instance lives on (C, A)
715
  # we can then change A to D via:
716
  node1# gnt-instance replace -n nodeD instance1
717

    
718
Which brings it into the final configuration of ``(C, D)``. Note that we
719
needed to do two replace-disks operation (two copies of the instance
720
disks), because we needed to get rid of both the original nodes (A and
721
B).
722

    
723
Node operations
724
---------------
725

    
726
There are much fewer node operations available than for instances, but
727
they are equivalently important for maintaining a healthy cluster.
728

    
729
Add/readd
730
+++++++++
731

    
732
It is at any time possible to extend the cluster with one more node, by
733
using the node add operation::
734

    
735
  gnt-node add NEW_NODE
736

    
737
If the cluster has a replication network defined, then you need to pass
738
the ``-s REPLICATION_IP`` parameter to this option.
739

    
740
A variation of this command can be used to re-configure a node if its
741
Ganeti configuration is broken, for example if it has been reinstalled
742
by mistake::
743

    
744
  gnt-node add --readd EXISTING_NODE
745

    
746
This will reinitialise the node as if it's been newly added, but while
747
keeping its existing configuration in the cluster (primary/secondary IP,
748
etc.), in other words you won't need to use ``-s`` here.
749

    
750
Changing the node role
751
++++++++++++++++++++++
752

    
753
A node can be in different roles, as explained in the
754
:ref:`terminology-label` section. Promoting a node to the master role is
755
special, while the other roles are handled all via a single command.
756

    
757
Failing over the master node
758
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
759

    
760
If you want to promote a different node to the master role (for whatever
761
reason), run on any other master-candidate node the command::
762

    
763
  gnt-cluster master-failover
764

    
765
and the node you ran it on is now the new master. In case you try to run
766
this on a non master-candidate node, you will get an error telling you
767
which nodes are valid.
768

    
769
Changing between the other roles
770
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
771

    
772
The ``gnt-node modify`` command can be used to select a new role::
773

    
774
  # change to master candidate
775
  gnt-node modify -C yes NODE
776
  # change to drained status
777
  gnt-node modify -D yes NODE
778
  # change to offline status
779
  gnt-node modify -O yes NODE
780
  # change to regular mode (reset all flags)
781
  gnt-node modify -O no -D no -C no NODE
782

    
783
Note that the cluster requires that at any point in time, a certain
784
number of nodes are master candidates, so changing from master candidate
785
to other roles might fail. It is recommended to either force the
786
operation (via the ``--force`` option) or first change the number of
787
master candidates in the cluster - see :ref:`cluster-config-label`.
788

    
789
Evacuating nodes
790
++++++++++++++++
791

    
792
There are two steps of moving instances off a node:
793

    
794
- moving the primary instances (actually converting them into secondary
795
  instances)
796
- moving the secondary instances (including any instances converted in
797
  the step above)
798

    
799
Primary instance conversion
800
~~~~~~~~~~~~~~~~~~~~~~~~~~~
801

    
802
For this step, you can use either individual instance move
803
commands (as seen in :ref:`instance-change-primary-label`) or the bulk
804
per-node versions; these are::
805

    
806
  gnt-node migrate NODE
807
  gnt-node evacuate NODE
808

    
809
Note that the instance “move” command doesn't currently have a node
810
equivalent.
811

    
812
Both these commands, or the equivalent per-instance command, will make
813
this node the secondary node for the respective instances, whereas their
814
current secondary node will become primary. Note that it is not possible
815
to change in one step the primary node to another node as primary, while
816
keeping the same secondary node.
817

    
818
Secondary instance evacuation
819
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
820

    
821
For the evacuation of secondary instances, a command called
822
:command:`gnt-node evacuate` is provided and its syntax is::
823

    
824
  gnt-node evacuate -I IALLOCATOR_SCRIPT NODE
825
  gnt-node evacuate -n DESTINATION_NODE NODE
826

    
827
The first version will compute the new secondary for each instance in
828
turn using the given iallocator script, whereas the second one will
829
simply move all instances to DESTINATION_NODE.
830

    
831
Removal
832
+++++++
833

    
834
Once a node no longer has any instances (neither primary nor secondary),
835
it's easy to remove it from the cluster::
836

    
837
  gnt-node remove NODE_NAME
838

    
839
This will deconfigure the node, stop the ganeti daemons on it and leave
840
it hopefully like before it joined to the cluster.
841

    
842
Storage handling
843
++++++++++++++++
844

    
845
When using LVM (either standalone or with DRBD), it can become tedious
846
to debug and fix it in case of errors. Furthermore, even file-based
847
storage can become complicated to handle manually on many hosts. Ganeti
848
provides a couple of commands to help with automation.
849

    
850
Logical volumes
851
~~~~~~~~~~~~~~~
852

    
853
This is a command specific to LVM handling. It allows listing the
854
logical volumes on a given node or on all nodes and their association to
855
instances via the ``volumes`` command::
856

    
857
  node1# gnt-node volumes
858
  Node  PhysDev   VG    Name             Size Instance
859
  node1 /dev/sdb1 xenvg e61fbc97-….disk0 512M instance17
860
  node1 /dev/sdb1 xenvg ebd1a7d1-….disk0 512M instance19
861
  node2 /dev/sdb1 xenvg 0af08a3d-….disk0 512M instance20
862
  node2 /dev/sdb1 xenvg cc012285-….disk0 512M instance16
863
  node2 /dev/sdb1 xenvg f0fac192-….disk0 512M instance18
864

    
865
The above command maps each logical volume to a volume group and
866
underlying physical volume and (possibly) to an instance.
867

    
868
.. _storage-units-label:
869

    
870
Generalized storage handling
871
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
872

    
873
.. versionadded:: 2.1
874

    
875
Starting with Ganeti 2.1, a new storage framework has been implemented
876
that tries to abstract the handling of the storage type the cluster
877
uses.
878

    
879
First is listing the backend storage and their space situation::
880

    
881
  node1# gnt-node list-storage
882
  Node  Name        Size Used   Free
883
  node1 /dev/sda7 673.8G   0M 673.8G
884
  node1 /dev/sdb1 698.6G 1.5G 697.1G
885
  node2 /dev/sda7 673.8G   0M 673.8G
886
  node2 /dev/sdb1 698.6G 1.0G 697.6G
887

    
888
The default is to list LVM physical volumes. It's also possible to list
889
the LVM volume groups::
890

    
891
  node1# gnt-node list-storage -t lvm-vg
892
  Node  Name  Size
893
  node1 xenvg 1.3T
894
  node2 xenvg 1.3T
895

    
896
Next is repairing storage units, which is currently only implemented for
897
volume groups and does the equivalent of ``vgreduce --removemissing``::
898

    
899
  node1# gnt-node repair-storage node2 lvm-vg xenvg
900
  Sun Oct 25 22:21:45 2009 Repairing storage unit 'xenvg' on node2 ...
901

    
902
Last is the modification of volume properties, which is (again) only
903
implemented for LVM physical volumes and allows toggling the
904
``allocatable`` value::
905

    
906
  node1# gnt-node modify-storage --allocatable=no node2 lvm-pv /dev/sdb1
907

    
908
Use of the storage commands
909
~~~~~~~~~~~~~~~~~~~~~~~~~~~
910

    
911
All these commands are needed when recovering a node from a disk
912
failure:
913

    
914
- first, we need to recover from complete LVM failure (due to missing
915
  disk), by running the ``repair-storage`` command
916
- second, we need to change allocation on any partially-broken disk
917
  (i.e. LVM still sees it, but it has bad blocks) by running
918
  ``modify-storage``
919
- then we can evacuate the instances as needed
920

    
921

    
922
Cluster operations
923
------------------
924

    
925
Beside the cluster initialisation command (which is detailed in the
926
:doc:`install` document) and the master failover command which is
927
explained under node handling, there are a couple of other cluster
928
operations available.
929

    
930
.. _cluster-config-label:
931

    
932
Standard operations
933
+++++++++++++++++++
934

    
935
One of the few commands that can be run on any node (not only the
936
master) is the ``getmaster`` command::
937

    
938
  node2# gnt-cluster getmaster
939
  node1.example.com
940
  node2#
941

    
942
It is possible to query and change global cluster parameters via the
943
``info`` and ``modify`` commands::
944

    
945
  node1# gnt-cluster info
946
  Cluster name: cluster.example.com
947
  Cluster UUID: 07805e6f-f0af-4310-95f1-572862ee939c
948
  Creation time: 2009-09-25 05:04:15
949
  Modification time: 2009-10-18 22:11:47
950
  Master node: node1.example.com
951
  Architecture (this node): 64bit (x86_64)
952
953
  Tags: foo
954
  Default hypervisor: xen-pvm
955
  Enabled hypervisors: xen-pvm
956
  Hypervisor parameters:
957
    - xen-pvm:
958
        root_path: /dev/sda1
959
960
  Cluster parameters:
961
    - candidate pool size: 10
962
963
  Default instance parameters:
964
    - default:
965
        memory: 128
966
967
  Default nic parameters:
968
    - default:
969
        link: xen-br0
970
971

    
972
There various parameters above can be changed via the ``modify``
973
commands as follows:
974

    
975
- the hypervisor parameters can be changed via ``modify -H
976
  xen-pvm:root_path=…``, and so on for other hypervisors/key/values
977
- the "default instance parameters" are changeable via ``modify -B
978
  parameter=value…`` syntax
979
- the cluster parameters are changeable via separate options to the
980
  modify command (e.g. ``--candidate-pool-size``, etc.)
981

    
982
For detailed option list see the :manpage:`gnt-cluster(8)` man page.
983

    
984
The cluster version can be obtained via the ``version`` command::
985
  node1# gnt-cluster version
986
  Software version: 2.1.0
987
  Internode protocol: 20
988
  Configuration format: 2010000
989
  OS api version: 15
990
  Export interface: 0
991

    
992
This is not very useful except when debugging Ganeti.
993

    
994
Global node commands
995
++++++++++++++++++++
996

    
997
There are two commands provided for replicating files to all nodes of a
998
cluster and for running commands on all the nodes::
999

    
1000
  node1# gnt-cluster copyfile /path/to/file
1001
  node1# gnt-cluster command ls -l /path/to/file
1002

    
1003
These are simple wrappers over scp/ssh and more advanced usage can be
1004
obtained using :manpage:`dsh(1)` and similar commands. But they are
1005
useful to update an OS script from the master node, for example.
1006

    
1007
Cluster verification
1008
++++++++++++++++++++
1009

    
1010
There are three commands that relate to global cluster checks. The first
1011
one is ``verify`` which gives an overview on the cluster state,
1012
highlighting any issues. In normal operation, this command should return
1013
no ``ERROR`` messages::
1014

    
1015
  node1# gnt-cluster verify
1016
  Sun Oct 25 23:08:58 2009 * Verifying global settings
1017
  Sun Oct 25 23:08:58 2009 * Gathering data (2 nodes)
1018
  Sun Oct 25 23:09:00 2009 * Verifying node status
1019
  Sun Oct 25 23:09:00 2009 * Verifying instance status
1020
  Sun Oct 25 23:09:00 2009 * Verifying orphan volumes
1021
  Sun Oct 25 23:09:00 2009 * Verifying remaining instances
1022
  Sun Oct 25 23:09:00 2009 * Verifying N+1 Memory redundancy
1023
  Sun Oct 25 23:09:00 2009 * Other Notes
1024
  Sun Oct 25 23:09:00 2009   - NOTICE: 5 non-redundant instance(s) found.
1025
  Sun Oct 25 23:09:00 2009 * Hooks Results
1026

    
1027
The second command is ``verify-disks``, which checks that the instance's
1028
disks have the correct status based on the desired instance state
1029
(up/down)::
1030

    
1031
  node1# gnt-cluster verify-disks
1032

    
1033
Note that this command will show no output when disks are healthy.
1034

    
1035
The last command is used to repair any discrepancies in Ganeti's
1036
recorded disk size and the actual disk size (disk size information is
1037
needed for proper activation and growth of DRBD-based disks)::
1038

    
1039
  node1# gnt-cluster repair-disk-sizes
1040
  Sun Oct 25 23:13:16 2009  - INFO: Disk 0 of instance instance1 has mismatched size, correcting: recorded 512, actual 2048
1041
  Sun Oct 25 23:13:17 2009  - WARNING: Invalid result from node node4, ignoring node results
1042

    
1043
The above shows one instance having wrong disk size, and a node which
1044
returned invalid data, and thus we ignored all primary instances of that
1045
node.
1046

    
1047
Configuration redistribution
1048
++++++++++++++++++++++++++++
1049

    
1050
If the verify command complains about file mismatches between the master
1051
and other nodes, due to some node problems or if you manually modified
1052
configuration files, you can force an push of the master configuration
1053
to all other nodes via the ``redist-conf`` command::
1054

    
1055
  node1# gnt-cluster redist-conf
1056
  node1#
1057

    
1058
This command will be silent unless there are problems sending updates to
1059
the other nodes.
1060

    
1061

    
1062
Cluster renaming
1063
++++++++++++++++
1064

    
1065
It is possible to rename a cluster, or to change its IP address, via the
1066
``rename`` command. If only the IP has changed, you need to pass the
1067
current name and Ganeti will realise its IP has changed::
1068

    
1069
  node1# gnt-cluster rename cluster.example.com
1070
  This will rename the cluster to 'cluster.example.com'. If
1071
  you are connected over the network to the cluster name, the operation
1072
  is very dangerous as the IP address will be removed from the node and
1073
  the change may not go through. Continue?
1074
  y/[n]/?: y
1075
  Failure: prerequisites not met for this operation:
1076
  Neither the name nor the IP address of the cluster has changed
1077

    
1078
In the above output, neither value has changed since the cluster
1079
initialisation so the operation is not completed.
1080

    
1081
Queue operations
1082
++++++++++++++++
1083

    
1084
The job queue execution in Ganeti 2.0 and higher can be inspected,
1085
suspended and resumed via the ``queue`` command::
1086

    
1087
  node1~# gnt-cluster queue info
1088
  The drain flag is unset
1089
  node1~# gnt-cluster queue drain
1090
  node1~# gnt-instance stop instance1
1091
  Failed to submit job for instance1: Job queue is drained, refusing job
1092
  node1~# gnt-cluster queue info
1093
  The drain flag is set
1094
  node1~# gnt-cluster queue undrain
1095

    
1096
This is most useful if you have an active cluster and you need to
1097
upgrade the Ganeti software, or simply restart the software on any node:
1098

    
1099
#. suspend the queue via ``queue drain``
1100
#. wait until there are no more running jobs via ``gnt-job list``
1101
#. restart the master or another node, or upgrade the software
1102
#. resume the queue via ``queue undrain``
1103

    
1104
.. note:: this command only stores a local flag file, and if you
1105
   failover the master, it will not have effect on the new master.
1106

    
1107

    
1108
Watcher control
1109
+++++++++++++++
1110

    
1111
The :manpage:`ganeti-watcher` is a program, usually scheduled via
1112
``cron``, that takes care of cluster maintenance operations (restarting
1113
downed instances, activating down DRBD disks, etc.). However, during
1114
maintenance and troubleshooting, this can get in your way; disabling it
1115
via commenting out the cron job is not so good as this can be
1116
forgotten. Thus there are some commands for automated control of the
1117
watcher: ``pause``, ``info`` and ``continue``::
1118

    
1119
  node1~# gnt-cluster watcher info
1120
  The watcher is not paused.
1121
  node1~# gnt-cluster watcher pause 1h
1122
  The watcher is paused until Mon Oct 26 00:30:37 2009.
1123
  node1~# gnt-cluster watcher info
1124
  The watcher is paused until Mon Oct 26 00:30:37 2009.
1125
  node1~# ganeti-watcher -d
1126
  2009-10-25 23:30:47,984:  pid=28867 ganeti-watcher:486 DEBUG Pause has been set, exiting
1127
  node1~# gnt-cluster watcher continue
1128
  The watcher is no longer paused.
1129
  node1~# ganeti-watcher -d
1130
  2009-10-25 23:31:04,789:  pid=28976 ganeti-watcher:345 DEBUG Archived 0 jobs, left 0
1131
  2009-10-25 23:31:05,884:  pid=28976 ganeti-watcher:280 DEBUG Got data from cluster, writing instance status file
1132
  2009-10-25 23:31:06,061:  pid=28976 ganeti-watcher:150 DEBUG Data didn't change, just touching status file
1133
  node1~# gnt-cluster watcher info
1134
  The watcher is not paused.
1135
  node1~#
1136

    
1137
The exact details of the argument to the ``pause`` command are available
1138
in the manpage.
1139

    
1140
.. note:: this command only stores a local flag file, and if you
1141
   failover the master, it will not have effect on the new master.
1142

    
1143
Node auto-maintenance
1144
+++++++++++++++++++++
1145

    
1146
If the cluster parameter ``maintain_node_health`` is enabled (see the
1147
manpage for :command:`gnt-cluster`, the init and modify subcommands),
1148
then the following will happen automatically:
1149

    
1150
- the watcher will shutdown any instances running on offline nodes
1151
- the watcher will deactivate any DRBD devices on offline nodes
1152

    
1153
In the future, more actions are planned, so only enable this parameter
1154
if the nodes are completely dedicated to Ganeti; otherwise it might be
1155
possible to lose data due to auto-maintenance actions.
1156

    
1157
Removing a cluster entirely
1158
+++++++++++++++++++++++++++
1159

    
1160
The usual method to cleanup a cluster is to run ``gnt-cluster destroy``
1161
however if the Ganeti installation is broken in any way then this will
1162
not run.
1163

    
1164
It is possible in such a case to cleanup manually most if not all traces
1165
of a cluster installation by following these steps on all of the nodes:
1166

    
1167
1. Shutdown all instances. This depends on the virtualisation method
1168
   used (Xen, KVM, etc.):
1169

    
1170
  - Xen: run ``xm list`` and ``xm destroy`` on all the non-Domain-0
1171
    instances
1172
  - KVM: kill all the KVM processes
1173
  - chroot: kill all processes under the chroot mountpoints
1174

    
1175
2. If using DRBD, shutdown all DRBD minors (which should by at this time
1176
   no-longer in use by instances); on each node, run ``drbdsetup
1177
   /dev/drbdN down`` for each active DRBD minor.
1178

    
1179
3. If using LVM, cleanup the Ganeti volume group; if only Ganeti created
1180
   logical volumes (and you are not sharing the volume group with the
1181
   OS, for example), then simply running ``lvremove -f xenvg`` (replace
1182
   'xenvg' with your volume group name) should do the required cleanup.
1183

    
1184
4. If using file-based storage, remove recursively all files and
1185
   directories under your file-storage directory: ``rm -rf
1186
   /srv/ganeti/file-storage/*`` replacing the path with the correct path
1187
   for your cluster.
1188

    
1189
5. Stop the ganeti daemons (``/etc/init.d/ganeti stop``) and kill any
1190
   that remain alive (``pgrep ganeti`` and ``pkill ganeti``).
1191

    
1192
6. Remove the ganeti state directory (``rm -rf /var/lib/ganeti/*``),
1193
   replacing the path with the correct path for your installation.
1194

    
1195
On the master node, remove the cluster from the master-netdev (usually
1196
``xen-br0`` for bridged mode, otherwise ``eth0`` or similar), by running
1197
``ip a del $clusterip/32 dev xen-br0`` (use the correct cluster ip and
1198
network device name).
1199

    
1200
At this point, the machines are ready for a cluster creation; in case
1201
you want to remove Ganeti completely, you need to also undo some of the
1202
SSH changes and log directories:
1203

    
1204
- ``rm -rf /var/log/ganeti /srv/ganeti`` (replace with the correct
1205
  paths)
1206
- remove from ``/root/.ssh`` the keys that Ganeti added (check the
1207
  ``authorized_keys`` and ``id_dsa`` files)
1208
- regenerate the host's SSH keys (check the OpenSSH startup scripts)
1209
- uninstall Ganeti
1210

    
1211
Otherwise, if you plan to re-create the cluster, you can just go ahead
1212
and rerun ``gnt-cluster init``.
1213

    
1214
Tags handling
1215
-------------
1216

    
1217
The tags handling (addition, removal, listing) is similar for all the
1218
objects that support it (instances, nodes, and the cluster).
1219

    
1220
Limitations
1221
+++++++++++
1222

    
1223
Note that the set of characters present in a tag and the maximum tag
1224
length are restricted. Currently the maximum length is 128 characters,
1225
there can be at most 4096 tags per object, and the set of characters is
1226
comprised by alphanumeric characters and additionally ``.+*/:@-``.
1227

    
1228
Operations
1229
++++++++++
1230

    
1231
Tags can be added via ``add-tags``::
1232

    
1233
  gnt-instance add-tags INSTANCE a b c
1234
  gnt-node add-tags INSTANCE a b c
1235
  gnt-cluster add-tags a b c
1236

    
1237

    
1238
The above commands add three tags to an instance, to a node and to the
1239
cluster. Note that the cluster command only takes tags as arguments,
1240
whereas the node and instance commands first required the node and
1241
instance name.
1242

    
1243
Tags can also be added from a file, via the ``--from=FILENAME``
1244
argument. The file is expected to contain one tag per line.
1245

    
1246
Tags can also be remove via a syntax very similar to the add one::
1247

    
1248
  gnt-instance remove-tags INSTANCE a b c
1249

    
1250
And listed via::
1251

    
1252
  gnt-instance list-tags
1253
  gnt-node list-tags
1254
  gnt-cluster list-tags
1255

    
1256
Global tag search
1257
+++++++++++++++++
1258

    
1259
It is also possible to execute a global search on the all tags defined
1260
in the cluster configuration, via a cluster command::
1261

    
1262
  gnt-cluster search-tags REGEXP
1263

    
1264
The parameter expected is a regular expression (see
1265
:manpage:`regex(7)`). This will return all tags that match the search,
1266
together with the object they are defined in (the names being show in a
1267
hierarchical kind of way)::
1268

    
1269
  node1# gnt-cluster search-tags o
1270
  /cluster foo
1271
  /instances/instance1 owner:bar
1272

    
1273

    
1274
Job operations
1275
--------------
1276

    
1277
The various jobs submitted by the instance/node/cluster commands can be
1278
examined, canceled and archived by various invocations of the
1279
``gnt-job`` command.
1280

    
1281
First is the job list command::
1282

    
1283
  node1# gnt-job list
1284
  17771 success INSTANCE_QUERY_DATA
1285
  17773 success CLUSTER_VERIFY_DISKS
1286
  17775 success CLUSTER_REPAIR_DISK_SIZES
1287
  17776 error   CLUSTER_RENAME(cluster.example.com)
1288
  17780 success CLUSTER_REDIST_CONF
1289
  17792 success INSTANCE_REBOOT(instance1.example.com)
1290

    
1291
More detailed information about a job can be found via the ``info``
1292
command::
1293

    
1294
  node1# gnt-job info 17776
1295
  Job ID: 17776
1296
    Status: error
1297
    Received:         2009-10-25 23:18:02.180569
1298
    Processing start: 2009-10-25 23:18:02.200335 (delta 0.019766s)
1299
    Processing end:   2009-10-25 23:18:02.279743 (delta 0.079408s)
1300
    Total processing time: 0.099174 seconds
1301
    Opcodes:
1302
      OP_CLUSTER_RENAME
1303
        Status: error
1304
        Processing start: 2009-10-25 23:18:02.200335
1305
        Processing end:   2009-10-25 23:18:02.252282
1306
        Input fields:
1307
          name: cluster.example.com
1308
        Result:
1309
          OpPrereqError
1310
          [Neither the name nor the IP address of the cluster has changed]
1311
        Execution log:
1312

    
1313
During the execution of a job, it's possible to follow the output of a
1314
job, similar to the log that one get from the ``gnt-`` commands, via the
1315
watch command::
1316

    
1317
  node1# gnt-instance add --submit … instance1
1318
  JobID: 17818
1319
  node1# gnt-job watch 17818
1320
  Output from job 17818 follows
1321
  -----------------------------
1322
  Mon Oct 26 00:22:48 2009  - INFO: Selected nodes for instance instance1 via iallocator dumb: node1, node2
1323
  Mon Oct 26 00:22:49 2009 * creating instance disks...
1324
  Mon Oct 26 00:22:52 2009 adding instance instance1 to cluster config
1325
  Mon Oct 26 00:22:52 2009  - INFO: Waiting for instance instance1 to sync disks.
1326
1327
  Mon Oct 26 00:23:03 2009 creating os for instance instance1 on node node1
1328
  Mon Oct 26 00:23:03 2009 * running the instance OS create scripts...
1329
  Mon Oct 26 00:23:13 2009 * starting instance...
1330
  node1#
1331

    
1332
This is useful if you need to follow a job's progress from multiple
1333
terminals.
1334

    
1335
A job that has not yet started to run can be canceled::
1336

    
1337
  node1# gnt-job cancel 17810
1338

    
1339
But not one that has already started execution::
1340

    
1341
  node1# gnt-job cancel 17805
1342
  Job 17805 is no longer waiting in the queue
1343

    
1344
There are two queues for jobs: the *current* and the *archive*
1345
queue. Jobs are initially submitted to the current queue, and they stay
1346
in that queue until they have finished execution (either successfully or
1347
not). At that point, they can be moved into the archive queue, and the
1348
ganeti-watcher script will do this automatically after 6 hours. The
1349
ganeti-cleaner script will remove the jobs from the archive directory
1350
after three weeks.
1351

    
1352
Note that only jobs in the current queue can be viewed via the list and
1353
info commands; Ganeti itself doesn't examine the archive directory. If
1354
you need to see an older job, either move the file manually in the
1355
top-level queue directory, or look at its contents (it's a
1356
JSON-formatted file).
1357

    
1358
Special Ganeti deployments
1359
--------------------------
1360

    
1361
Since Ganeti 2.4, it is possible to extend the Ganeti deployment with
1362
two custom scenarios: Ganeti inside Ganeti and multi-site model.
1363

    
1364
Running Ganeti under Ganeti
1365
+++++++++++++++++++++++++++
1366

    
1367
It is sometimes useful to be able to use a Ganeti instance as a Ganeti
1368
node (part of another cluster, usually). One example scenario is two
1369
small clusters, where we want to have an additional master candidate
1370
that holds the cluster configuration and can be used for helping with
1371
the master voting process.
1372

    
1373
However, these Ganeti instance should not host instances themselves, and
1374
should not be considered in the normal capacity planning, evacuation
1375
strategies, etc. In order to accomplish this, mark these nodes as
1376
non-``vm_capable``::
1377

    
1378
  node1# gnt-node modify --vm-capable=no node3
1379

    
1380
The vm_capable status can be listed as usual via ``gnt-node list``::
1381

    
1382
  node1# gnt-node list -oname,vm_capable
1383
  Node  VMCapable
1384
  node1 Y
1385
  node2 Y
1386
  node3 N
1387

    
1388
When this flag is set, the cluster will not do any operations that
1389
relate to instances on such nodes, e.g. hypervisor operations,
1390
disk-related operations, etc. Basically they will just keep the ssconf
1391
files, and if master candidates the full configuration.
1392

    
1393
Multi-site model
1394
++++++++++++++++
1395

    
1396
If Ganeti is deployed in multi-site model, with each site being a node
1397
group (so that instances are not relocated across the WAN by mistake),
1398
it is conceivable that either the WAN latency is high or that some sites
1399
have a lower reliability than others. In this case, it doesn't make
1400
sense to replicate the job information across all sites (or even outside
1401
of a “central” node group), so it should be possible to restrict which
1402
nodes can become master candidates via the auto-promotion algorithm.
1403

    
1404
Ganeti 2.4 introduces for this purpose a new ``master_capable`` flag,
1405
which (when unset) prevents nodes from being marked as master
1406
candidates, either manually or automatically.
1407

    
1408
As usual, the node modify operation can change this flag::
1409

    
1410
  node1# gnt-node modify --auto-promote --master-capable=no node3
1411
  Fri Jan  7 06:23:07 2011  - INFO: Demoting from master candidate
1412
  Fri Jan  7 06:23:08 2011  - INFO: Promoted nodes to master candidate role: node4
1413
  Modified node node3
1414
   - master_capable -> False
1415
   - master_candidate -> False
1416

    
1417
And the node list operation will list this flag::
1418

    
1419
  node1# gnt-node list -oname,master_capable node1 node2 node3
1420
  Node  MasterCapable
1421
  node1 Y
1422
  node2 Y
1423
  node3 N
1424

    
1425
Note that marking a node both not ``vm_capable`` and not
1426
``master_capable`` makes the node practically unusable from Ganeti's
1427
point of view. Hence these two flags should be used probably in
1428
contrast: some nodes will be only master candidates (master_capable but
1429
not vm_capable), and other nodes will only hold instances (vm_capable
1430
but not master_capable).
1431

    
1432

    
1433
Ganeti tools
1434
------------
1435

    
1436
Beside the usual ``gnt-`` and ``ganeti-`` commands which are provided
1437
and installed in ``$prefix/sbin`` at install time, there are a couple of
1438
other tools installed which are used seldom but can be helpful in some
1439
cases.
1440

    
1441
lvmstrap
1442
++++++++
1443

    
1444
The ``lvmstrap`` tool, introduced in :ref:`configure-lvm-label` section,
1445
has two modes of operation:
1446

    
1447
- ``diskinfo`` shows the discovered disks on the system and their status
1448
- ``create`` takes all not-in-use disks and creates a volume group out
1449
  of them
1450

    
1451
.. warning:: The ``create`` argument to this command causes data-loss!
1452

    
1453
cfgupgrade
1454
++++++++++
1455

    
1456
The ``cfgupgrade`` tools is used to upgrade between major (and minor)
1457
Ganeti versions. Point-releases are usually transparent for the admin.
1458

    
1459
More information about the upgrade procedure is listed on the wiki at
1460
http://code.google.com/p/ganeti/wiki/UpgradeNotes.
1461

    
1462
There is also a script designed to upgrade from Ganeti 1.2 to 2.0,
1463
called ``cfgupgrade12``.
1464

    
1465
cfgshell
1466
++++++++
1467

    
1468
.. note:: This command is not actively maintained; make sure you backup
1469
   your configuration before using it
1470

    
1471
This can be used as an alternative to direct editing of the
1472
main configuration file if Ganeti has a bug and prevents you, for
1473
example, from removing an instance or a node from the configuration
1474
file.
1475

    
1476
.. _burnin-label:
1477

    
1478
burnin
1479
++++++
1480

    
1481
.. warning:: This command will erase existing instances if given as
1482
   arguments!
1483

    
1484
This tool is used to exercise either the hardware of machines or
1485
alternatively the Ganeti software. It is safe to run on an existing
1486
cluster **as long as you don't pass it existing instance names**.
1487

    
1488
The command will, by default, execute a comprehensive set of operations
1489
against a list of instances, these being:
1490

    
1491
- creation
1492
- disk replacement (for redundant instances)
1493
- failover and migration (for redundant instances)
1494
- move (for non-redundant instances)
1495
- disk growth
1496
- add disks, remove disk
1497
- add NICs, remove NICs
1498
- export and then import
1499
- rename
1500
- reboot
1501
- shutdown/startup
1502
- and finally removal of the test instances
1503

    
1504
Executing all these operations will test that the hardware performs
1505
well: the creation, disk replace, disk add and disk growth will exercise
1506
the storage and network; the migrate command will test the memory of the
1507
systems. Depending on the passed options, it can also test that the
1508
instance OS definitions are executing properly the rename, import and
1509
export operations.
1510

    
1511
sanitize-config
1512
+++++++++++++++
1513

    
1514
This tool takes the Ganeti configuration and outputs a "sanitized"
1515
version, by randomizing or clearing:
1516

    
1517
- DRBD secrets and cluster public key (always)
1518
- host names (optional)
1519
- IPs (optional)
1520
- OS names (optional)
1521
- LV names (optional, only useful for very old clusters which still have
1522
  instances whose LVs are based on the instance name)
1523

    
1524
By default, all optional items are activated except the LV name
1525
randomization. When passing ``--no-randomization``, which disables the
1526
optional items (i.e. just the DRBD secrets and cluster public keys are
1527
randomized), the resulting file can be used as a safety copy of the
1528
cluster config - while not trivial, the layout of the cluster can be
1529
recreated from it and if the instance disks have not been lost it
1530
permits recovery from the loss of all master candidates.
1531

    
1532
move-instance
1533
+++++++++++++
1534

    
1535
See :doc:`separate documentation for move-instance <move-instance>`.
1536

    
1537
.. TODO: document cluster-merge tool
1538

    
1539

    
1540
Other Ganeti projects
1541
---------------------
1542

    
1543
There are two other Ganeti-related projects that can be useful in a
1544
Ganeti deployment. These can be downloaded from the project site
1545
(http://code.google.com/p/ganeti/) and the repositories are also on the
1546
project git site (http://git.ganeti.org).
1547

    
1548
NBMA tools
1549
++++++++++
1550

    
1551
The ``ganeti-nbma`` software is designed to allow instances to live on a
1552
separate, virtual network from the nodes, and in an environment where
1553
nodes are not guaranteed to be able to reach each other via multicasting
1554
or broadcasting. For more information see the README in the source
1555
archive.
1556

    
1557
ganeti-htools
1558
+++++++++++++
1559

    
1560
The ``ganeti-htools`` software consists of a set of tools:
1561

    
1562
- ``hail``: an advanced iallocator script compared to Ganeti's builtin
1563
  one
1564
- ``hbal``: a tool for rebalancing the cluster, i.e. moving instances
1565
  around in order to better use the resources on the nodes
1566
- ``hspace``: a tool for estimating the available capacity of a cluster,
1567
  so that capacity planning can be done efficiently
1568

    
1569
For more information and installation instructions, see the README file
1570
in the source archive.
1571

    
1572
.. vim: set textwidth=72 :
1573
.. Local Variables:
1574
.. mode: rst
1575
.. fill-column: 72
1576
.. End: