Statistics
| Branch: | Tag: | Revision:

root / doc / admin.rst @ d2baa21d

History | View | Annotate | Download (45.4 kB)

1 ffa6869f Iustin Pop
Ganeti administrator's guide
2 ffa6869f Iustin Pop
============================
3 ffa6869f Iustin Pop
4 fd07c6b3 Iustin Pop
Documents Ganeti version |version|
5 ffa6869f Iustin Pop
6 ffa6869f Iustin Pop
.. contents::
7 ffa6869f Iustin Pop
8 c71a1a3d Iustin Pop
.. highlight:: text
9 c71a1a3d Iustin Pop
10 ffa6869f Iustin Pop
Introduction
11 ffa6869f Iustin Pop
------------
12 ffa6869f Iustin Pop
13 c71a1a3d Iustin Pop
Ganeti is a virtualization cluster management software. You are expected
14 c71a1a3d Iustin Pop
to be a system administrator familiar with your Linux distribution and
15 c71a1a3d Iustin Pop
the Xen or KVM virtualization environments before using it.
16 ffa6869f Iustin Pop
17 ffa6869f Iustin Pop
The various components of Ganeti all have man pages and interactive
18 c71a1a3d Iustin Pop
help. This manual though will help you getting familiar with the system
19 c71a1a3d Iustin Pop
by explaining the most common operations, grouped by related use.
20 ffa6869f Iustin Pop
21 ffa6869f Iustin Pop
After a terminology glossary and a section on the prerequisites needed
22 c71a1a3d Iustin Pop
to use this manual, the rest of this document is divided in sections
23 c71a1a3d Iustin Pop
for the different targets that a command affects: instance, nodes, etc.
24 ffa6869f Iustin Pop
25 c71a1a3d Iustin Pop
.. _terminology-label:
26 ffa6869f Iustin Pop
27 ffa6869f Iustin Pop
Ganeti terminology
28 c71a1a3d Iustin Pop
++++++++++++++++++
29 ffa6869f Iustin Pop
30 c71a1a3d Iustin Pop
This section provides a small introduction to Ganeti terminology, which
31 c71a1a3d Iustin Pop
might be useful when reading the rest of the document.
32 ffa6869f Iustin Pop
33 ffa6869f Iustin Pop
Cluster
34 c71a1a3d Iustin Pop
~~~~~~~
35 ffa6869f Iustin Pop
36 c71a1a3d Iustin Pop
A set of machines (nodes) that cooperate to offer a coherent, highly
37 c71a1a3d Iustin Pop
available virtualization service under a single administration domain.
38 ffa6869f Iustin Pop
39 c71a1a3d Iustin Pop
Node
40 c71a1a3d Iustin Pop
~~~~
41 c71a1a3d Iustin Pop
42 c71a1a3d Iustin Pop
A physical machine which is member of a cluster.  Nodes are the basic
43 c71a1a3d Iustin Pop
cluster infrastructure, and they don't need to be fault tolerant in
44 c71a1a3d Iustin Pop
order to achieve high availability for instances.
45 c71a1a3d Iustin Pop
46 c71a1a3d Iustin Pop
Node can be added and removed (if they host no instances) at will from
47 c71a1a3d Iustin Pop
the cluster. In a HA cluster and only with HA instances, the loss of any
48 c71a1a3d Iustin Pop
single node will not cause disk data loss for any instance; of course,
49 c71a1a3d Iustin Pop
a node crash will cause the crash of the its primary instances.
50 c71a1a3d Iustin Pop
51 c71a1a3d Iustin Pop
A node belonging to a cluster can be in one of the following roles at a
52 c71a1a3d Iustin Pop
given time:
53 c71a1a3d Iustin Pop
54 c71a1a3d Iustin Pop
- *master* node, which is the node from which the cluster is controlled
55 c71a1a3d Iustin Pop
- *master candidate* node, only nodes in this role have the full cluster
56 c71a1a3d Iustin Pop
  configuration and knowledge, and only master candidates can become the
57 c71a1a3d Iustin Pop
  master node
58 c71a1a3d Iustin Pop
- *regular* node, which is the state in which most nodes will be on
59 c71a1a3d Iustin Pop
  bigger clusters (>20 nodes)
60 c71a1a3d Iustin Pop
- *drained* node, nodes in this state are functioning normally but the
61 c71a1a3d Iustin Pop
  cannot receive new instances; the intention is that nodes in this role
62 c71a1a3d Iustin Pop
  have some issue and they are being evacuated for hardware repairs
63 c71a1a3d Iustin Pop
- *offline* node, in which there is a record in the cluster
64 c71a1a3d Iustin Pop
  configuration about the node, but the daemons on the master node will
65 c71a1a3d Iustin Pop
  not talk to this node; any instances declared as having an offline
66 c71a1a3d Iustin Pop
  node as either primary or secondary will be flagged as an error in the
67 c71a1a3d Iustin Pop
  cluster verify operation
68 c71a1a3d Iustin Pop
69 c71a1a3d Iustin Pop
Depending on the role, each node will run a set of daemons:
70 c71a1a3d Iustin Pop
71 c71a1a3d Iustin Pop
- the :command:`ganeti-noded` daemon, which control the manipulation of
72 c71a1a3d Iustin Pop
  this node's hardware resources; it runs on all nodes which are in a
73 c71a1a3d Iustin Pop
  cluster
74 c71a1a3d Iustin Pop
- the :command:`ganeti-confd` daemon (Ganeti 2.1+) which runs on all
75 c71a1a3d Iustin Pop
  nodes, but is only functional on master candidate nodes
76 c71a1a3d Iustin Pop
- the :command:`ganeti-rapi` daemon which runs on the master node and
77 c71a1a3d Iustin Pop
  offers an HTTP-based API for the cluster
78 c71a1a3d Iustin Pop
- the :command:`ganeti-masterd` daemon which runs on the master node and
79 c71a1a3d Iustin Pop
  allows control of the cluster
80 ffa6869f Iustin Pop
81 ffa6869f Iustin Pop
Instance
82 c71a1a3d Iustin Pop
~~~~~~~~
83 ffa6869f Iustin Pop
84 c71a1a3d Iustin Pop
A virtual machine which runs on a cluster. It can be a fault tolerant,
85 c71a1a3d Iustin Pop
highly available entity.
86 ffa6869f Iustin Pop
87 c71a1a3d Iustin Pop
An instance has various parameters, which are classified in three
88 c71a1a3d Iustin Pop
categories: hypervisor related-parameters (called ``hvparams``), general
89 c71a1a3d Iustin Pop
parameters (called ``beparams``) and per network-card parameters (called
90 c71a1a3d Iustin Pop
``nicparams``). All these parameters can be modified either at instance
91 c71a1a3d Iustin Pop
level or via defaults at cluster level.
92 ffa6869f Iustin Pop
93 c71a1a3d Iustin Pop
Disk template
94 ffa6869f Iustin Pop
~~~~~~~~~~~~~
95 ffa6869f Iustin Pop
96 c71a1a3d Iustin Pop
The are multiple options for the storage provided to an instance; while
97 c71a1a3d Iustin Pop
the instance sees the same virtual drive in all cases, the node-level
98 c71a1a3d Iustin Pop
configuration varies between them.
99 ffa6869f Iustin Pop
100 c71a1a3d Iustin Pop
There are four disk templates you can choose from:
101 ffa6869f Iustin Pop
102 c71a1a3d Iustin Pop
diskless
103 c71a1a3d Iustin Pop
  The instance has no disks. Only used for special purpose operating
104 c71a1a3d Iustin Pop
  systems or for testing.
105 c71a1a3d Iustin Pop
106 c71a1a3d Iustin Pop
file
107 c71a1a3d Iustin Pop
  The instance will use plain files as backend for its disks. No
108 c71a1a3d Iustin Pop
  redundancy is provided, and this is somewhat more difficult to
109 c71a1a3d Iustin Pop
  configure for high performance.
110 c71a1a3d Iustin Pop
111 c71a1a3d Iustin Pop
plain
112 c71a1a3d Iustin Pop
  The instance will use LVM devices as backend for its disks. No
113 c71a1a3d Iustin Pop
  redundancy is provided.
114 c71a1a3d Iustin Pop
115 c71a1a3d Iustin Pop
drbd
116 c71a1a3d Iustin Pop
  .. note:: This is only valid for multi-node clusters using DRBD 8.0+
117 c71a1a3d Iustin Pop
118 c71a1a3d Iustin Pop
  A mirror is set between the local node and a remote one, which must be
119 c71a1a3d Iustin Pop
  specified with the second value of the --node option. Use this option
120 c71a1a3d Iustin Pop
  to obtain a highly available instance that can be failed over to a
121 c71a1a3d Iustin Pop
  remote node should the primary one fail.
122 c71a1a3d Iustin Pop
123 c71a1a3d Iustin Pop
IAllocator
124 c71a1a3d Iustin Pop
~~~~~~~~~~
125 c71a1a3d Iustin Pop
126 c71a1a3d Iustin Pop
A framework for using external (user-provided) scripts to compute the
127 c71a1a3d Iustin Pop
placement of instances on the cluster nodes. This eliminates the need to
128 c71a1a3d Iustin Pop
manually specify nodes in instance add, instance moves, node evacuate,
129 c71a1a3d Iustin Pop
etc.
130 c71a1a3d Iustin Pop
131 c71a1a3d Iustin Pop
In order for Ganeti to be able to use these scripts, they must be place
132 c71a1a3d Iustin Pop
in the iallocator directory (usually ``lib/ganeti/iallocators`` under
133 c71a1a3d Iustin Pop
the installation prefix, e.g. ``/usr/local``).
134 c71a1a3d Iustin Pop
135 c71a1a3d Iustin Pop
“Primary” and “secondary” concepts
136 c71a1a3d Iustin Pop
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
137 c71a1a3d Iustin Pop
138 c71a1a3d Iustin Pop
An instance has a primary and depending on the disk configuration, might
139 c71a1a3d Iustin Pop
also have a secondary node. The instance always runs on the primary node
140 c71a1a3d Iustin Pop
and only uses its secondary node for disk replication.
141 c71a1a3d Iustin Pop
142 c71a1a3d Iustin Pop
Similarly, the term of primary and secondary instances when talking
143 c71a1a3d Iustin Pop
about a node refers to the set of instances having the given node as
144 c71a1a3d Iustin Pop
primary, respectively secondary.
145 c71a1a3d Iustin Pop
146 c71a1a3d Iustin Pop
Tags
147 c71a1a3d Iustin Pop
~~~~
148 c71a1a3d Iustin Pop
149 c71a1a3d Iustin Pop
Tags are short strings that can be attached to either to cluster itself,
150 c71a1a3d Iustin Pop
or to nodes or instances. They are useful as a very simplistic
151 c71a1a3d Iustin Pop
information store for helping with cluster administration, for example
152 c71a1a3d Iustin Pop
by attaching owner information to each instance after it's created::
153 ffa6869f Iustin Pop
154 c71a1a3d Iustin Pop
  gnt-instance add … instance1
155 c71a1a3d Iustin Pop
  gnt-instance add-tags instance1 owner:user2
156 c71a1a3d Iustin Pop
157 c71a1a3d Iustin Pop
And then by listing each instance and its tags, this information could
158 c71a1a3d Iustin Pop
be used for contacting the users of each instance.
159 c71a1a3d Iustin Pop
160 c71a1a3d Iustin Pop
Jobs and OpCodes
161 c71a1a3d Iustin Pop
~~~~~~~~~~~~~~~~
162 c71a1a3d Iustin Pop
163 c71a1a3d Iustin Pop
While not directly visible by an end-user, it's useful to know that a
164 c71a1a3d Iustin Pop
basic cluster operation (e.g. starting an instance) is represented
165 c71a1a3d Iustin Pop
internall by Ganeti as an *OpCode* (abbreviation from operation
166 c71a1a3d Iustin Pop
code). These OpCodes are executed as part of a *Job*. The OpCodes in a
167 c71a1a3d Iustin Pop
single Job are processed serially by Ganeti, but different Jobs will be
168 c71a1a3d Iustin Pop
processed (depending on resource availability) in parallel.
169 c71a1a3d Iustin Pop
170 c71a1a3d Iustin Pop
For example, shutting down the entire cluster can be done by running the
171 c71a1a3d Iustin Pop
command ``gnt-instance shutdown --all``, which will submit for each
172 c71a1a3d Iustin Pop
instance a separate job containing the “shutdown instance” OpCode.
173 c71a1a3d Iustin Pop
174 c71a1a3d Iustin Pop
175 c71a1a3d Iustin Pop
Prerequisites
176 c71a1a3d Iustin Pop
+++++++++++++
177 c71a1a3d Iustin Pop
178 c71a1a3d Iustin Pop
You need to have your Ganeti cluster installed and configured before you
179 c71a1a3d Iustin Pop
try any of the commands in this document. Please follow the
180 c71a1a3d Iustin Pop
:doc:`install` for instructions on how to do that.
181 c71a1a3d Iustin Pop
182 c71a1a3d Iustin Pop
Instance management
183 c71a1a3d Iustin Pop
-------------------
184 c71a1a3d Iustin Pop
185 c71a1a3d Iustin Pop
Adding an instance
186 c71a1a3d Iustin Pop
++++++++++++++++++
187 c71a1a3d Iustin Pop
188 c71a1a3d Iustin Pop
The add operation might seem complex due to the many parameters it
189 c71a1a3d Iustin Pop
accepts, but once you have understood the (few) required parameters and
190 c71a1a3d Iustin Pop
the customisation capabilities you will see it is an easy operation.
191 c71a1a3d Iustin Pop
192 c71a1a3d Iustin Pop
The add operation requires at minimum five parameters:
193 c71a1a3d Iustin Pop
194 c71a1a3d Iustin Pop
- the OS for the instance
195 c71a1a3d Iustin Pop
- the disk template
196 c71a1a3d Iustin Pop
- the disk count and size
197 c71a1a3d Iustin Pop
- the node specification or alternatively the iallocator to use
198 c71a1a3d Iustin Pop
- and finally the instance name
199 c71a1a3d Iustin Pop
200 c71a1a3d Iustin Pop
The OS for the instance must be visible in the output of the command
201 c71a1a3d Iustin Pop
``gnt-os list`` and specifies which guest OS to install on the instance.
202 c71a1a3d Iustin Pop
203 c71a1a3d Iustin Pop
The disk template specifies what kind of storage to use as backend for
204 c71a1a3d Iustin Pop
the (virtual) disks presented to the instance; note that for instances
205 c71a1a3d Iustin Pop
with multiple virtual disks, they all must be of the same type.
206 c71a1a3d Iustin Pop
207 c71a1a3d Iustin Pop
The node(s) on which the instance will run can be given either manually,
208 c71a1a3d Iustin Pop
via the ``-n`` option, or computed automatically by Ganeti, if you have
209 c71a1a3d Iustin Pop
installed any iallocator script.
210 c71a1a3d Iustin Pop
211 c71a1a3d Iustin Pop
With the above parameters in mind, the command is::
212 ffa6869f Iustin Pop
213 ffa6869f Iustin Pop
  gnt-instance add \
214 c71a1a3d Iustin Pop
    -n TARGET_NODE:SECONDARY_NODE \
215 c71a1a3d Iustin Pop
    -o OS_TYPE \
216 c71a1a3d Iustin Pop
    -t DISK_TEMPLATE -s DISK_SIZE \
217 ffa6869f Iustin Pop
    INSTANCE_NAME
218 ffa6869f Iustin Pop
219 ffa6869f Iustin Pop
The instance name must be resolvable (e.g. exist in DNS) and usually
220 c71a1a3d Iustin Pop
points to an address in the same subnet as the cluster itself.
221 ffa6869f Iustin Pop
222 c71a1a3d Iustin Pop
The above command has the minimum required options; other options you
223 c71a1a3d Iustin Pop
can give include, among others:
224 ffa6869f Iustin Pop
225 ffa6869f Iustin Pop
- The memory size (``-B memory``)
226 ffa6869f Iustin Pop
227 ffa6869f Iustin Pop
- The number of virtual CPUs (``-B vcpus``)
228 ffa6869f Iustin Pop
229 ffa6869f Iustin Pop
- Arguments for the NICs of the instance; by default, a single-NIC
230 ffa6869f Iustin Pop
  instance is created. The IP and/or bridge of the NIC can be changed
231 ffa6869f Iustin Pop
  via ``--nic 0:ip=IP,bridge=BRIDGE``
232 ffa6869f Iustin Pop
233 c71a1a3d Iustin Pop
See the manpage for gnt-instance for the detailed option list.
234 ffa6869f Iustin Pop
235 c71a1a3d Iustin Pop
For example if you want to create an highly available instance, with a
236 c71a1a3d Iustin Pop
single disk of 50GB and the default memory size, having primary node
237 c71a1a3d Iustin Pop
``node1`` and secondary node ``node3``, use the following command::
238 ffa6869f Iustin Pop
239 c71a1a3d Iustin Pop
  gnt-instance add -n node1:node3 -o debootstrap -t drbd \
240 c71a1a3d Iustin Pop
    instance1
241 ffa6869f Iustin Pop
242 c71a1a3d Iustin Pop
There is a also a command for batch instance creation from a
243 c71a1a3d Iustin Pop
specification file, see the ``batch-create`` operation in the
244 c71a1a3d Iustin Pop
gnt-instance manual page.
245 ffa6869f Iustin Pop
246 c71a1a3d Iustin Pop
Regular instance operations
247 c71a1a3d Iustin Pop
+++++++++++++++++++++++++++
248 ffa6869f Iustin Pop
249 c71a1a3d Iustin Pop
Removal
250 c71a1a3d Iustin Pop
~~~~~~~
251 ffa6869f Iustin Pop
252 c71a1a3d Iustin Pop
Removing an instance is even easier than creating one. This operation is
253 c71a1a3d Iustin Pop
irreversible and destroys all the contents of your instance. Use with
254 c71a1a3d Iustin Pop
care::
255 ffa6869f Iustin Pop
256 ffa6869f Iustin Pop
  gnt-instance remove INSTANCE_NAME
257 ffa6869f Iustin Pop
258 c71a1a3d Iustin Pop
Startup/shutdown
259 c71a1a3d Iustin Pop
~~~~~~~~~~~~~~~~
260 ffa6869f Iustin Pop
261 ffa6869f Iustin Pop
Instances are automatically started at instance creation time. To
262 ffa6869f Iustin Pop
manually start one which is currently stopped you can run::
263 ffa6869f Iustin Pop
264 ffa6869f Iustin Pop
  gnt-instance startup INSTANCE_NAME
265 ffa6869f Iustin Pop
266 ffa6869f Iustin Pop
While the command to stop one is::
267 ffa6869f Iustin Pop
268 ffa6869f Iustin Pop
  gnt-instance shutdown INSTANCE_NAME
269 ffa6869f Iustin Pop
270 c71a1a3d Iustin Pop
.. warning:: Do not use the Xen or KVM commands directly to stop
271 c71a1a3d Iustin Pop
   instances. If you run for example ``xm shutdown`` or ``xm destroy``
272 c71a1a3d Iustin Pop
   on an instance Ganeti will automatically restart it (via the
273 c71a1a3d Iustin Pop
   :command:`ganeti-watcher` command which is launched via cron).
274 c71a1a3d Iustin Pop
275 c71a1a3d Iustin Pop
Querying instances
276 c71a1a3d Iustin Pop
~~~~~~~~~~~~~~~~~~
277 c71a1a3d Iustin Pop
278 c71a1a3d Iustin Pop
There are two ways to get information about instances: listing
279 c71a1a3d Iustin Pop
instances, which does a tabular output containing a given set of fields
280 c71a1a3d Iustin Pop
about each instance, and querying detailed information about a set of
281 c71a1a3d Iustin Pop
instances.
282 c71a1a3d Iustin Pop
283 ffa6869f Iustin Pop
The command to see all the instances configured and their status is::
284 ffa6869f Iustin Pop
285 ffa6869f Iustin Pop
  gnt-instance list
286 ffa6869f Iustin Pop
287 c71a1a3d Iustin Pop
The command can return a custom set of information when using the ``-o``
288 c71a1a3d Iustin Pop
option (as always, check the manpage for a detailed specification). Each
289 c71a1a3d Iustin Pop
instance will be represented on a line, thus making it easy to parse
290 c71a1a3d Iustin Pop
this output via the usual shell utilities (grep, sed, etc.).
291 c71a1a3d Iustin Pop
292 c71a1a3d Iustin Pop
To get more detailed information about an instance, you can run::
293 c71a1a3d Iustin Pop
294 c71a1a3d Iustin Pop
  gnt-instance info INSTANCE
295 c71a1a3d Iustin Pop
296 c71a1a3d Iustin Pop
which will give a multi-line block of information about the instance,
297 c71a1a3d Iustin Pop
it's hardware resources (especially its disks and their redundancy
298 c71a1a3d Iustin Pop
status), etc. This is harder to parse and is more expensive than the
299 c71a1a3d Iustin Pop
list operation, but returns much more detailed information.
300 ffa6869f Iustin Pop
301 ffa6869f Iustin Pop
302 c71a1a3d Iustin Pop
Export/Import
303 c71a1a3d Iustin Pop
+++++++++++++
304 c71a1a3d Iustin Pop
305 c71a1a3d Iustin Pop
You can create a snapshot of an instance disk and its Ganeti
306 ffa6869f Iustin Pop
configuration, which then you can backup, or import into another
307 ffa6869f Iustin Pop
cluster. The way to export an instance is::
308 ffa6869f Iustin Pop
309 ffa6869f Iustin Pop
  gnt-backup export -n TARGET_NODE INSTANCE_NAME
310 ffa6869f Iustin Pop
311 c71a1a3d Iustin Pop
312 ffa6869f Iustin Pop
The target node can be any node in the cluster with enough space under
313 c71a1a3d Iustin Pop
``/srv/ganeti`` to hold the instance image. Use the ``--noshutdown``
314 c71a1a3d Iustin Pop
option to snapshot an instance without rebooting it. Note that Ganeti
315 c71a1a3d Iustin Pop
only keeps one snapshot for an instance - any previous snapshot of the
316 c71a1a3d Iustin Pop
same instance existing cluster-wide under ``/srv/ganeti`` will be
317 c71a1a3d Iustin Pop
removed by this operation: if you want to keep them, you need to move
318 c71a1a3d Iustin Pop
them out of the Ganeti exports directory.
319 ffa6869f Iustin Pop
320 c71a1a3d Iustin Pop
Importing an instance is similar to creating a new one, but additionally
321 c71a1a3d Iustin Pop
one must specify the location of the snapshot. The command is::
322 ffa6869f Iustin Pop
323 ffa6869f Iustin Pop
  gnt-backup import -n TARGET_NODE -t DISK_TEMPLATE \
324 ffa6869f Iustin Pop
    --src-node=NODE --src-dir=DIR INSTANCE_NAME
325 ffa6869f Iustin Pop
326 fd07c6b3 Iustin Pop
Most of the options available for the command :command:`gnt-instance
327 fd07c6b3 Iustin Pop
add` are supported here too.
328 ffa6869f Iustin Pop
329 c71a1a3d Iustin Pop
Instance HA features
330 c71a1a3d Iustin Pop
--------------------
331 ffa6869f Iustin Pop
332 ffa6869f Iustin Pop
.. note:: This section only applies to multi-node clusters
333 ffa6869f Iustin Pop
334 c71a1a3d Iustin Pop
.. _instance-change-primary-label:
335 c71a1a3d Iustin Pop
336 c71a1a3d Iustin Pop
Changing the primary node
337 c71a1a3d Iustin Pop
+++++++++++++++++++++++++
338 c71a1a3d Iustin Pop
339 c71a1a3d Iustin Pop
There are three ways to exchange an instance's primary and secondary
340 c71a1a3d Iustin Pop
nodes; the right one to choose depends on how the instance has been
341 c71a1a3d Iustin Pop
created and the status of its current primary node. See
342 c71a1a3d Iustin Pop
:ref:`rest-redundancy-label` for information on changing the secondary
343 c71a1a3d Iustin Pop
node. Note that it's only possible to change the primary node to the
344 c71a1a3d Iustin Pop
secondary and vice-versa; a direct change of the primary node with a
345 c71a1a3d Iustin Pop
third node, while keeping the current secondary is not possible in a
346 c71a1a3d Iustin Pop
single step, only via multiple operations as detailed in
347 c71a1a3d Iustin Pop
:ref:`instance-relocation-label`.
348 c71a1a3d Iustin Pop
349 ffa6869f Iustin Pop
Failing over an instance
350 ffa6869f Iustin Pop
~~~~~~~~~~~~~~~~~~~~~~~~
351 ffa6869f Iustin Pop
352 ffa6869f Iustin Pop
If an instance is built in highly available mode you can at any time
353 ffa6869f Iustin Pop
fail it over to its secondary node, even if the primary has somehow
354 ffa6869f Iustin Pop
failed and it's not up anymore. Doing it is really easy, on the master
355 ffa6869f Iustin Pop
node you can just run::
356 ffa6869f Iustin Pop
357 ffa6869f Iustin Pop
  gnt-instance failover INSTANCE_NAME
358 ffa6869f Iustin Pop
359 ffa6869f Iustin Pop
That's it. After the command completes the secondary node is now the
360 c71a1a3d Iustin Pop
primary, and vice-versa.
361 ffa6869f Iustin Pop
362 ffa6869f Iustin Pop
Live migrating an instance
363 ffa6869f Iustin Pop
~~~~~~~~~~~~~~~~~~~~~~~~~~
364 ffa6869f Iustin Pop
365 c71a1a3d Iustin Pop
If an instance is built in highly available mode, it currently runs and
366 c71a1a3d Iustin Pop
both its nodes are running fine, you can at migrate it over to its
367 c71a1a3d Iustin Pop
secondary node, without downtime. On the master node you need to run::
368 ffa6869f Iustin Pop
369 ffa6869f Iustin Pop
  gnt-instance migrate INSTANCE_NAME
370 ffa6869f Iustin Pop
371 c71a1a3d Iustin Pop
The current load on the instance and its memory size will influence how
372 c71a1a3d Iustin Pop
long the migration will take. In any case, for both KVM and Xen
373 c71a1a3d Iustin Pop
hypervisors, the migration will be transparent to the instance.
374 ffa6869f Iustin Pop
375 c71a1a3d Iustin Pop
Moving an instance (offline)
376 c71a1a3d Iustin Pop
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
377 ffa6869f Iustin Pop
378 c71a1a3d Iustin Pop
If an instance has not been create as mirrored, then the only way to
379 c71a1a3d Iustin Pop
change its primary node is to execute the move command::
380 ffa6869f Iustin Pop
381 c71a1a3d Iustin Pop
  gnt-instance move -n NEW_NODE INSTANCE
382 ffa6869f Iustin Pop
383 c71a1a3d Iustin Pop
This has a few prerequisites:
384 ffa6869f Iustin Pop
385 c71a1a3d Iustin Pop
- the instance must be stopped
386 c71a1a3d Iustin Pop
- its current primary node must be on-line and healthy
387 c71a1a3d Iustin Pop
- the disks of the instance must not have any errors
388 ffa6869f Iustin Pop
389 c71a1a3d Iustin Pop
Since this operation actually copies the data from the old node to the
390 c71a1a3d Iustin Pop
new node, expect it to take proportional to the size of the instance's
391 c71a1a3d Iustin Pop
disks and the speed of both the nodes' I/O system and their networking.
392 ffa6869f Iustin Pop
393 c71a1a3d Iustin Pop
Disk operations
394 c71a1a3d Iustin Pop
+++++++++++++++
395 ffa6869f Iustin Pop
396 c71a1a3d Iustin Pop
Disk failures are a common cause of errors in any server
397 c71a1a3d Iustin Pop
deployment. Ganeti offers protection from single-node failure if your
398 c71a1a3d Iustin Pop
instances were created in HA mode, and it also offers ways to restore
399 c71a1a3d Iustin Pop
redundancy after a failure.
400 ffa6869f Iustin Pop
401 c71a1a3d Iustin Pop
Preparing for disk operations
402 c71a1a3d Iustin Pop
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
403 ffa6869f Iustin Pop
404 c71a1a3d Iustin Pop
It is important to note that for Ganeti to be able to do any disk
405 c71a1a3d Iustin Pop
operation, the Linux machines on top of which Ganeti must be consistent;
406 c71a1a3d Iustin Pop
for LVM, this means that the LVM commands must not return failures; it
407 c71a1a3d Iustin Pop
is common that after a complete disk failure, any LVM command aborts
408 c71a1a3d Iustin Pop
with an error similar to::
409 ffa6869f Iustin Pop
410 c71a1a3d Iustin Pop
  # vgs
411 c71a1a3d Iustin Pop
  /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
412 c71a1a3d Iustin Pop
  /dev/sdb1: read failed after 0 of 4096 at 750153695232: Input/output
413 c71a1a3d Iustin Pop
  error
414 c71a1a3d Iustin Pop
  /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
415 c71a1a3d Iustin Pop
  Couldn't find device with uuid
416 c71a1a3d Iustin Pop
  't30jmN-4Rcf-Fr5e-CURS-pawt-z0jU-m1TgeJ'.
417 c71a1a3d Iustin Pop
  Couldn't find all physical volumes for volume group xenvg.
418 ffa6869f Iustin Pop
419 c71a1a3d Iustin Pop
Before restoring an instance's disks to healthy status, it's needed to
420 c71a1a3d Iustin Pop
fix the volume group used by Ganeti so that we can actually create and
421 c71a1a3d Iustin Pop
manage the logical volumes. This is usually done in a multi-step
422 c71a1a3d Iustin Pop
process:
423 ffa6869f Iustin Pop
424 c71a1a3d Iustin Pop
#. first, if the disk is completely gone and LVM commands exit with
425 c71a1a3d Iustin Pop
   “Couldn't find device with uuid…” then you need to run the command::
426 c71a1a3d Iustin Pop
427 c71a1a3d Iustin Pop
    vgreduce --removemissing VOLUME_GROUP
428 c71a1a3d Iustin Pop
429 c71a1a3d Iustin Pop
#. after the above command, the LVM commands should be executing
430 c71a1a3d Iustin Pop
   normally (warnings are normal, but the commands will not fail
431 c71a1a3d Iustin Pop
   completely).
432 c71a1a3d Iustin Pop
433 c71a1a3d Iustin Pop
#. if the failed disk is still visible in the output of the ``pvs``
434 c71a1a3d Iustin Pop
   command, you need to deactivate it from allocations by running::
435 c71a1a3d Iustin Pop
436 c71a1a3d Iustin Pop
    pvs -x n /dev/DISK
437 ffa6869f Iustin Pop
438 c71a1a3d Iustin Pop
At this point, the volume group should be consistent and any bad
439 c71a1a3d Iustin Pop
physical volumes should not longer be available for allocation.
440 c71a1a3d Iustin Pop
441 c71a1a3d Iustin Pop
Note that since version 2.1 Ganeti provides some commands to automate
442 c71a1a3d Iustin Pop
these two operations, see :ref:`storage-units-label`.
443 c71a1a3d Iustin Pop
444 c71a1a3d Iustin Pop
.. _rest-redundancy-label:
445 c71a1a3d Iustin Pop
446 c71a1a3d Iustin Pop
Restoring redundancy for DRBD-based instances
447 c71a1a3d Iustin Pop
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
448 c71a1a3d Iustin Pop
449 c71a1a3d Iustin Pop
A DRBD instance has two nodes, and the storage on one of them has
450 c71a1a3d Iustin Pop
failed. Depending on which node (primary or secondary) has failed, you
451 c71a1a3d Iustin Pop
have three options at hand:
452 c71a1a3d Iustin Pop
453 c71a1a3d Iustin Pop
- if the storage on the primary node has failed, you need to re-create
454 c71a1a3d Iustin Pop
  the disks on it
455 c71a1a3d Iustin Pop
- if the storage on the secondary node has failed, you can either
456 c71a1a3d Iustin Pop
  re-create the disks on it or change the secondary and recreate
457 c71a1a3d Iustin Pop
  redundancy on the new secondary node
458 c71a1a3d Iustin Pop
459 c71a1a3d Iustin Pop
Of course, at any point it's possible to force re-creation of disks even
460 c71a1a3d Iustin Pop
though everything is already fine.
461 c71a1a3d Iustin Pop
462 c71a1a3d Iustin Pop
For all three cases, the ``replace-disks`` operation can be used::
463 c71a1a3d Iustin Pop
464 c71a1a3d Iustin Pop
  # re-create disks on the primary node
465 c71a1a3d Iustin Pop
  gnt-instance replace-disks -p INSTANCE_NAME
466 c71a1a3d Iustin Pop
  # re-create disks on the current secondary
467 c71a1a3d Iustin Pop
  gnt-instance replace-disks -s INSTANCE_NAME
468 c71a1a3d Iustin Pop
  # change the secondary node, via manual specification
469 c71a1a3d Iustin Pop
  gnt-instance replace-disks -n NODE INSTANCE_NAME
470 c71a1a3d Iustin Pop
  # change the secondary node, via an iallocator script
471 c71a1a3d Iustin Pop
  gnt-instance replace-disks -I SCRIPT INSTANCE_NAME
472 c71a1a3d Iustin Pop
  # since Ganeti 2.1: automatically fix the primary or secondary node
473 c71a1a3d Iustin Pop
  gnt-instance replace-disks -a INSTANCE_NAME
474 c71a1a3d Iustin Pop
475 c71a1a3d Iustin Pop
Since the process involves copying all data from the working node to the
476 c71a1a3d Iustin Pop
target node, it will take a while, depending on the instance's disk
477 c71a1a3d Iustin Pop
size, node I/O system and network speed. But it is (baring any network
478 c71a1a3d Iustin Pop
interruption) completely transparent for the instance.
479 c71a1a3d Iustin Pop
480 c71a1a3d Iustin Pop
Re-creating disks for non-redundant instances
481 c71a1a3d Iustin Pop
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
482 c71a1a3d Iustin Pop
483 c71a1a3d Iustin Pop
.. versionadded:: 2.1
484 c71a1a3d Iustin Pop
485 c71a1a3d Iustin Pop
For non-redundant instances, there isn't a copy (except backups) to
486 c71a1a3d Iustin Pop
re-create the disks. But it's possible to at-least re-create empty
487 c71a1a3d Iustin Pop
disks, after which a reinstall can be run, via the ``recreate-disks``
488 c71a1a3d Iustin Pop
command::
489 c71a1a3d Iustin Pop
490 c71a1a3d Iustin Pop
  gnt-instance recreate-disks INSTANCE
491 c71a1a3d Iustin Pop
492 c71a1a3d Iustin Pop
Note that this will fail if the disks already exists.
493 c71a1a3d Iustin Pop
494 c71a1a3d Iustin Pop
Debugging instances
495 c71a1a3d Iustin Pop
+++++++++++++++++++
496 ffa6869f Iustin Pop
497 ffa6869f Iustin Pop
Accessing an instance's disks
498 ffa6869f Iustin Pop
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
499 ffa6869f Iustin Pop
500 c71a1a3d Iustin Pop
From an instance's primary node you can have access to its disks. Never
501 ffa6869f Iustin Pop
ever mount the underlying logical volume manually on a fault tolerant
502 c71a1a3d Iustin Pop
instance, or will break replication and your data will be
503 c71a1a3d Iustin Pop
inconsistent. The correct way to access an instance's disks is to run
504 c71a1a3d Iustin Pop
(on the master node, as usual) the command::
505 c71a1a3d Iustin Pop
506 c71a1a3d Iustin Pop
  gnt-instance activate-disks INSTANCE
507 c71a1a3d Iustin Pop
508 c71a1a3d Iustin Pop
And then, *on the primary node of the instance*, access the device that
509 c71a1a3d Iustin Pop
gets created. For example, you could mount the given disks, then edit
510 c71a1a3d Iustin Pop
files on the filesystem, etc.
511 c71a1a3d Iustin Pop
512 c71a1a3d Iustin Pop
Note that with partitioned disks (as opposed to whole-disk filesystems),
513 c71a1a3d Iustin Pop
you will need to use a tool like :manpage:`kpartx(8)`::
514 ffa6869f Iustin Pop
515 c71a1a3d Iustin Pop
  node1# gnt-instance activate-disks instance1
516 c71a1a3d Iustin Pop
517 c71a1a3d Iustin Pop
  node1# ssh node3
518 c71a1a3d Iustin Pop
  node3# kpartx -l /dev/…
519 c71a1a3d Iustin Pop
  node3# kpartx -a /dev/…
520 c71a1a3d Iustin Pop
  node3# mount /dev/mapper/… /mnt/
521 c71a1a3d Iustin Pop
  # edit files under mnt as desired
522 c71a1a3d Iustin Pop
  node3# umount /mnt/
523 c71a1a3d Iustin Pop
  node3# kpartx -d /dev/…
524 c71a1a3d Iustin Pop
  node3# exit
525 c71a1a3d Iustin Pop
  node1#
526 ffa6869f Iustin Pop
527 c71a1a3d Iustin Pop
After you've finished you can deactivate them with the deactivate-disks
528 c71a1a3d Iustin Pop
command, which works in the same way::
529 c71a1a3d Iustin Pop
530 c71a1a3d Iustin Pop
  gnt-instance deactivate-disks INSTANCE
531 c71a1a3d Iustin Pop
532 c71a1a3d Iustin Pop
Note that if any process started by you is still using the disks, the
533 c71a1a3d Iustin Pop
above command will error out, and you **must** cleanup and ensure that
534 c71a1a3d Iustin Pop
the above command runs successfully before you start the instance,
535 c71a1a3d Iustin Pop
otherwise the instance will suffer corruption.
536 ffa6869f Iustin Pop
537 ffa6869f Iustin Pop
Accessing an instance's console
538 fd07c6b3 Iustin Pop
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
539 ffa6869f Iustin Pop
540 ffa6869f Iustin Pop
The command to access a running instance's console is::
541 ffa6869f Iustin Pop
542 ffa6869f Iustin Pop
  gnt-instance console INSTANCE_NAME
543 ffa6869f Iustin Pop
544 c71a1a3d Iustin Pop
Use the console normally and then type ``^]`` when done, to exit.
545 c71a1a3d Iustin Pop
546 c71a1a3d Iustin Pop
Other instance operations
547 c71a1a3d Iustin Pop
+++++++++++++++++++++++++
548 c71a1a3d Iustin Pop
549 c71a1a3d Iustin Pop
Reboot
550 c71a1a3d Iustin Pop
~~~~~~
551 ffa6869f Iustin Pop
552 c71a1a3d Iustin Pop
There is a wrapper command for rebooting instances::
553 ffa6869f Iustin Pop
554 c71a1a3d Iustin Pop
  gnt-instance reboot instance2
555 c71a1a3d Iustin Pop
556 c71a1a3d Iustin Pop
By default, this does the equivalent of shutting down and then starting
557 c71a1a3d Iustin Pop
the instance, but it accepts parameters to perform a soft-reboot (via
558 c71a1a3d Iustin Pop
the hypervisor), a hard reboot (hypervisor shutdown and then startup) or
559 c71a1a3d Iustin Pop
a full one (the default, which also de-configures and then configures
560 c71a1a3d Iustin Pop
again the disks of the instance).
561 c71a1a3d Iustin Pop
562 c71a1a3d Iustin Pop
Instance OS definitions debugging
563 c71a1a3d Iustin Pop
+++++++++++++++++++++++++++++++++
564 c71a1a3d Iustin Pop
565 c71a1a3d Iustin Pop
Should you have any problems with instance operating systems the command
566 c71a1a3d Iustin Pop
to see a complete status for all your nodes is::
567 ffa6869f Iustin Pop
568 ffa6869f Iustin Pop
   gnt-os diagnose
569 ffa6869f Iustin Pop
570 c71a1a3d Iustin Pop
.. _instance-relocation-label:
571 ffa6869f Iustin Pop
572 c71a1a3d Iustin Pop
Instance relocation
573 c71a1a3d Iustin Pop
~~~~~~~~~~~~~~~~~~~
574 ffa6869f Iustin Pop
575 c71a1a3d Iustin Pop
While it is not possible to move an instance from nodes ``(A, B)`` to
576 c71a1a3d Iustin Pop
nodes ``(C, D)`` in a single move, it is possible to do so in a few
577 c71a1a3d Iustin Pop
steps::
578 ffa6869f Iustin Pop
579 c71a1a3d Iustin Pop
  # instance is located on A, B
580 c71a1a3d Iustin Pop
  node1# gnt-instance replace -n nodeC instance1
581 c71a1a3d Iustin Pop
  # instance has moved from (A, B) to (A, C)
582 c71a1a3d Iustin Pop
  # we now flip the primary/secondary nodes
583 c71a1a3d Iustin Pop
  node1# gnt-instance migrate instance1
584 c71a1a3d Iustin Pop
  # instance lives on (C, A)
585 c71a1a3d Iustin Pop
  # we can then change A to D via:
586 c71a1a3d Iustin Pop
  node1# gnt-instance replace -n nodeD instance1
587 56c9a709 Iustin Pop
588 c71a1a3d Iustin Pop
Which brings it into the final configuration of ``(C, D)``. Note that we
589 c71a1a3d Iustin Pop
needed to do two replace-disks operation (two copies of the instance
590 c71a1a3d Iustin Pop
disks), because we needed to get rid of both the original nodes (A and
591 c71a1a3d Iustin Pop
B).
592 c71a1a3d Iustin Pop
593 c71a1a3d Iustin Pop
Node operations
594 c71a1a3d Iustin Pop
---------------
595 c71a1a3d Iustin Pop
596 c71a1a3d Iustin Pop
There are much fewer node operations available than for instances, but
597 c71a1a3d Iustin Pop
they are equivalently important for maintaining a healthy cluster.
598 c71a1a3d Iustin Pop
599 c71a1a3d Iustin Pop
Add/readd
600 c71a1a3d Iustin Pop
+++++++++
601 c71a1a3d Iustin Pop
602 c71a1a3d Iustin Pop
It is at any time possible to extend the cluster with one more node, by
603 c71a1a3d Iustin Pop
using the node add operation::
604 c71a1a3d Iustin Pop
605 c71a1a3d Iustin Pop
  gnt-node add NEW_NODE
606 c71a1a3d Iustin Pop
607 c71a1a3d Iustin Pop
If the cluster has a replication network defined, then you need to pass
608 c71a1a3d Iustin Pop
the ``-s REPLICATION_IP`` parameter to this option.
609 c71a1a3d Iustin Pop
610 c71a1a3d Iustin Pop
A variation of this command can be used to re-configure a node if its
611 c71a1a3d Iustin Pop
Ganeti configuration is broken, for example if it has been reinstalled
612 c71a1a3d Iustin Pop
by mistake::
613 c71a1a3d Iustin Pop
614 c71a1a3d Iustin Pop
  gnt-node add --readd EXISTING_NODE
615 c71a1a3d Iustin Pop
616 c71a1a3d Iustin Pop
This will reinitialise the node as if it's been newly added, but while
617 c71a1a3d Iustin Pop
keeping its existing configuration in the cluster (primary/secondary IP,
618 c71a1a3d Iustin Pop
etc.), in other words you won't need to use ``-s`` here.
619 c71a1a3d Iustin Pop
620 c71a1a3d Iustin Pop
Changing the node role
621 c71a1a3d Iustin Pop
++++++++++++++++++++++
622 c71a1a3d Iustin Pop
623 c71a1a3d Iustin Pop
A node can be in different roles, as explained in the
624 c71a1a3d Iustin Pop
:ref:`terminology-label` section. Promoting a node to the master role is
625 c71a1a3d Iustin Pop
special, while the other roles are handled all via a single command.
626 c71a1a3d Iustin Pop
627 c71a1a3d Iustin Pop
Failing over the master node
628 c71a1a3d Iustin Pop
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
629 c71a1a3d Iustin Pop
630 c71a1a3d Iustin Pop
If you want to promote a different node to the master role (for whatever
631 c71a1a3d Iustin Pop
reason), run on any other master-candidate node the command::
632 c71a1a3d Iustin Pop
633 c71a1a3d Iustin Pop
  gnt-cluster masterfailover
634 c71a1a3d Iustin Pop
635 c71a1a3d Iustin Pop
and the node you ran it on is now the new master. In case you try to run
636 c71a1a3d Iustin Pop
this on a non master-candidate node, you will get an error telling you
637 c71a1a3d Iustin Pop
which nodes are valid.
638 c71a1a3d Iustin Pop
639 c71a1a3d Iustin Pop
Changing between the other roles
640 c71a1a3d Iustin Pop
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
641 c71a1a3d Iustin Pop
642 c71a1a3d Iustin Pop
The ``gnt-node modify`` command can be used to select a new role::
643 c71a1a3d Iustin Pop
644 c71a1a3d Iustin Pop
  # change to master candidate
645 c71a1a3d Iustin Pop
  gnt-node modify -C yes NODE
646 c71a1a3d Iustin Pop
  # change to drained status
647 c71a1a3d Iustin Pop
  gnt-node modify -D yes NODE
648 c71a1a3d Iustin Pop
  # change to offline status
649 c71a1a3d Iustin Pop
  gnt-node modify -O yes NODE
650 c71a1a3d Iustin Pop
  # change to regular mode (reset all flags)
651 c71a1a3d Iustin Pop
  gnt-node modify -O no -D no -C no NODE
652 c71a1a3d Iustin Pop
653 c71a1a3d Iustin Pop
Note that the cluster requires that at any point in time, a certain
654 c71a1a3d Iustin Pop
number of nodes are master candidates, so changing from master candidate
655 c71a1a3d Iustin Pop
to other roles might fail. It is recommended to either force the
656 c71a1a3d Iustin Pop
operation (via the ``--force`` option) or first change the number of
657 c71a1a3d Iustin Pop
master candidates in the cluster - see :ref:`cluster-config-label`.
658 c71a1a3d Iustin Pop
659 c71a1a3d Iustin Pop
Evacuating nodes
660 c71a1a3d Iustin Pop
++++++++++++++++
661 c71a1a3d Iustin Pop
662 c71a1a3d Iustin Pop
There are two steps of moving instances off a node:
663 c71a1a3d Iustin Pop
664 c71a1a3d Iustin Pop
- moving the primary instances (actually converting them into secondary
665 c71a1a3d Iustin Pop
  instances)
666 c71a1a3d Iustin Pop
- moving the secondary instances (including any instances converted in
667 c71a1a3d Iustin Pop
  the step above)
668 c71a1a3d Iustin Pop
669 c71a1a3d Iustin Pop
Primary instance conversion
670 c71a1a3d Iustin Pop
~~~~~~~~~~~~~~~~~~~~~~~~~~~
671 c71a1a3d Iustin Pop
672 c71a1a3d Iustin Pop
For this step, you can use either individual instance move
673 c71a1a3d Iustin Pop
commands (as seen in :ref:`instance-change-primary-label`) or the bulk
674 c71a1a3d Iustin Pop
per-node versions; these are::
675 c71a1a3d Iustin Pop
676 c71a1a3d Iustin Pop
  gnt-node migrate NODE
677 c71a1a3d Iustin Pop
  gnt-node evacuate NODE
678 c71a1a3d Iustin Pop
679 c71a1a3d Iustin Pop
Note that the instance “move” command doesn't currently have a node
680 c71a1a3d Iustin Pop
equivalent.
681 c71a1a3d Iustin Pop
682 c71a1a3d Iustin Pop
Both these commands, or the equivalent per-instance command, will make
683 c71a1a3d Iustin Pop
this node the secondary node for the respective instances, whereas their
684 c71a1a3d Iustin Pop
current secondary node will become primary. Note that it is not possible
685 c71a1a3d Iustin Pop
to change in one step the primary node to another node as primary, while
686 c71a1a3d Iustin Pop
keeping the same secondary node.
687 c71a1a3d Iustin Pop
688 c71a1a3d Iustin Pop
Secondary instance evacuation
689 c71a1a3d Iustin Pop
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
690 c71a1a3d Iustin Pop
691 c71a1a3d Iustin Pop
For the evacuation of secondary instances, a command called
692 c71a1a3d Iustin Pop
:command:`gnt-node evacuate` is provided and its syntax is::
693 c71a1a3d Iustin Pop
694 c71a1a3d Iustin Pop
  gnt-node evacuate -I IALLOCATOR_SCRIPT NODE
695 c71a1a3d Iustin Pop
  gnt-node evacuate -n DESTINATION_NODE NODE
696 c71a1a3d Iustin Pop
697 c71a1a3d Iustin Pop
The first version will compute the new secondary for each instance in
698 c71a1a3d Iustin Pop
turn using the given iallocator script, whereas the second one will
699 c71a1a3d Iustin Pop
simply move all instances to DESTINATION_NODE.
700 c71a1a3d Iustin Pop
701 c71a1a3d Iustin Pop
Removal
702 c71a1a3d Iustin Pop
+++++++
703 c71a1a3d Iustin Pop
704 c71a1a3d Iustin Pop
Once a node no longer has any instances (neither primary nor secondary),
705 c71a1a3d Iustin Pop
it's easy to remove it from the cluster::
706 c71a1a3d Iustin Pop
707 c71a1a3d Iustin Pop
  gnt-node remove NODE_NAME
708 c71a1a3d Iustin Pop
709 c71a1a3d Iustin Pop
This will deconfigure the node, stop the ganeti daemons on it and leave
710 c71a1a3d Iustin Pop
it hopefully like before it joined to the cluster.
711 c71a1a3d Iustin Pop
712 c71a1a3d Iustin Pop
Storage handling
713 c71a1a3d Iustin Pop
++++++++++++++++
714 c71a1a3d Iustin Pop
715 c71a1a3d Iustin Pop
When using LVM (either standalone or with DRBD), it can become tedious
716 c71a1a3d Iustin Pop
to debug and fix it in case of errors. Furthermore, even file-based
717 c71a1a3d Iustin Pop
storage can become complicated to handle manually on many hosts. Ganeti
718 c71a1a3d Iustin Pop
provides a couple of commands to help with automation.
719 c71a1a3d Iustin Pop
720 c71a1a3d Iustin Pop
Logical volumes
721 c71a1a3d Iustin Pop
~~~~~~~~~~~~~~~
722 c71a1a3d Iustin Pop
723 c71a1a3d Iustin Pop
This is a command specific to LVM handling. It allows listing the
724 c71a1a3d Iustin Pop
logical volumes on a given node or on all nodes and their association to
725 c71a1a3d Iustin Pop
instances via the ``volumes`` command::
726 c71a1a3d Iustin Pop
727 c71a1a3d Iustin Pop
  node1# gnt-node volumes
728 c71a1a3d Iustin Pop
  Node  PhysDev   VG    Name             Size Instance
729 c71a1a3d Iustin Pop
  node1 /dev/sdb1 xenvg e61fbc97-….disk0 512M instance17
730 c71a1a3d Iustin Pop
  node1 /dev/sdb1 xenvg ebd1a7d1-….disk0 512M instance19
731 c71a1a3d Iustin Pop
  node2 /dev/sdb1 xenvg 0af08a3d-….disk0 512M instance20
732 c71a1a3d Iustin Pop
  node2 /dev/sdb1 xenvg cc012285-….disk0 512M instance16
733 c71a1a3d Iustin Pop
  node2 /dev/sdb1 xenvg f0fac192-….disk0 512M instance18
734 c71a1a3d Iustin Pop
735 c71a1a3d Iustin Pop
The above command maps each logical volume to a volume group and
736 c71a1a3d Iustin Pop
underlying physical volume and (possibly) to an instance.
737 c71a1a3d Iustin Pop
738 c71a1a3d Iustin Pop
.. _storage-units-label:
739 c71a1a3d Iustin Pop
740 c71a1a3d Iustin Pop
Generalized storage handling
741 c71a1a3d Iustin Pop
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
742 c71a1a3d Iustin Pop
743 c71a1a3d Iustin Pop
.. versionadded:: 2.1
744 c71a1a3d Iustin Pop
745 c71a1a3d Iustin Pop
Starting with Ganeti 2.1, a new storage framework has been implemented
746 c71a1a3d Iustin Pop
that tries to abstract the handling of the storage type the cluster
747 c71a1a3d Iustin Pop
uses.
748 c71a1a3d Iustin Pop
749 c71a1a3d Iustin Pop
First is listing the backend storage and their space situation::
750 c71a1a3d Iustin Pop
751 c71a1a3d Iustin Pop
  node1# gnt-node list-storage
752 c71a1a3d Iustin Pop
  Node  Name        Size Used   Free
753 c71a1a3d Iustin Pop
  node1 /dev/sda7 673.8G   0M 673.8G
754 c71a1a3d Iustin Pop
  node1 /dev/sdb1 698.6G 1.5G 697.1G
755 c71a1a3d Iustin Pop
  node2 /dev/sda7 673.8G   0M 673.8G
756 c71a1a3d Iustin Pop
  node2 /dev/sdb1 698.6G 1.0G 697.6G
757 c71a1a3d Iustin Pop
758 c71a1a3d Iustin Pop
The default is to list LVM physical volumes. It's also possible to list
759 c71a1a3d Iustin Pop
the LVM volume groups::
760 c71a1a3d Iustin Pop
761 c71a1a3d Iustin Pop
  node1# gnt-node list-storage -t lvm-vg
762 c71a1a3d Iustin Pop
  Node  Name  Size
763 c71a1a3d Iustin Pop
  node1 xenvg 1.3T
764 c71a1a3d Iustin Pop
  node2 xenvg 1.3T
765 c71a1a3d Iustin Pop
766 c71a1a3d Iustin Pop
Next is repairing storage units, which is currently only implemented for
767 c71a1a3d Iustin Pop
volume groups and does the equivalent of ``vgreduce --removemissing``::
768 c71a1a3d Iustin Pop
769 c71a1a3d Iustin Pop
  node1# gnt-node repair-storage node2 lvm-vg xenvg
770 c71a1a3d Iustin Pop
  Sun Oct 25 22:21:45 2009 Repairing storage unit 'xenvg' on node2 ...
771 c71a1a3d Iustin Pop
772 c71a1a3d Iustin Pop
Last is the modification of volume properties, which is (again) only
773 c71a1a3d Iustin Pop
implemented for LVM physical volumes and allows toggling the
774 c71a1a3d Iustin Pop
``allocatable`` value::
775 c71a1a3d Iustin Pop
776 c71a1a3d Iustin Pop
  node1# gnt-node modify-storage --allocatable=no node2 lvm-pv /dev/sdb1
777 c71a1a3d Iustin Pop
778 c71a1a3d Iustin Pop
Use of the storage commands
779 56c9a709 Iustin Pop
~~~~~~~~~~~~~~~~~~~~~~~~~~~
780 56c9a709 Iustin Pop
781 c71a1a3d Iustin Pop
All these commands are needed when recovering a node from a disk
782 c71a1a3d Iustin Pop
failure:
783 c71a1a3d Iustin Pop
784 c71a1a3d Iustin Pop
- first, we need to recover from complete LVM failure (due to missing
785 c71a1a3d Iustin Pop
  disk), by running the ``repair-storage`` command
786 c71a1a3d Iustin Pop
- second, we need to change allocation on any partially-broken disk
787 c71a1a3d Iustin Pop
  (i.e. LVM still sees it, but it has bad blocks) by running
788 c71a1a3d Iustin Pop
  ``modify-storage``
789 c71a1a3d Iustin Pop
- then we can evacuate the instances as needed
790 56c9a709 Iustin Pop
791 56c9a709 Iustin Pop
792 c71a1a3d Iustin Pop
Cluster operations
793 c71a1a3d Iustin Pop
------------------
794 c71a1a3d Iustin Pop
795 c71a1a3d Iustin Pop
Beside the cluster initialisation command (which is detailed in the
796 c71a1a3d Iustin Pop
:doc:`install` document) and the master failover command which is
797 c71a1a3d Iustin Pop
explained under node handling, there are a couple of other cluster
798 c71a1a3d Iustin Pop
operations available.
799 c71a1a3d Iustin Pop
800 c71a1a3d Iustin Pop
.. _cluster-config-label:
801 c71a1a3d Iustin Pop
802 c71a1a3d Iustin Pop
Standard operations
803 c71a1a3d Iustin Pop
+++++++++++++++++++
804 c71a1a3d Iustin Pop
805 c71a1a3d Iustin Pop
One of the few commands that can be run on any node (not only the
806 c71a1a3d Iustin Pop
master) is the ``getmaster`` command::
807 c71a1a3d Iustin Pop
808 c71a1a3d Iustin Pop
  node2# gnt-cluster getmaster
809 c71a1a3d Iustin Pop
  node1.example.com
810 c71a1a3d Iustin Pop
  node2#
811 c71a1a3d Iustin Pop
812 c71a1a3d Iustin Pop
It is possible to query and change global cluster parameters via the
813 c71a1a3d Iustin Pop
``info`` and ``modify`` commands::
814 c71a1a3d Iustin Pop
815 c71a1a3d Iustin Pop
  node1# gnt-cluster info
816 c71a1a3d Iustin Pop
  Cluster name: cluster.example.com
817 c71a1a3d Iustin Pop
  Cluster UUID: 07805e6f-f0af-4310-95f1-572862ee939c
818 c71a1a3d Iustin Pop
  Creation time: 2009-09-25 05:04:15
819 c71a1a3d Iustin Pop
  Modification time: 2009-10-18 22:11:47
820 c71a1a3d Iustin Pop
  Master node: node1.example.com
821 c71a1a3d Iustin Pop
  Architecture (this node): 64bit (x86_64)
822 c71a1a3d Iustin Pop
823 c71a1a3d Iustin Pop
  Tags: foo
824 c71a1a3d Iustin Pop
  Default hypervisor: xen-pvm
825 c71a1a3d Iustin Pop
  Enabled hypervisors: xen-pvm
826 c71a1a3d Iustin Pop
  Hypervisor parameters:
827 c71a1a3d Iustin Pop
    - xen-pvm:
828 c71a1a3d Iustin Pop
        root_path: /dev/sda1
829 c71a1a3d Iustin Pop
830 c71a1a3d Iustin Pop
  Cluster parameters:
831 c71a1a3d Iustin Pop
    - candidate pool size: 10
832 c71a1a3d Iustin Pop
833 c71a1a3d Iustin Pop
  Default instance parameters:
834 c71a1a3d Iustin Pop
    - default:
835 c71a1a3d Iustin Pop
        memory: 128
836 c71a1a3d Iustin Pop
837 c71a1a3d Iustin Pop
  Default nic parameters:
838 c71a1a3d Iustin Pop
    - default:
839 c71a1a3d Iustin Pop
        link: xen-br0
840 c71a1a3d Iustin Pop
841 c71a1a3d Iustin Pop
842 c71a1a3d Iustin Pop
There various parameters above can be changed via the ``modify``
843 c71a1a3d Iustin Pop
commands as follows:
844 c71a1a3d Iustin Pop
845 c71a1a3d Iustin Pop
- the hypervisor parameters can be changed via ``modify -H
846 c71a1a3d Iustin Pop
  xen-pvm:root_path=…``, and so on for other hypervisors/key/values
847 c71a1a3d Iustin Pop
- the "default instance parameters" are changeable via ``modify -B
848 c71a1a3d Iustin Pop
  parameter=value…`` syntax
849 c71a1a3d Iustin Pop
- the cluster parameters are changeable via separate options to the
850 c71a1a3d Iustin Pop
  modify command (e.g. ``--candidate-pool-size``, etc.)
851 c71a1a3d Iustin Pop
852 c71a1a3d Iustin Pop
For detailed option list see the :manpage:`gnt-cluster(8)` man page.
853 c71a1a3d Iustin Pop
854 c71a1a3d Iustin Pop
The cluster version can be obtained via the ``version`` command::
855 c71a1a3d Iustin Pop
  node1# gnt-cluster version
856 c71a1a3d Iustin Pop
  Software version: 2.1.0
857 c71a1a3d Iustin Pop
  Internode protocol: 20
858 c71a1a3d Iustin Pop
  Configuration format: 2010000
859 c71a1a3d Iustin Pop
  OS api version: 15
860 c71a1a3d Iustin Pop
  Export interface: 0
861 c71a1a3d Iustin Pop
862 c71a1a3d Iustin Pop
This is not very useful except when debugging Ganeti.
863 c71a1a3d Iustin Pop
864 c71a1a3d Iustin Pop
Global node commands
865 c71a1a3d Iustin Pop
++++++++++++++++++++
866 c71a1a3d Iustin Pop
867 c71a1a3d Iustin Pop
There are two commands provided for replicating files to all nodes of a
868 c71a1a3d Iustin Pop
cluster and for running commands on all the nodes::
869 c71a1a3d Iustin Pop
870 c71a1a3d Iustin Pop
  node1# gnt-cluster copyfile /path/to/file
871 c71a1a3d Iustin Pop
  node1# gnt-cluster command ls -l /path/to/file
872 c71a1a3d Iustin Pop
873 c71a1a3d Iustin Pop
These are simple wrappers over scp/ssh and more advanced usage can be
874 c71a1a3d Iustin Pop
obtained using :manpage:`dsh(1)` and similar commands. But they are
875 c71a1a3d Iustin Pop
useful to update an OS script from the master node, for example.
876 c71a1a3d Iustin Pop
877 c71a1a3d Iustin Pop
Cluster verification
878 c71a1a3d Iustin Pop
++++++++++++++++++++
879 c71a1a3d Iustin Pop
880 c71a1a3d Iustin Pop
There are three commands that relate to global cluster checks. The first
881 c71a1a3d Iustin Pop
one is ``verify`` which gives an overview on the cluster state,
882 c71a1a3d Iustin Pop
highlighting any issues. In normal operation, this command should return
883 c71a1a3d Iustin Pop
no ``ERROR`` messages::
884 c71a1a3d Iustin Pop
885 c71a1a3d Iustin Pop
  node1# gnt-cluster verify
886 c71a1a3d Iustin Pop
  Sun Oct 25 23:08:58 2009 * Verifying global settings
887 c71a1a3d Iustin Pop
  Sun Oct 25 23:08:58 2009 * Gathering data (2 nodes)
888 c71a1a3d Iustin Pop
  Sun Oct 25 23:09:00 2009 * Verifying node status
889 c71a1a3d Iustin Pop
  Sun Oct 25 23:09:00 2009 * Verifying instance status
890 c71a1a3d Iustin Pop
  Sun Oct 25 23:09:00 2009 * Verifying orphan volumes
891 c71a1a3d Iustin Pop
  Sun Oct 25 23:09:00 2009 * Verifying remaining instances
892 c71a1a3d Iustin Pop
  Sun Oct 25 23:09:00 2009 * Verifying N+1 Memory redundancy
893 c71a1a3d Iustin Pop
  Sun Oct 25 23:09:00 2009 * Other Notes
894 c71a1a3d Iustin Pop
  Sun Oct 25 23:09:00 2009   - NOTICE: 5 non-redundant instance(s) found.
895 c71a1a3d Iustin Pop
  Sun Oct 25 23:09:00 2009 * Hooks Results
896 c71a1a3d Iustin Pop
897 c71a1a3d Iustin Pop
The second command is ``verify-disks``, which checks that the instance's
898 c71a1a3d Iustin Pop
disks have the correct status based on the desired instance state
899 c71a1a3d Iustin Pop
(up/down)::
900 c71a1a3d Iustin Pop
901 c71a1a3d Iustin Pop
  node1# gnt-cluster verify-disks
902 c71a1a3d Iustin Pop
903 c71a1a3d Iustin Pop
Note that this command will show no output when disks are healthy.
904 c71a1a3d Iustin Pop
905 c71a1a3d Iustin Pop
The last command is used to repair any discrepancies in Ganeti's
906 c71a1a3d Iustin Pop
recorded disk size and the actual disk size (disk size information is
907 c71a1a3d Iustin Pop
needed for proper activation and growth of DRBD-based disks)::
908 c71a1a3d Iustin Pop
909 c71a1a3d Iustin Pop
  node1# gnt-cluster repair-disk-sizes
910 c71a1a3d Iustin Pop
  Sun Oct 25 23:13:16 2009  - INFO: Disk 0 of instance instance1 has mismatched size, correcting: recorded 512, actual 2048
911 c71a1a3d Iustin Pop
  Sun Oct 25 23:13:17 2009  - WARNING: Invalid result from node node4, ignoring node results
912 c71a1a3d Iustin Pop
913 c71a1a3d Iustin Pop
The above shows one instance having wrong disk size, and a node which
914 c71a1a3d Iustin Pop
returned invalid data, and thus we ignored all primary instances of that
915 c71a1a3d Iustin Pop
node.
916 c71a1a3d Iustin Pop
917 c71a1a3d Iustin Pop
Configuration redistribution
918 c71a1a3d Iustin Pop
++++++++++++++++++++++++++++
919 c71a1a3d Iustin Pop
920 c71a1a3d Iustin Pop
If the verify command complains about file mismatches between the master
921 c71a1a3d Iustin Pop
and other nodes, due to some node problems or if you manually modified
922 c71a1a3d Iustin Pop
configuration files, you can force an push of the master configuration
923 c71a1a3d Iustin Pop
to all other nodes via the ``redist-conf`` command::
924 c71a1a3d Iustin Pop
925 c71a1a3d Iustin Pop
  node1# gnt-cluster redist-conf
926 c71a1a3d Iustin Pop
  node1#
927 c71a1a3d Iustin Pop
928 c71a1a3d Iustin Pop
This command will be silent unless there are problems sending updates to
929 c71a1a3d Iustin Pop
the other nodes.
930 c71a1a3d Iustin Pop
931 c71a1a3d Iustin Pop
932 c71a1a3d Iustin Pop
Cluster renaming
933 c71a1a3d Iustin Pop
++++++++++++++++
934 c71a1a3d Iustin Pop
935 c71a1a3d Iustin Pop
It is possible to rename a cluster, or to change its IP address, via the
936 c71a1a3d Iustin Pop
``rename`` command. If only the IP has changed, you need to pass the
937 c71a1a3d Iustin Pop
current name and Ganeti will realise its IP has changed::
938 c71a1a3d Iustin Pop
939 c71a1a3d Iustin Pop
  node1# gnt-cluster rename cluster.example.com
940 c71a1a3d Iustin Pop
  This will rename the cluster to 'cluster.example.com'. If
941 c71a1a3d Iustin Pop
  you are connected over the network to the cluster name, the operation
942 c71a1a3d Iustin Pop
  is very dangerous as the IP address will be removed from the node and
943 c71a1a3d Iustin Pop
  the change may not go through. Continue?
944 c71a1a3d Iustin Pop
  y/[n]/?: y
945 c71a1a3d Iustin Pop
  Failure: prerequisites not met for this operation:
946 c71a1a3d Iustin Pop
  Neither the name nor the IP address of the cluster has changed
947 c71a1a3d Iustin Pop
948 c71a1a3d Iustin Pop
In the above output, neither value has changed since the cluster
949 c71a1a3d Iustin Pop
initialisation so the operation is not completed.
950 c71a1a3d Iustin Pop
951 c71a1a3d Iustin Pop
Queue operations
952 c71a1a3d Iustin Pop
++++++++++++++++
953 c71a1a3d Iustin Pop
954 c71a1a3d Iustin Pop
The job queue execution in Ganeti 2.0 and higher can be inspected,
955 c71a1a3d Iustin Pop
suspended and resumed via the ``queue`` command::
956 c71a1a3d Iustin Pop
957 c71a1a3d Iustin Pop
  node1~# gnt-cluster queue info
958 c71a1a3d Iustin Pop
  The drain flag is unset
959 c71a1a3d Iustin Pop
  node1~# gnt-cluster queue drain
960 c71a1a3d Iustin Pop
  node1~# gnt-instance stop instance1
961 c71a1a3d Iustin Pop
  Failed to submit job for instance1: Job queue is drained, refusing job
962 c71a1a3d Iustin Pop
  node1~# gnt-cluster queue info
963 c71a1a3d Iustin Pop
  The drain flag is set
964 c71a1a3d Iustin Pop
  node1~# gnt-cluster queue undrain
965 c71a1a3d Iustin Pop
966 c71a1a3d Iustin Pop
This is most useful if you have an active cluster and you need to
967 c71a1a3d Iustin Pop
upgrade the Ganeti software, or simply restart the software on any node:
968 c71a1a3d Iustin Pop
969 c71a1a3d Iustin Pop
#. suspend the queue via ``queue drain``
970 c71a1a3d Iustin Pop
#. wait until there are no more running jobs via ``gnt-job list``
971 c71a1a3d Iustin Pop
#. restart the master or another node, or upgrade the software
972 c71a1a3d Iustin Pop
#. resume the queue via ``queue undrain``
973 c71a1a3d Iustin Pop
974 c71a1a3d Iustin Pop
.. note:: this command only stores a local flag file, and if you
975 c71a1a3d Iustin Pop
   failover the master, it will not have effect on the new master.
976 c71a1a3d Iustin Pop
977 c71a1a3d Iustin Pop
978 c71a1a3d Iustin Pop
Watcher control
979 c71a1a3d Iustin Pop
+++++++++++++++
980 c71a1a3d Iustin Pop
981 c71a1a3d Iustin Pop
The :manpage:`ganeti-watcher` is a program, usually scheduled via
982 c71a1a3d Iustin Pop
``cron``, that takes care of cluster maintenance operations (restarting
983 c71a1a3d Iustin Pop
downed instances, activating down DRBD disks, etc.). However, during
984 c71a1a3d Iustin Pop
maintenance and troubleshooting, this can get in your way; disabling it
985 c71a1a3d Iustin Pop
via commenting out the cron job is not so good as this can be
986 c71a1a3d Iustin Pop
forgotten. Thus there are some commands for automated control of the
987 c71a1a3d Iustin Pop
watcher: ``pause``, ``info`` and ``continue``::
988 c71a1a3d Iustin Pop
989 c71a1a3d Iustin Pop
  node1~# gnt-cluster watcher info
990 c71a1a3d Iustin Pop
  The watcher is not paused.
991 c71a1a3d Iustin Pop
  node1~# gnt-cluster watcher pause 1h
992 c71a1a3d Iustin Pop
  The watcher is paused until Mon Oct 26 00:30:37 2009.
993 c71a1a3d Iustin Pop
  node1~# gnt-cluster watcher info
994 c71a1a3d Iustin Pop
  The watcher is paused until Mon Oct 26 00:30:37 2009.
995 c71a1a3d Iustin Pop
  node1~# ganeti-watcher -d
996 c71a1a3d Iustin Pop
  2009-10-25 23:30:47,984:  pid=28867 ganeti-watcher:486 DEBUG Pause has been set, exiting
997 c71a1a3d Iustin Pop
  node1~# gnt-cluster watcher continue
998 c71a1a3d Iustin Pop
  The watcher is no longer paused.
999 c71a1a3d Iustin Pop
  node1~# ganeti-watcher -d
1000 c71a1a3d Iustin Pop
  2009-10-25 23:31:04,789:  pid=28976 ganeti-watcher:345 DEBUG Archived 0 jobs, left 0
1001 c71a1a3d Iustin Pop
  2009-10-25 23:31:05,884:  pid=28976 ganeti-watcher:280 DEBUG Got data from cluster, writing instance status file
1002 c71a1a3d Iustin Pop
  2009-10-25 23:31:06,061:  pid=28976 ganeti-watcher:150 DEBUG Data didn't change, just touching status file
1003 c71a1a3d Iustin Pop
  node1~# gnt-cluster watcher info
1004 c71a1a3d Iustin Pop
  The watcher is not paused.
1005 c71a1a3d Iustin Pop
  node1~#
1006 c71a1a3d Iustin Pop
1007 c71a1a3d Iustin Pop
The exact details of the argument to the ``pause`` command are available
1008 c71a1a3d Iustin Pop
in the manpage.
1009 c71a1a3d Iustin Pop
1010 c71a1a3d Iustin Pop
.. note:: this command only stores a local flag file, and if you
1011 c71a1a3d Iustin Pop
   failover the master, it will not have effect on the new master.
1012 c71a1a3d Iustin Pop
1013 c71a1a3d Iustin Pop
Removing a cluster entirely
1014 c71a1a3d Iustin Pop
+++++++++++++++++++++++++++
1015 c71a1a3d Iustin Pop
1016 c71a1a3d Iustin Pop
The usual method to cleanup a cluster is to run ``gnt-cluster destroy``
1017 c71a1a3d Iustin Pop
however if the Ganeti installation is broken in any way then this will
1018 c71a1a3d Iustin Pop
not run.
1019 c71a1a3d Iustin Pop
1020 c71a1a3d Iustin Pop
It is possible in such a case to cleanup manually most if not all traces
1021 c71a1a3d Iustin Pop
of a cluster installation by following these steps on all of the nodes:
1022 c71a1a3d Iustin Pop
1023 c71a1a3d Iustin Pop
1. Shutdown all instances. This depends on the virtualisation method
1024 c71a1a3d Iustin Pop
   used (Xen, KVM, etc.):
1025 56c9a709 Iustin Pop
1026 56c9a709 Iustin Pop
  - Xen: run ``xm list`` and ``xm destroy`` on all the non-Domain-0
1027 56c9a709 Iustin Pop
    instances
1028 56c9a709 Iustin Pop
  - KVM: kill all the KVM processes
1029 56c9a709 Iustin Pop
  - chroot: kill all processes under the chroot mountpoints
1030 56c9a709 Iustin Pop
1031 c71a1a3d Iustin Pop
2. If using DRBD, shutdown all DRBD minors (which should by at this time
1032 c71a1a3d Iustin Pop
   no-longer in use by instances); on each node, run ``drbdsetup
1033 56c9a709 Iustin Pop
   /dev/drbdN down`` for each active DRBD minor.
1034 56c9a709 Iustin Pop
1035 c71a1a3d Iustin Pop
3. If using LVM, cleanup the Ganeti volume group; if only Ganeti created
1036 c71a1a3d Iustin Pop
   logical volumes (and you are not sharing the volume group with the
1037 c71a1a3d Iustin Pop
   OS, for example), then simply running ``lvremove -f xenvg`` (replace
1038 c71a1a3d Iustin Pop
   'xenvg' with your volume group name) should do the required cleanup.
1039 56c9a709 Iustin Pop
1040 56c9a709 Iustin Pop
4. If using file-based storage, remove recursively all files and
1041 56c9a709 Iustin Pop
   directories under your file-storage directory: ``rm -rf
1042 c71a1a3d Iustin Pop
   /srv/ganeti/file-storage/*`` replacing the path with the correct path
1043 c71a1a3d Iustin Pop
   for your cluster.
1044 56c9a709 Iustin Pop
1045 56c9a709 Iustin Pop
5. Stop the ganeti daemons (``/etc/init.d/ganeti stop``) and kill any
1046 56c9a709 Iustin Pop
   that remain alive (``pgrep ganeti`` and ``pkill ganeti``).
1047 56c9a709 Iustin Pop
1048 56c9a709 Iustin Pop
6. Remove the ganeti state directory (``rm -rf /var/lib/ganeti/*``),
1049 56c9a709 Iustin Pop
   replacing the path with the correct path for your installation.
1050 56c9a709 Iustin Pop
1051 56c9a709 Iustin Pop
On the master node, remove the cluster from the master-netdev (usually
1052 c71a1a3d Iustin Pop
``xen-br0`` for bridged mode, otherwise ``eth0`` or similar), by running
1053 c71a1a3d Iustin Pop
``ip a del $clusterip/32 dev xen-br0`` (use the correct cluster ip and
1054 c71a1a3d Iustin Pop
network device name).
1055 56c9a709 Iustin Pop
1056 56c9a709 Iustin Pop
At this point, the machines are ready for a cluster creation; in case
1057 c71a1a3d Iustin Pop
you want to remove Ganeti completely, you need to also undo some of the
1058 c71a1a3d Iustin Pop
SSH changes and log directories:
1059 56c9a709 Iustin Pop
1060 7faf5110 Michael Hanselmann
- ``rm -rf /var/log/ganeti /srv/ganeti`` (replace with the correct
1061 7faf5110 Michael Hanselmann
  paths)
1062 c71a1a3d Iustin Pop
- remove from ``/root/.ssh`` the keys that Ganeti added (check the
1063 c71a1a3d Iustin Pop
  ``authorized_keys`` and ``id_dsa`` files)
1064 56c9a709 Iustin Pop
- regenerate the host's SSH keys (check the OpenSSH startup scripts)
1065 56c9a709 Iustin Pop
- uninstall Ganeti
1066 56c9a709 Iustin Pop
1067 56c9a709 Iustin Pop
Otherwise, if you plan to re-create the cluster, you can just go ahead
1068 56c9a709 Iustin Pop
and rerun ``gnt-cluster init``.
1069 558fd122 Michael Hanselmann
1070 c71a1a3d Iustin Pop
Tags handling
1071 c71a1a3d Iustin Pop
-------------
1072 c71a1a3d Iustin Pop
1073 c71a1a3d Iustin Pop
The tags handling (addition, removal, listing) is similar for all the
1074 c71a1a3d Iustin Pop
objects that support it (instances, nodes, and the cluster).
1075 c71a1a3d Iustin Pop
1076 c71a1a3d Iustin Pop
Limitations
1077 c71a1a3d Iustin Pop
+++++++++++
1078 c71a1a3d Iustin Pop
1079 c71a1a3d Iustin Pop
Note that the set of characters present in a tag and the maximum tag
1080 c71a1a3d Iustin Pop
length are restricted. Currently the maximum length is 128 characters,
1081 c71a1a3d Iustin Pop
there can be at most 4096 tags per object, and the set of characters is
1082 c71a1a3d Iustin Pop
comprised by alphanumeric characters and additionally ``.+*/:-``.
1083 c71a1a3d Iustin Pop
1084 c71a1a3d Iustin Pop
Operations
1085 c71a1a3d Iustin Pop
++++++++++
1086 c71a1a3d Iustin Pop
1087 c71a1a3d Iustin Pop
Tags can be added via ``add-tags``::
1088 c71a1a3d Iustin Pop
1089 c71a1a3d Iustin Pop
  gnt-instance add-tags INSTANCE a b c
1090 c71a1a3d Iustin Pop
  gnt-node add-tags INSTANCE a b c
1091 c71a1a3d Iustin Pop
  gnt-cluster add-tags a b c
1092 c71a1a3d Iustin Pop
1093 c71a1a3d Iustin Pop
1094 c71a1a3d Iustin Pop
The above commands add three tags to an instance, to a node and to the
1095 c71a1a3d Iustin Pop
cluster. Note that the cluster command only takes tags as arguments,
1096 c71a1a3d Iustin Pop
whereas the node and instance commands first required the node and
1097 c71a1a3d Iustin Pop
instance name.
1098 c71a1a3d Iustin Pop
1099 c71a1a3d Iustin Pop
Tags can also be added from a file, via the ``--from=FILENAME``
1100 c71a1a3d Iustin Pop
argument. The file is expected to contain one tag per line.
1101 c71a1a3d Iustin Pop
1102 c71a1a3d Iustin Pop
Tags can also be remove via a syntax very similar to the add one::
1103 c71a1a3d Iustin Pop
1104 c71a1a3d Iustin Pop
  gnt-instance remove-tags INSTANCE a b c
1105 c71a1a3d Iustin Pop
1106 c71a1a3d Iustin Pop
And listed via::
1107 c71a1a3d Iustin Pop
1108 c71a1a3d Iustin Pop
  gnt-instance list-tags
1109 c71a1a3d Iustin Pop
  gnt-node list-tags
1110 c71a1a3d Iustin Pop
  gnt-cluster list-tags
1111 c71a1a3d Iustin Pop
1112 c71a1a3d Iustin Pop
Global tag search
1113 c71a1a3d Iustin Pop
+++++++++++++++++
1114 c71a1a3d Iustin Pop
1115 c71a1a3d Iustin Pop
It is also possible to execute a global search on the all tags defined
1116 c71a1a3d Iustin Pop
in the cluster configuration, via a cluster command::
1117 c71a1a3d Iustin Pop
1118 c71a1a3d Iustin Pop
  gnt-cluster search-tags REGEXP
1119 c71a1a3d Iustin Pop
1120 c71a1a3d Iustin Pop
The parameter expected is a regular expression (see
1121 c71a1a3d Iustin Pop
:manpage:`regex(7)`). This will return all tags that match the search,
1122 c71a1a3d Iustin Pop
together with the object they are defined in (the names being show in a
1123 c71a1a3d Iustin Pop
hierarchical kind of way)::
1124 c71a1a3d Iustin Pop
1125 c71a1a3d Iustin Pop
  node1# gnt-cluster search-tags o
1126 c71a1a3d Iustin Pop
  /cluster foo
1127 c71a1a3d Iustin Pop
  /instances/instance1 owner:bar
1128 c71a1a3d Iustin Pop
1129 c71a1a3d Iustin Pop
1130 c71a1a3d Iustin Pop
Job operations
1131 c71a1a3d Iustin Pop
--------------
1132 c71a1a3d Iustin Pop
1133 c71a1a3d Iustin Pop
The various jobs submitted by the instance/node/cluster commands can be
1134 c71a1a3d Iustin Pop
examined, canceled and archived by various invocations of the
1135 c71a1a3d Iustin Pop
``gnt-job`` command.
1136 c71a1a3d Iustin Pop
1137 c71a1a3d Iustin Pop
First is the job list command::
1138 c71a1a3d Iustin Pop
1139 c71a1a3d Iustin Pop
  node1# gnt-job list
1140 c71a1a3d Iustin Pop
  17771 success INSTANCE_QUERY_DATA
1141 c71a1a3d Iustin Pop
  17773 success CLUSTER_VERIFY_DISKS
1142 c71a1a3d Iustin Pop
  17775 success CLUSTER_REPAIR_DISK_SIZES
1143 c71a1a3d Iustin Pop
  17776 error   CLUSTER_RENAME(cluster.example.com)
1144 c71a1a3d Iustin Pop
  17780 success CLUSTER_REDIST_CONF
1145 c71a1a3d Iustin Pop
  17792 success INSTANCE_REBOOT(instance1.example.com)
1146 c71a1a3d Iustin Pop
1147 c71a1a3d Iustin Pop
More detailed information about a job can be found via the ``info``
1148 c71a1a3d Iustin Pop
command::
1149 c71a1a3d Iustin Pop
1150 c71a1a3d Iustin Pop
  node1# gnt-job info 17776
1151 c71a1a3d Iustin Pop
  Job ID: 17776
1152 c71a1a3d Iustin Pop
    Status: error
1153 c71a1a3d Iustin Pop
    Received:         2009-10-25 23:18:02.180569
1154 c71a1a3d Iustin Pop
    Processing start: 2009-10-25 23:18:02.200335 (delta 0.019766s)
1155 c71a1a3d Iustin Pop
    Processing end:   2009-10-25 23:18:02.279743 (delta 0.079408s)
1156 c71a1a3d Iustin Pop
    Total processing time: 0.099174 seconds
1157 c71a1a3d Iustin Pop
    Opcodes:
1158 c71a1a3d Iustin Pop
      OP_CLUSTER_RENAME
1159 c71a1a3d Iustin Pop
        Status: error
1160 c71a1a3d Iustin Pop
        Processing start: 2009-10-25 23:18:02.200335
1161 c71a1a3d Iustin Pop
        Processing end:   2009-10-25 23:18:02.252282
1162 c71a1a3d Iustin Pop
        Input fields:
1163 c71a1a3d Iustin Pop
          name: cluster.example.com
1164 c71a1a3d Iustin Pop
        Result:
1165 c71a1a3d Iustin Pop
          OpPrereqError
1166 c71a1a3d Iustin Pop
          [Neither the name nor the IP address of the cluster has changed]
1167 c71a1a3d Iustin Pop
        Execution log:
1168 c71a1a3d Iustin Pop
1169 c71a1a3d Iustin Pop
During the execution of a job, it's possible to follow the output of a
1170 c71a1a3d Iustin Pop
job, similar to the log that one get from the ``gnt-`` commands, via the
1171 c71a1a3d Iustin Pop
watch command::
1172 c71a1a3d Iustin Pop
1173 c71a1a3d Iustin Pop
  node1# gnt-instance add --submit … instance1
1174 c71a1a3d Iustin Pop
  JobID: 17818
1175 c71a1a3d Iustin Pop
  node1# gnt-job watch 17818
1176 c71a1a3d Iustin Pop
  Output from job 17818 follows
1177 c71a1a3d Iustin Pop
  -----------------------------
1178 c71a1a3d Iustin Pop
  Mon Oct 26 00:22:48 2009  - INFO: Selected nodes for instance instance1 via iallocator dumb: node1, node2
1179 c71a1a3d Iustin Pop
  Mon Oct 26 00:22:49 2009 * creating instance disks...
1180 c71a1a3d Iustin Pop
  Mon Oct 26 00:22:52 2009 adding instance instance1 to cluster config
1181 c71a1a3d Iustin Pop
  Mon Oct 26 00:22:52 2009  - INFO: Waiting for instance instance1 to sync disks.
1182 c71a1a3d Iustin Pop
1183 c71a1a3d Iustin Pop
  Mon Oct 26 00:23:03 2009 creating os for instance xen-devi-18.fra.corp.google.com on node mpgntac4.fra.corp.google.com
1184 c71a1a3d Iustin Pop
  Mon Oct 26 00:23:03 2009 * running the instance OS create scripts...
1185 c71a1a3d Iustin Pop
  Mon Oct 26 00:23:13 2009 * starting instance...
1186 c71a1a3d Iustin Pop
  node1#
1187 c71a1a3d Iustin Pop
1188 c71a1a3d Iustin Pop
This is useful if you need to follow a job's progress from multiple
1189 c71a1a3d Iustin Pop
terminals.
1190 c71a1a3d Iustin Pop
1191 c71a1a3d Iustin Pop
A job that has not yet started to run can be canceled::
1192 c71a1a3d Iustin Pop
1193 c71a1a3d Iustin Pop
  node1# gnt-job cancel 17810
1194 c71a1a3d Iustin Pop
1195 c71a1a3d Iustin Pop
But not one that has already started execution::
1196 c71a1a3d Iustin Pop
1197 c71a1a3d Iustin Pop
  node1# gnt-job cancel 17805
1198 c71a1a3d Iustin Pop
  Job 17805 is no longer waiting in the queue
1199 c71a1a3d Iustin Pop
1200 c71a1a3d Iustin Pop
There are two queues for jobs: the *current* and the *archive*
1201 c71a1a3d Iustin Pop
queue. Jobs are initially submitted to the current queue, and they stay
1202 c71a1a3d Iustin Pop
in that queue until they have finished execution (either successfully or
1203 c71a1a3d Iustin Pop
not). At that point, they can be moved into the archive queue, and the
1204 c71a1a3d Iustin Pop
ganeti-watcher script will do this automatically after 6 hours. The
1205 c71a1a3d Iustin Pop
ganeti-cleaner script will remove the jobs from the archive directory
1206 c71a1a3d Iustin Pop
after three weeks.
1207 c71a1a3d Iustin Pop
1208 c71a1a3d Iustin Pop
Note that only jobs in the current queue can be viewed via the list and
1209 c71a1a3d Iustin Pop
info commands; Ganeti itself doesn't examine the archive directory. If
1210 c71a1a3d Iustin Pop
you need to see an older job, either move the file manually in the
1211 c71a1a3d Iustin Pop
top-level queue directory, or look at its contents (it's a
1212 c71a1a3d Iustin Pop
JSON-formatted file).
1213 c71a1a3d Iustin Pop
1214 c71a1a3d Iustin Pop
Ganeti tools
1215 c71a1a3d Iustin Pop
------------
1216 c71a1a3d Iustin Pop
1217 c71a1a3d Iustin Pop
Beside the usual ``gnt-`` and ``ganeti-`` commands which are provided
1218 c71a1a3d Iustin Pop
and installed in ``$prefix/sbin`` at install time, there are a couple of
1219 c71a1a3d Iustin Pop
other tools installed which are used seldom but can be helpful in some
1220 c71a1a3d Iustin Pop
cases.
1221 c71a1a3d Iustin Pop
1222 c71a1a3d Iustin Pop
lvmstrap
1223 c71a1a3d Iustin Pop
++++++++
1224 c71a1a3d Iustin Pop
1225 c71a1a3d Iustin Pop
The ``lvmstrap`` tool, introduced in :ref:`configure-lvm-label` section,
1226 c71a1a3d Iustin Pop
has two modes of operation:
1227 c71a1a3d Iustin Pop
1228 c71a1a3d Iustin Pop
- ``diskinfo`` shows the discovered disks on the system and their status
1229 c71a1a3d Iustin Pop
- ``create`` takes all not-in-use disks and creates a volume group out
1230 c71a1a3d Iustin Pop
  of them
1231 c71a1a3d Iustin Pop
1232 c71a1a3d Iustin Pop
.. warning:: The ``create`` argument to this command causes data-loss!
1233 c71a1a3d Iustin Pop
1234 c71a1a3d Iustin Pop
cfgupgrade
1235 c71a1a3d Iustin Pop
++++++++++
1236 c71a1a3d Iustin Pop
1237 c71a1a3d Iustin Pop
The ``cfgupgrade`` tools is used to upgrade between major (and minor)
1238 c71a1a3d Iustin Pop
Ganeti versions. Point-releases are usually transparent for the admin.
1239 c71a1a3d Iustin Pop
1240 c71a1a3d Iustin Pop
More information about the upgrade procedure is listed on the wiki at
1241 c71a1a3d Iustin Pop
http://code.google.com/p/ganeti/wiki/UpgradeNotes.
1242 c71a1a3d Iustin Pop
1243 c71a1a3d Iustin Pop
cfgshell
1244 c71a1a3d Iustin Pop
++++++++
1245 c71a1a3d Iustin Pop
1246 c71a1a3d Iustin Pop
.. note:: This command is not actively maintained; make sure you backup
1247 c71a1a3d Iustin Pop
   your configuration before using it
1248 c71a1a3d Iustin Pop
1249 c71a1a3d Iustin Pop
This can be used as an alternative to direct editing of the
1250 c71a1a3d Iustin Pop
main configuration file if Ganeti has a bug and prevents you, for
1251 c71a1a3d Iustin Pop
example, from removing an instance or a node from the configuration
1252 c71a1a3d Iustin Pop
file.
1253 c71a1a3d Iustin Pop
1254 c71a1a3d Iustin Pop
.. _burnin-label:
1255 c71a1a3d Iustin Pop
1256 c71a1a3d Iustin Pop
burnin
1257 c71a1a3d Iustin Pop
++++++
1258 c71a1a3d Iustin Pop
1259 c71a1a3d Iustin Pop
.. warning:: This command will erase existing instances if given as
1260 c71a1a3d Iustin Pop
   arguments!
1261 c71a1a3d Iustin Pop
1262 c71a1a3d Iustin Pop
This tool is used to exercise either the hardware of machines or
1263 c71a1a3d Iustin Pop
alternatively the Ganeti software. It is safe to run on an existing
1264 c71a1a3d Iustin Pop
cluster **as long as you don't pass it existing instance names**.
1265 c71a1a3d Iustin Pop
1266 c71a1a3d Iustin Pop
The command will, by default, execute a comprehensive set of operations
1267 c71a1a3d Iustin Pop
against a list of instances, these being:
1268 c71a1a3d Iustin Pop
1269 c71a1a3d Iustin Pop
- creation
1270 c71a1a3d Iustin Pop
- disk replacement (for redundant instances)
1271 c71a1a3d Iustin Pop
- failover and migration (for redundant instances)
1272 c71a1a3d Iustin Pop
- move (for non-redundant instances)
1273 c71a1a3d Iustin Pop
- disk growth
1274 c71a1a3d Iustin Pop
- add disks, remove disk
1275 c71a1a3d Iustin Pop
- add NICs, remove NICs
1276 c71a1a3d Iustin Pop
- export and then import
1277 c71a1a3d Iustin Pop
- rename
1278 c71a1a3d Iustin Pop
- reboot
1279 c71a1a3d Iustin Pop
- shutdown/startup
1280 c71a1a3d Iustin Pop
- and finally removal of the test instances
1281 c71a1a3d Iustin Pop
1282 c71a1a3d Iustin Pop
Executing all these operations will test that the hardware performs
1283 c71a1a3d Iustin Pop
well: the creation, disk replace, disk add and disk growth will exercise
1284 c71a1a3d Iustin Pop
the storage and network; the migrate command will test the memory of the
1285 c71a1a3d Iustin Pop
systems. Depending on the passed options, it can also test that the
1286 c71a1a3d Iustin Pop
instance OS definitions are executing properly the rename, import and
1287 c71a1a3d Iustin Pop
export operations.
1288 c71a1a3d Iustin Pop
1289 c71a1a3d Iustin Pop
Other Ganeti projects
1290 c71a1a3d Iustin Pop
---------------------
1291 c71a1a3d Iustin Pop
1292 c71a1a3d Iustin Pop
There are two other Ganeti-related projects that can be useful in a
1293 c71a1a3d Iustin Pop
Ganeti deployment. These can be downloaded from the project site
1294 c71a1a3d Iustin Pop
(http://code.google.com/p/ganeti/) and the repositories are also on the
1295 c71a1a3d Iustin Pop
project git site (http://git.ganeti.org).
1296 c71a1a3d Iustin Pop
1297 c71a1a3d Iustin Pop
NBMA tools
1298 c71a1a3d Iustin Pop
++++++++++
1299 c71a1a3d Iustin Pop
1300 c71a1a3d Iustin Pop
The ``ganeti-nbma`` software is designed to allow instances to live on a
1301 c71a1a3d Iustin Pop
separate, virtual network from the nodes, and in an environment where
1302 c71a1a3d Iustin Pop
nodes are not guaranteed to be able to reach each other via multicasting
1303 c71a1a3d Iustin Pop
or broadcasting. For more information see the README in the source
1304 c71a1a3d Iustin Pop
archive.
1305 c71a1a3d Iustin Pop
1306 c71a1a3d Iustin Pop
ganeti-htools
1307 c71a1a3d Iustin Pop
+++++++++++++
1308 c71a1a3d Iustin Pop
1309 c71a1a3d Iustin Pop
The ``ganeti-htools`` software consists of a set of tools:
1310 c71a1a3d Iustin Pop
1311 c71a1a3d Iustin Pop
- ``hail``: an advanced iallocator script compared to Ganeti's builtin
1312 c71a1a3d Iustin Pop
  one
1313 c71a1a3d Iustin Pop
- ``hbal``: a tool for rebalancing the cluster, i.e. moving instances
1314 c71a1a3d Iustin Pop
  around in order to better use the resources on the nodes
1315 c71a1a3d Iustin Pop
- ``hspace``: a tool for estimating the available capacity of a cluster,
1316 c71a1a3d Iustin Pop
  so that capacity planning can be done efficiently
1317 c71a1a3d Iustin Pop
1318 c71a1a3d Iustin Pop
For more information and installation instructions, see the README file
1319 c71a1a3d Iustin Pop
in the source archive.
1320 c71a1a3d Iustin Pop
1321 558fd122 Michael Hanselmann
.. vim: set textwidth=72 :
1322 c71a1a3d Iustin Pop
.. Local Variables:
1323 c71a1a3d Iustin Pop
.. mode: rst
1324 c71a1a3d Iustin Pop
.. fill-column: 72
1325 c71a1a3d Iustin Pop
.. End: