Statistics
| Branch: | Tag: | Revision:

root / doc / install.rst @ d17e74b4

History | View | Annotate | Download (21 kB)

1 28e15341 Iustin Pop
Ganeti installation tutorial
2 28e15341 Iustin Pop
============================
3 28e15341 Iustin Pop
4 77031881 Iustin Pop
Documents Ganeti version 2.0
5 28e15341 Iustin Pop
6 28e15341 Iustin Pop
.. contents::
7 28e15341 Iustin Pop
8 28e15341 Iustin Pop
Introduction
9 28e15341 Iustin Pop
------------
10 28e15341 Iustin Pop
11 28e15341 Iustin Pop
Ganeti is a cluster virtualization management system based on Xen or
12 28e15341 Iustin Pop
KVM. This document explains how to bootstrap a Ganeti node (Xen
13 28e15341 Iustin Pop
*dom0*), create a running cluster and install virtual instance (Xen
14 28e15341 Iustin Pop
*domU*).  You need to repeat most of the steps in this document for
15 28e15341 Iustin Pop
every node you want to install, but of course we recommend creating
16 28e15341 Iustin Pop
some semi-automatic procedure if you plan to deploy Ganeti on a
17 28e15341 Iustin Pop
medium/large scale.
18 28e15341 Iustin Pop
19 28e15341 Iustin Pop
A basic Ganeti terminology glossary is provided in the introductory
20 28e15341 Iustin Pop
section of the *Ganeti administrator's guide*. Please refer to that
21 28e15341 Iustin Pop
document if you are uncertain about the terms we are using.
22 28e15341 Iustin Pop
23 28e15341 Iustin Pop
Ganeti has been developed for Linux and is distribution-agnostic.
24 28e15341 Iustin Pop
This documentation will use Debian Lenny as an example system but the
25 28e15341 Iustin Pop
examples can easily be translated to any other distribution. ou are
26 28e15341 Iustin Pop
expected to be familiar with your distribution, its package management
27 28e15341 Iustin Pop
system, and Xen or KVM before trying to use Ganeti.
28 28e15341 Iustin Pop
29 28e15341 Iustin Pop
This document is divided into two main sections:
30 28e15341 Iustin Pop
31 28e15341 Iustin Pop
- Installation of the base system and base components
32 28e15341 Iustin Pop
33 28e15341 Iustin Pop
- Configuration of the environment for Ganeti
34 28e15341 Iustin Pop
35 28e15341 Iustin Pop
Each of these is divided into sub-sections. While a full Ganeti system
36 28e15341 Iustin Pop
will need all of the steps specified, some are not strictly required
37 28e15341 Iustin Pop
for every environment. Which ones they are, and why, is specified in
38 28e15341 Iustin Pop
the corresponding sections.
39 28e15341 Iustin Pop
40 28e15341 Iustin Pop
Installing the base system and base components
41 28e15341 Iustin Pop
----------------------------------------------
42 28e15341 Iustin Pop
43 28e15341 Iustin Pop
Hardware requirements
44 28e15341 Iustin Pop
+++++++++++++++++++++
45 28e15341 Iustin Pop
46 28e15341 Iustin Pop
Any system supported by your Linux distribution is fine. 64-bit
47 28e15341 Iustin Pop
systems are better as they can support more memory.
48 28e15341 Iustin Pop
49 28e15341 Iustin Pop
Any disk drive recognized by Linux (``IDE``/``SCSI``/``SATA``/etc.)
50 28e15341 Iustin Pop
is supported in Ganeti. Note that no shared storage (e.g.  ``SAN``) is
51 28e15341 Iustin Pop
needed to get high-availability features (but of course, one can be
52 28e15341 Iustin Pop
used to store the images). It is highly recommended to use more than
53 28e15341 Iustin Pop
one disk drive to improve speed. But Ganeti also works with one disk
54 28e15341 Iustin Pop
per machine.
55 28e15341 Iustin Pop
56 28e15341 Iustin Pop
Installing the base system
57 28e15341 Iustin Pop
++++++++++++++++++++++++++
58 28e15341 Iustin Pop
59 28e15341 Iustin Pop
**Mandatory** on all nodes.
60 28e15341 Iustin Pop
61 28e15341 Iustin Pop
It is advised to start with a clean, minimal install of the operating
62 28e15341 Iustin Pop
system. The only requirement you need to be aware of at this stage is
63 28e15341 Iustin Pop
to partition leaving enough space for a big (**minimum** 20GiB) LVM
64 28e15341 Iustin Pop
volume group which will then host your instance filesystems, if you
65 28e15341 Iustin Pop
want to use all Ganeti features. The volume group name Ganeti 2.0 uses
66 28e15341 Iustin Pop
(by default) is ``xenvg``.
67 28e15341 Iustin Pop
68 28e15341 Iustin Pop
You can also use file-based storage only, without LVM, but this setup
69 28e15341 Iustin Pop
is not detailed in this document.
70 28e15341 Iustin Pop
71 28e15341 Iustin Pop
72 28e15341 Iustin Pop
While you can use an existing system, please note that the Ganeti
73 28e15341 Iustin Pop
installation is intrusive in terms of changes to the system
74 28e15341 Iustin Pop
configuration, and it's best to use a newly-installed system without
75 28e15341 Iustin Pop
important data on it.
76 28e15341 Iustin Pop
77 28e15341 Iustin Pop
Also, for best results, it's advised that the nodes have as much as
78 28e15341 Iustin Pop
possible the same hardware and software configuration. This will make
79 28e15341 Iustin Pop
administration much easier.
80 28e15341 Iustin Pop
81 28e15341 Iustin Pop
Hostname issues
82 28e15341 Iustin Pop
~~~~~~~~~~~~~~~
83 28e15341 Iustin Pop
84 28e15341 Iustin Pop
Note that Ganeti requires the hostnames of the systems (i.e. what the
85 28e15341 Iustin Pop
``hostname`` command outputs to be a fully-qualified name, not a short
86 28e15341 Iustin Pop
name. In other words, you should use *node1.example.com* as a hostname
87 28e15341 Iustin Pop
and not just *node1*.
88 28e15341 Iustin Pop
89 28e15341 Iustin Pop
.. admonition:: Debian
90 28e15341 Iustin Pop
91 28e15341 Iustin Pop
   Debian Lenny and Etch configures the hostname differently than you
92 28e15341 Iustin Pop
   need it for Ganeti. For example, this is what Etch puts in
93 28e15341 Iustin Pop
   ``/etc/hosts`` in certain situations::
94 28e15341 Iustin Pop
95 28e15341 Iustin Pop
     127.0.0.1       localhost
96 28e15341 Iustin Pop
     127.0.1.1       node1.example.com node1
97 28e15341 Iustin Pop
98 28e15341 Iustin Pop
   but for Ganeti you need to have::
99 28e15341 Iustin Pop
100 28e15341 Iustin Pop
     127.0.0.1       localhost
101 28e15341 Iustin Pop
     192.168.1.1     node1.example.com node1
102 28e15341 Iustin Pop
103 28e15341 Iustin Pop
   replacing ``192.168.1.1`` with your node's address. Also, the file
104 28e15341 Iustin Pop
   ``/etc/hostname`` which configures the hostname of the system
105 28e15341 Iustin Pop
   should contain ``node1.example.com`` and not just ``node1`` (you
106 28e15341 Iustin Pop
   need to run the command ``/etc/init.d/hostname.sh start`` after
107 28e15341 Iustin Pop
   changing the file).
108 28e15341 Iustin Pop
109 28e15341 Iustin Pop
Installing Xen
110 28e15341 Iustin Pop
++++++++++++++
111 28e15341 Iustin Pop
112 28e15341 Iustin Pop
**Mandatory** on all nodes.
113 28e15341 Iustin Pop
114 28e15341 Iustin Pop
While Ganeti is developed with the ability to modularly run on
115 28e15341 Iustin Pop
different virtualization environments in mind the only two currently
116 28e15341 Iustin Pop
useable on a live system are Xen and KVM. Supported
117 28e15341 Iustin Pop
Xen versions are: 3.0.3, 3.0.4 and 3.1.
118 28e15341 Iustin Pop
119 28e15341 Iustin Pop
Please follow your distribution's recommended way to install and set
120 28e15341 Iustin Pop
up Xen, or install Xen from the upstream source, if you wish,
121 28e15341 Iustin Pop
following their manual. For KVM, make sure you have a KVM-enabled
122 28e15341 Iustin Pop
kernel and the KVM tools.
123 28e15341 Iustin Pop
124 28e15341 Iustin Pop
After installing either hypervisor, you need to reboot into your new
125 28e15341 Iustin Pop
system. On some distributions this might involve configuring GRUB
126 28e15341 Iustin Pop
appropriately, whereas others will configure it automatically when you
127 28e15341 Iustin Pop
install the respective kernels.
128 28e15341 Iustin Pop
129 28e15341 Iustin Pop
.. admonition:: Debian
130 28e15341 Iustin Pop
131 28e15341 Iustin Pop
   Under Lenny or Etch you can install the relevant
132 28e15341 Iustin Pop
   ``xen-linux-system`` package, which will pull in both the
133 28e15341 Iustin Pop
   hypervisor and the relevant kernel. Also, if you are installing a
134 28e15341 Iustin Pop
   32-bit Lenny/Etch, you should install the ``libc6-xen`` package
135 28e15341 Iustin Pop
   (run ``apt-get install libc6-xen``).
136 28e15341 Iustin Pop
137 28e15341 Iustin Pop
Xen settings
138 28e15341 Iustin Pop
~~~~~~~~~~~~
139 28e15341 Iustin Pop
140 28e15341 Iustin Pop
It's recommended that dom0 is restricted to a low amount of memory
141 28e15341 Iustin Pop
(512MiB or 1GiB is reasonable) and that memory ballooning is disabled
142 28e15341 Iustin Pop
in the file ``/etc/xen/xend-config.sxp`` by setting
143 28e15341 Iustin Pop
the value ``dom0-min-mem`` to 0,
144 28e15341 Iustin Pop
like this::
145 28e15341 Iustin Pop
146 28e15341 Iustin Pop
  (dom0-min-mem 0)
147 28e15341 Iustin Pop
148 28e15341 Iustin Pop
For optimum performance when running both CPU and I/O intensive
149 28e15341 Iustin Pop
instances, it's also recommended that the dom0 is restricted to one
150 28e15341 Iustin Pop
CPU only, for example by booting with the kernel parameter ``nosmp``.
151 28e15341 Iustin Pop
152 28e15341 Iustin Pop
It is recommended that you disable xen's automatic save of virtual
153 28e15341 Iustin Pop
machines at system shutdown and subsequent restore of them at reboot.
154 28e15341 Iustin Pop
To obtain this make sure the variable ``XENDOMAINS_SAVE`` in the file
155 28e15341 Iustin Pop
``/etc/default/xendomains`` is set to an empty value.
156 28e15341 Iustin Pop
157 28e15341 Iustin Pop
.. admonition:: Debian
158 28e15341 Iustin Pop
159 28e15341 Iustin Pop
   Besides the ballooning change which you need to set in
160 28e15341 Iustin Pop
   ``/etc/xen/xend-config.sxp``, you need to set the memory and nosmp
161 28e15341 Iustin Pop
   parameters in the file ``/boot/grub/menu.lst``. You need to modify
162 28e15341 Iustin Pop
   the variable ``xenhopt`` to add ``dom0_mem=1024M`` like this::
163 28e15341 Iustin Pop
164 28e15341 Iustin Pop
     ## Xen hypervisor options to use with the default Xen boot option
165 28e15341 Iustin Pop
     # xenhopt=dom0_mem=1024M
166 28e15341 Iustin Pop
167 28e15341 Iustin Pop
   and the ``xenkopt`` needs to include the ``nosmp`` option like
168 28e15341 Iustin Pop
   this::
169 28e15341 Iustin Pop
170 28e15341 Iustin Pop
     ## Xen Linux kernel options to use with the default Xen boot option
171 28e15341 Iustin Pop
     # xenkopt=nosmp
172 28e15341 Iustin Pop
173 28e15341 Iustin Pop
   Any existing parameters can be left in place: it's ok to have
174 28e15341 Iustin Pop
   ``xenkopt=console=tty0 nosmp``, for example. After modifying the
175 28e15341 Iustin Pop
   files, you need to run::
176 28e15341 Iustin Pop
177 28e15341 Iustin Pop
     /sbin/update-grub
178 28e15341 Iustin Pop
179 28e15341 Iustin Pop
If you want to run HVM instances too with Ganeti and want VNC access
180 28e15341 Iustin Pop
to the console of your instances, set the following two entries in
181 28e15341 Iustin Pop
``/etc/xen/xend-config.sxp``::
182 28e15341 Iustin Pop
183 28e15341 Iustin Pop
  (vnc-listen '0.0.0.0') (vncpasswd '')
184 28e15341 Iustin Pop
185 28e15341 Iustin Pop
You need to restart the Xen daemon for these settings to take effect::
186 28e15341 Iustin Pop
187 28e15341 Iustin Pop
  /etc/init.d/xend restart
188 28e15341 Iustin Pop
189 28e15341 Iustin Pop
Selecting the instance kernel
190 28e15341 Iustin Pop
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
191 28e15341 Iustin Pop
192 28e15341 Iustin Pop
After you have installed Xen, you need to tell Ganeti exactly what
193 28e15341 Iustin Pop
kernel to use for the instances it will create. This is done by
194 28e15341 Iustin Pop
creating a symlink from your actual kernel to
195 28e15341 Iustin Pop
``/boot/vmlinuz-2.6-xenU``, and one from your initrd
196 28e15341 Iustin Pop
to ``/boot/initrd-2.6-xenU``. Note that if you don't
197 28e15341 Iustin Pop
use an initrd for the domU kernel, you don't need
198 28e15341 Iustin Pop
to create the initrd symlink.
199 28e15341 Iustin Pop
200 28e15341 Iustin Pop
.. admonition:: Debian
201 28e15341 Iustin Pop
202 28e15341 Iustin Pop
   After installation of the ``xen-linux-system`` package, you need to
203 28e15341 Iustin Pop
   run (replace the exact version number with the one you have)::
204 28e15341 Iustin Pop
205 28e15341 Iustin Pop
     cd /boot
206 28e15341 Iustin Pop
     ln -s vmlinuz-2.6.26-1-xen-amd64 vmlinuz-2.6-xenU
207 28e15341 Iustin Pop
     ln -s initrd.img-2.6.26-1-xen-amd64 initrd-2.6-xenU
208 28e15341 Iustin Pop
209 28e15341 Iustin Pop
Installing DRBD
210 28e15341 Iustin Pop
+++++++++++++++
211 28e15341 Iustin Pop
212 28e15341 Iustin Pop
Recommended on all nodes: DRBD_ is required if you want to use the
213 28e15341 Iustin Pop
high availability (HA) features of Ganeti, but optional if you don't
214 28e15341 Iustin Pop
require HA or only run Ganeti on single-node clusters. You can upgrade
215 28e15341 Iustin Pop
a non-HA cluster to an HA one later, but you might need to export and
216 28e15341 Iustin Pop
re-import all your instances to take advantage of the new features.
217 28e15341 Iustin Pop
218 28e15341 Iustin Pop
.. _DRBD: http://www.drbd.org/
219 28e15341 Iustin Pop
220 28e15341 Iustin Pop
Supported DRBD versions: 8.0.x. It's recommended to have at least
221 28e15341 Iustin Pop
version 8.0.12.
222 28e15341 Iustin Pop
223 28e15341 Iustin Pop
Now the bad news: unless your distribution already provides it
224 28e15341 Iustin Pop
installing DRBD might involve recompiling your kernel or anyway
225 28e15341 Iustin Pop
fiddling with it. Hopefully at least the Xen-ified kernel source to
226 28e15341 Iustin Pop
start from will be provided.
227 28e15341 Iustin Pop
228 28e15341 Iustin Pop
The good news is that you don't need to configure DRBD at all. Ganeti
229 28e15341 Iustin Pop
will do it for you for every instance you set up.  If you have the
230 28e15341 Iustin Pop
DRBD utils installed and the module in your kernel you're fine. Please
231 28e15341 Iustin Pop
check that your system is configured to load the module at every boot,
232 28e15341 Iustin Pop
and that it passes the following option to the module
233 28e15341 Iustin Pop
``minor_count=255``. This will allow you to use up to 128 instances
234 28e15341 Iustin Pop
per node (for most clusters 128 should be enough, though).
235 28e15341 Iustin Pop
236 28e15341 Iustin Pop
.. admonition:: Debian
237 28e15341 Iustin Pop
238 28e15341 Iustin Pop
   On Debian, you can just install (build) the DRBD 8.0.x module with
239 28e15341 Iustin Pop
   the following commands (make sure you are running the Xen kernel)::
240 28e15341 Iustin Pop
241 28e15341 Iustin Pop
     apt-get install drbd8-source drbd8-utils
242 28e15341 Iustin Pop
     m-a update
243 28e15341 Iustin Pop
     m-a a-i drbd8
244 28e15341 Iustin Pop
     echo drbd minor_count=128 >> /etc/modules
245 28e15341 Iustin Pop
     depmod -a
246 28e15341 Iustin Pop
     modprobe drbd minor_count=128
247 28e15341 Iustin Pop
248 28e15341 Iustin Pop
   It is also recommended that you comment out the default resources
249 28e15341 Iustin Pop
   in the ``/etc/drbd.conf`` file, so that the init script doesn't try
250 28e15341 Iustin Pop
   to configure any drbd devices. You can do this by prefixing all
251 28e15341 Iustin Pop
   *resource* lines in the file with the keyword *skip*, like this::
252 28e15341 Iustin Pop
253 28e15341 Iustin Pop
     skip resource r0 {
254 28e15341 Iustin Pop
       ...
255 28e15341 Iustin Pop
     }
256 28e15341 Iustin Pop
257 28e15341 Iustin Pop
     skip resource "r1" {
258 28e15341 Iustin Pop
       ...
259 28e15341 Iustin Pop
     }
260 28e15341 Iustin Pop
261 28e15341 Iustin Pop
Other required software
262 28e15341 Iustin Pop
+++++++++++++++++++++++
263 28e15341 Iustin Pop
264 28e15341 Iustin Pop
Besides Xen and DRBD, you will need to install the following (on all
265 28e15341 Iustin Pop
nodes):
266 28e15341 Iustin Pop
267 28e15341 Iustin Pop
- LVM version 2, `<http://sourceware.org/lvm2/>`_
268 28e15341 Iustin Pop
269 28e15341 Iustin Pop
- OpenSSL, `<http://www.openssl.org/>`_
270 28e15341 Iustin Pop
271 28e15341 Iustin Pop
- OpenSSH, `<http://www.openssh.com/portable.html>`_
272 28e15341 Iustin Pop
273 28e15341 Iustin Pop
- bridge utilities, `<http://bridge.sourceforge.net/>`_
274 28e15341 Iustin Pop
275 28e15341 Iustin Pop
- iproute2, `<http://developer.osdl.org/dev/iproute2>`_
276 28e15341 Iustin Pop
277 28e15341 Iustin Pop
- arping (part of iputils package),
278 28e15341 Iustin Pop
  `<ftp://ftp.inr.ac.ru/ip-routing/iputils-current.tar.gz>`_
279 28e15341 Iustin Pop
280 28e15341 Iustin Pop
- Python version 2.4 or 2.5, `<http://www.python.org>`_
281 28e15341 Iustin Pop
282 28e15341 Iustin Pop
- Python OpenSSL bindings, `<http://pyopenssl.sourceforge.net/>`_
283 28e15341 Iustin Pop
284 28e15341 Iustin Pop
285 28e15341 Iustin Pop
- simplejson Python module, `<http://www.undefined.org/python/#simplejson>`_
286 28e15341 Iustin Pop
287 28e15341 Iustin Pop
- pyparsing Python module, `<http://pyparsing.wikispaces.com/>`_
288 28e15341 Iustin Pop
289 28e15341 Iustin Pop
These programs are supplied as part of most Linux distributions, so
290 28e15341 Iustin Pop
usually they can be installed via apt or similar methods. Also many of
291 28e15341 Iustin Pop
them will already be installed on a standard machine.
292 28e15341 Iustin Pop
293 28e15341 Iustin Pop
294 28e15341 Iustin Pop
.. admonition:: Debian
295 28e15341 Iustin Pop
296 28e15341 Iustin Pop
   You can use this command line to install all needed packages::
297 28e15341 Iustin Pop
298 28e15341 Iustin Pop
     # apt-get install lvm2 ssh bridge-utils iproute iputils-arping \
299 28e15341 Iustin Pop
     python python-pyopenssl openssl python-pyparsing python-simplejson
300 28e15341 Iustin Pop
301 28e15341 Iustin Pop
Setting up the environment for Ganeti
302 28e15341 Iustin Pop
-------------------------------------
303 28e15341 Iustin Pop
304 28e15341 Iustin Pop
Configuring the network
305 28e15341 Iustin Pop
+++++++++++++++++++++++
306 28e15341 Iustin Pop
307 28e15341 Iustin Pop
**Mandatory** on all nodes.
308 28e15341 Iustin Pop
309 28e15341 Iustin Pop
Ganeti relies on Xen running in "bridge mode", which means the
310 28e15341 Iustin Pop
instances network interfaces will be attached to a software bridge
311 28e15341 Iustin Pop
running in dom0. Xen by default creates such a bridge at startup, but
312 28e15341 Iustin Pop
your distribution might have a different way to do things.
313 28e15341 Iustin Pop
314 28e15341 Iustin Pop
Beware that the default name Ganeti uses is ``xen-br0`` (which was
315 28e15341 Iustin Pop
used in Xen 2.0) while Xen 3.0 uses ``xenbr0`` by default. The default
316 28e15341 Iustin Pop
bridge your Ganeti cluster will use for new instances can be specified
317 28e15341 Iustin Pop
at cluster initialization time.
318 28e15341 Iustin Pop
319 28e15341 Iustin Pop
.. admonition:: Debian
320 28e15341 Iustin Pop
321 28e15341 Iustin Pop
   The recommended way to configure the Xen bridge is to edit your
322 28e15341 Iustin Pop
   ``/etc/network/interfaces`` file and substitute your normal
323 28e15341 Iustin Pop
   Ethernet stanza with the following snippet::
324 28e15341 Iustin Pop
325 28e15341 Iustin Pop
     auto xen-br0
326 28e15341 Iustin Pop
     iface xen-br0 inet static
327 28e15341 Iustin Pop
        address YOUR_IP_ADDRESS
328 28e15341 Iustin Pop
        netmask YOUR_NETMASK
329 28e15341 Iustin Pop
        network YOUR_NETWORK
330 28e15341 Iustin Pop
        broadcast YOUR_BROADCAST_ADDRESS
331 28e15341 Iustin Pop
        gateway YOUR_GATEWAY
332 28e15341 Iustin Pop
        bridge_ports eth0
333 28e15341 Iustin Pop
        bridge_stp off
334 28e15341 Iustin Pop
        bridge_fd 0
335 28e15341 Iustin Pop
336 28e15341 Iustin Pop
The following commands need to be executed on the local console:
337 28e15341 Iustin Pop
338 28e15341 Iustin Pop
  ifdown eth0
339 28e15341 Iustin Pop
  ifup xen-br0
340 28e15341 Iustin Pop
341 28e15341 Iustin Pop
To check if the bridge is setup, use the ``ip`` and ``brctl show``
342 28e15341 Iustin Pop
commands::
343 28e15341 Iustin Pop
344 28e15341 Iustin Pop
  # ip a show xen-br0
345 28e15341 Iustin Pop
  9: xen-br0: <BROADCAST,MULTICAST,UP,10000> mtu 1500 qdisc noqueue
346 28e15341 Iustin Pop
      link/ether 00:20:fc:1e:d5:5d brd ff:ff:ff:ff:ff:ff
347 28e15341 Iustin Pop
      inet 10.1.1.200/24 brd 10.1.1.255 scope global xen-br0
348 28e15341 Iustin Pop
      inet6 fe80::220:fcff:fe1e:d55d/64 scope link
349 28e15341 Iustin Pop
         valid_lft forever preferred_lft forever
350 28e15341 Iustin Pop
351 28e15341 Iustin Pop
  # brctl show xen-br0
352 28e15341 Iustin Pop
  bridge name     bridge id               STP enabled     interfaces
353 28e15341 Iustin Pop
  xen-br0         8000.0020fc1ed55d       no              eth0
354 28e15341 Iustin Pop
355 28e15341 Iustin Pop
Configuring LVM
356 28e15341 Iustin Pop
+++++++++++++++
357 28e15341 Iustin Pop
358 28e15341 Iustin Pop
**Mandatory** on all nodes.
359 28e15341 Iustin Pop
360 28e15341 Iustin Pop
The volume group is required to be at least 20GiB.
361 28e15341 Iustin Pop
362 28e15341 Iustin Pop
If you haven't configured your LVM volume group at install time you
363 28e15341 Iustin Pop
need to do it before trying to initialize the Ganeti cluster. This is
364 28e15341 Iustin Pop
done by formatting the devices/partitions you want to use for it and
365 28e15341 Iustin Pop
then adding them to the relevant volume group::
366 28e15341 Iustin Pop
367 28e15341 Iustin Pop
  pvcreate /dev/sda3
368 28e15341 Iustin Pop
  vgcreate xenvg /dev/sda3
369 28e15341 Iustin Pop
370 28e15341 Iustin Pop
or::
371 28e15341 Iustin Pop
372 28e15341 Iustin Pop
  pvcreate /dev/sdb1
373 28e15341 Iustin Pop
  pvcreate /dev/sdc1
374 28e15341 Iustin Pop
  vgcreate xenvg /dev/sdb1 /dev/sdc1
375 28e15341 Iustin Pop
376 28e15341 Iustin Pop
If you want to add a device later you can do so with the *vgextend*
377 28e15341 Iustin Pop
command::
378 28e15341 Iustin Pop
379 28e15341 Iustin Pop
  pvcreate /dev/sdd1
380 28e15341 Iustin Pop
  vgextend xenvg /dev/sdd1
381 28e15341 Iustin Pop
382 28e15341 Iustin Pop
Optional: it is recommended to configure LVM not to scan the DRBD
383 28e15341 Iustin Pop
devices for physical volumes. This can be accomplished by editing
384 28e15341 Iustin Pop
``/etc/lvm/lvm.conf`` and adding the
385 28e15341 Iustin Pop
``/dev/drbd[0-9]+`` regular expression to the
386 28e15341 Iustin Pop
``filter`` variable, like this::
387 28e15341 Iustin Pop
388 28e15341 Iustin Pop
  filter = ["r|/dev/cdrom|", "r|/dev/drbd[0-9]+|" ]
389 28e15341 Iustin Pop
390 28e15341 Iustin Pop
Installing Ganeti
391 28e15341 Iustin Pop
+++++++++++++++++
392 28e15341 Iustin Pop
393 28e15341 Iustin Pop
**Mandatory** on all nodes.
394 28e15341 Iustin Pop
395 28e15341 Iustin Pop
It's now time to install the Ganeti software itself.  Download the
396 28e15341 Iustin Pop
source from the project page at `<http://code.google.com/p/ganeti/>`_,
397 28e15341 Iustin Pop
and install it (replace 2.0.0 with the latest version)::
398 28e15341 Iustin Pop
399 28e15341 Iustin Pop
  tar xvzf ganeti-2.0.0.tar.gz
400 28e15341 Iustin Pop
  cd ganeti-2.0.0
401 28e15341 Iustin Pop
  ./configure --localstatedir=/var --sysconfdir=/etc
402 28e15341 Iustin Pop
  make
403 28e15341 Iustin Pop
  make install
404 28e15341 Iustin Pop
  mkdir /srv/ganeti/ /srv/ganeti/os /srv/ganeti/export
405 28e15341 Iustin Pop
406 28e15341 Iustin Pop
You also need to copy the file
407 28e15341 Iustin Pop
``doc/examples/ganeti.initd`` from the source archive
408 28e15341 Iustin Pop
to ``/etc/init.d/ganeti`` and register it with your
409 28e15341 Iustin Pop
distribution's startup scripts, for example in Debian::
410 28e15341 Iustin Pop
411 28e15341 Iustin Pop
  update-rc.d ganeti defaults 20 80
412 28e15341 Iustin Pop
413 28e15341 Iustin Pop
In order to automatically restart failed instances, you need to setup
414 28e15341 Iustin Pop
a cron job run the *ganeti-watcher* command. A sample cron file is
415 28e15341 Iustin Pop
provided in the source at ``doc/examples/ganeti.cron`` and you can
416 28e15341 Iustin Pop
copy that (eventually altering the path) to ``/etc/cron.d/ganeti``.
417 28e15341 Iustin Pop
418 28e15341 Iustin Pop
Installing the Operating System support packages
419 28e15341 Iustin Pop
++++++++++++++++++++++++++++++++++++++++++++++++
420 28e15341 Iustin Pop
421 28e15341 Iustin Pop
**Mandatory** on all nodes.
422 28e15341 Iustin Pop
423 28e15341 Iustin Pop
To be able to install instances you need to have an Operating System
424 28e15341 Iustin Pop
installation script. An example OS that works under Debian and can
425 28e15341 Iustin Pop
install Debian and Ubuntu instace OSes is provided on the project web
426 28e15341 Iustin Pop
site.  Download it from the project page and follow the instructions
427 28e15341 Iustin Pop
in the ``README`` file.  Here is the installation procedure (replace
428 28e15341 Iustin Pop
0.7 with the latest version that is compatible with your ganeti
429 28e15341 Iustin Pop
version)::
430 28e15341 Iustin Pop
431 28e15341 Iustin Pop
  cd /usr/local/src/
432 28e15341 Iustin Pop
  wget http://ganeti.googlecode.com/files/ganeti-instance-debootstrap-0.7.tar.gz
433 28e15341 Iustin Pop
  tar xzf ganeti-instance-debootstrap-0.7.tar.gz
434 28e15341 Iustin Pop
  cd ganeti-instance-debootstrap-0.7
435 28e15341 Iustin Pop
  ./configure
436 28e15341 Iustin Pop
  make
437 28e15341 Iustin Pop
  make install
438 28e15341 Iustin Pop
439 28e15341 Iustin Pop
In order to use this OS definition, you need to have internet access
440 28e15341 Iustin Pop
from your nodes and have the *debootstrap*, *dump* and *restore*
441 28e15341 Iustin Pop
commands installed on all nodes. Also, if the OS is configured to
442 28e15341 Iustin Pop
partition the instance's disk in
443 28e15341 Iustin Pop
``/etc/default/ganeti-instance-debootstrap``, you will need *kpartx*
444 28e15341 Iustin Pop
installed.
445 28e15341 Iustin Pop
446 28e15341 Iustin Pop
.. admonition:: Debian
447 28e15341 Iustin Pop
448 28e15341 Iustin Pop
   Use this command on all nodes to install the required packages::
449 28e15341 Iustin Pop
450 28e15341 Iustin Pop
     apt-get install debootstrap dump kpartx
451 28e15341 Iustin Pop
452 28e15341 Iustin Pop
Alternatively, you can create your own OS definitions. See the manpage
453 28e15341 Iustin Pop
*ganeti-os-interface*.
454 28e15341 Iustin Pop
455 28e15341 Iustin Pop
Initializing the cluster
456 28e15341 Iustin Pop
++++++++++++++++++++++++
457 28e15341 Iustin Pop
458 28e15341 Iustin Pop
**Mandatory** on one node per cluster.
459 28e15341 Iustin Pop
460 28e15341 Iustin Pop
The last step is to initialize the cluster. After you've repeated the
461 28e15341 Iustin Pop
above process on all of your nodes, choose one as the master, and
462 28e15341 Iustin Pop
execute::
463 28e15341 Iustin Pop
464 28e15341 Iustin Pop
  gnt-cluster init <CLUSTERNAME>
465 28e15341 Iustin Pop
466 28e15341 Iustin Pop
The *CLUSTERNAME* is a hostname, which must be resolvable (e.g. it
467 28e15341 Iustin Pop
must exist in DNS or in ``/etc/hosts``) by all the nodes in the
468 28e15341 Iustin Pop
cluster. You must choose a name different from any of the nodes names
469 28e15341 Iustin Pop
for a multi-node cluster. In general the best choice is to have a
470 28e15341 Iustin Pop
unique name for a cluster, even if it consists of only one machine, as
471 28e15341 Iustin Pop
you will be able to expand it later without any problems. Please note
472 28e15341 Iustin Pop
that the hostname used for this must resolve to an IP address reserved
473 28e15341 Iustin Pop
**exclusively** for this purpose, and cannot be the name of the first
474 28e15341 Iustin Pop
(master) node.
475 28e15341 Iustin Pop
476 28e15341 Iustin Pop
If the bridge name you are using is not ``xen-br0``, use the *-b
477 28e15341 Iustin Pop
<BRIDGENAME>* option to specify the bridge name. In this case, you
478 28e15341 Iustin Pop
should also use the *--master-netdev <BRIDGENAME>* option with the
479 28e15341 Iustin Pop
same BRIDGENAME argument.
480 28e15341 Iustin Pop
481 28e15341 Iustin Pop
You can use a different name than ``xenvg`` for the volume group (but
482 28e15341 Iustin Pop
note that the name must be identical on all nodes). In this case you
483 28e15341 Iustin Pop
need to specify it by passing the *-g <VGNAME>* option to
484 28e15341 Iustin Pop
``gnt-cluster init``.
485 28e15341 Iustin Pop
486 28e15341 Iustin Pop
To set up the cluster as an HVM cluster, use the
487 28e15341 Iustin Pop
``--enabled-hypervisors=xen-hvm`` option to enable the HVM hypervisor
488 28e15341 Iustin Pop
(you can also add ``,xen-pvm`` to enable the PVM one too). You will
489 28e15341 Iustin Pop
also need to create the VNC cluster password file
490 28e15341 Iustin Pop
``/etc/ganeti/vnc-cluster-password`` which contains one line with the
491 28e15341 Iustin Pop
default VNC password for the cluster.
492 28e15341 Iustin Pop
493 28e15341 Iustin Pop
To setup the cluster for KVM-only usage (KVM and Xen cannot be mixed),
494 28e15341 Iustin Pop
pass ``--enabled-hypervisors=kvm`` to the init command.
495 28e15341 Iustin Pop
496 28e15341 Iustin Pop
You can also invoke the command with the ``--help`` option in order to
497 28e15341 Iustin Pop
see all the possibilities.
498 28e15341 Iustin Pop
499 28e15341 Iustin Pop
Joining the nodes to the cluster
500 28e15341 Iustin Pop
++++++++++++++++++++++++++++++++
501 28e15341 Iustin Pop
502 28e15341 Iustin Pop
**Mandatory** for all the other nodes.
503 28e15341 Iustin Pop
504 28e15341 Iustin Pop
After you have initialized your cluster you need to join the other
505 28e15341 Iustin Pop
nodes to it. You can do so by executing the following command on the
506 28e15341 Iustin Pop
master node::
507 28e15341 Iustin Pop
508 28e15341 Iustin Pop
  gnt-node add <NODENAME>
509 28e15341 Iustin Pop
510 28e15341 Iustin Pop
Separate replication network
511 28e15341 Iustin Pop
++++++++++++++++++++++++++++
512 28e15341 Iustin Pop
513 28e15341 Iustin Pop
**Optional**
514 28e15341 Iustin Pop
515 28e15341 Iustin Pop
Ganeti uses DRBD to mirror the disk of the virtual instances between
516 28e15341 Iustin Pop
nodes. To use a dedicated network interface for this (in order to
517 28e15341 Iustin Pop
improve performance or to enhance security) you need to configure an
518 28e15341 Iustin Pop
additional interface for each node.  Use the *-s* option with
519 28e15341 Iustin Pop
``gnt-cluster init`` and ``gnt-node add`` to specify the IP address of
520 28e15341 Iustin Pop
this secondary interface to use for each node. Note that if you
521 28e15341 Iustin Pop
specified this option at cluster setup time, you must afterwards use
522 28e15341 Iustin Pop
it for every node add operation.
523 28e15341 Iustin Pop
524 28e15341 Iustin Pop
Testing the setup
525 28e15341 Iustin Pop
+++++++++++++++++
526 28e15341 Iustin Pop
527 28e15341 Iustin Pop
Execute the ``gnt-node list`` command to see all nodes in the
528 28e15341 Iustin Pop
cluster::
529 28e15341 Iustin Pop
530 28e15341 Iustin Pop
  # gnt-node list
531 28e15341 Iustin Pop
  Node              DTotal  DFree MTotal MNode MFree Pinst Sinst
532 28e15341 Iustin Pop
  node1.example.com 197404 197404   2047  1896   125     0     0
533 28e15341 Iustin Pop
534 28e15341 Iustin Pop
Setting up and managing virtual instances
535 28e15341 Iustin Pop
-----------------------------------------
536 28e15341 Iustin Pop
537 28e15341 Iustin Pop
Setting up virtual instances
538 28e15341 Iustin Pop
++++++++++++++++++++++++++++
539 28e15341 Iustin Pop
540 28e15341 Iustin Pop
This step shows how to setup a virtual instance with either
541 28e15341 Iustin Pop
non-mirrored disks (``plain``) or with network mirrored disks
542 28e15341 Iustin Pop
(``drbd``).  All commands need to be executed on the Ganeti master
543 28e15341 Iustin Pop
node (the one on which ``gnt-cluster init`` was run).  Verify that the
544 28e15341 Iustin Pop
OS scripts are present on all cluster nodes with ``gnt-os list``.
545 28e15341 Iustin Pop
546 28e15341 Iustin Pop
547 28e15341 Iustin Pop
To create a virtual instance, you need a hostname which is resolvable
548 28e15341 Iustin Pop
(DNS or ``/etc/hosts`` on all nodes). The following command will
549 28e15341 Iustin Pop
create a non-mirrored instance for you::
550 28e15341 Iustin Pop
551 28e15341 Iustin Pop
  gnt-instance add -t plain -s 1G -n node1 -o debootstrap instance1.example.com
552 28e15341 Iustin Pop
  * creating instance disks...
553 28e15341 Iustin Pop
  adding instance instance1.example.com to cluster config
554 28e15341 Iustin Pop
   - INFO: Waiting for instance instance1.example.com to sync disks.
555 28e15341 Iustin Pop
   - INFO: Instance instance1.example.com's disks are in sync.
556 28e15341 Iustin Pop
  creating os for instance instance1.example.com on node node1.example.com
557 28e15341 Iustin Pop
  * running the instance OS create scripts...
558 28e15341 Iustin Pop
  * starting instance...
559 28e15341 Iustin Pop
560 28e15341 Iustin Pop
The above instance will have no network interface enabled. You can
561 28e15341 Iustin Pop
access it over the virtual console with ``gnt-instance console
562 28e15341 Iustin Pop
inst1``. There is no password for root. As this is a Debian instance,
563 28e15341 Iustin Pop
you can modify the ``/etc/network/interfaces`` file to setup the
564 28e15341 Iustin Pop
network interface (eth0 is the name of the interface provided to the
565 28e15341 Iustin Pop
instance).
566 28e15341 Iustin Pop
567 28e15341 Iustin Pop
To create a network mirrored instance, change the argument to the *-t*
568 28e15341 Iustin Pop
option from ``plain`` to ``drbd`` and specify the node on which the
569 28e15341 Iustin Pop
mirror should reside with the second value of the *--node* option,
570 28e15341 Iustin Pop
like this (note that the command output includes timestamps which have
571 28e15341 Iustin Pop
been removed for clarity)::
572 28e15341 Iustin Pop
573 28e15341 Iustin Pop
  # gnt-instance add -t drbd -s 1G -n node1:node2 -o debootstrap instance2
574 28e15341 Iustin Pop
  * creating instance disks...
575 28e15341 Iustin Pop
  adding instance instance2.example.com to cluster config
576 28e15341 Iustin Pop
   - INFO: Waiting for instance instance2.example.com to sync disks.
577 28e15341 Iustin Pop
   - INFO: - device disk/0: 35.50% done, 11 estimated seconds remaining
578 28e15341 Iustin Pop
   - INFO: - device disk/0: 100.00% done, 0 estimated seconds remaining
579 28e15341 Iustin Pop
   - INFO: Instance instance2.example.com's disks are in sync.
580 28e15341 Iustin Pop
  creating os for instance instance2.example.com on node node1.example.com
581 28e15341 Iustin Pop
  * running the instance OS create scripts...
582 28e15341 Iustin Pop
  * starting instance...
583 28e15341 Iustin Pop
584 28e15341 Iustin Pop
Managing virtual instances
585 28e15341 Iustin Pop
++++++++++++++++++++++++++
586 28e15341 Iustin Pop
587 28e15341 Iustin Pop
All commands need to be executed on the Ganeti master node.
588 28e15341 Iustin Pop
589 28e15341 Iustin Pop
To access the console of an instance, run::
590 28e15341 Iustin Pop
591 28e15341 Iustin Pop
  gnt-instance console INSTANCENAME
592 28e15341 Iustin Pop
593 28e15341 Iustin Pop
To shutdown an instance, run::
594 28e15341 Iustin Pop
595 28e15341 Iustin Pop
  gnt-instance shutdown INSTANCENAME
596 28e15341 Iustin Pop
597 28e15341 Iustin Pop
To startup an instance, run::
598 28e15341 Iustin Pop
599 28e15341 Iustin Pop
  gnt-instance startup INSTANCENAME
600 28e15341 Iustin Pop
601 28e15341 Iustin Pop
To failover an instance to its secondary node (only possible with
602 28e15341 Iustin Pop
``drbd`` disk templates), run::
603 28e15341 Iustin Pop
604 28e15341 Iustin Pop
  gnt-instance failover INSTANCENAME
605 28e15341 Iustin Pop
606 28e15341 Iustin Pop
For more instance and cluster administration details, see the
607 28e15341 Iustin Pop
*Ganeti administrator's guide*.