Statistics
| Branch: | Tag: | Revision:

root / doc / install.rst @ f154a7a3

History | View | Annotate | Download (23.5 kB)

1 28e15341 Iustin Pop
Ganeti installation tutorial
2 28e15341 Iustin Pop
============================
3 28e15341 Iustin Pop
4 fd07c6b3 Iustin Pop
Documents Ganeti version |version|
5 28e15341 Iustin Pop
6 28e15341 Iustin Pop
.. contents::
7 28e15341 Iustin Pop
8 c71a1a3d Iustin Pop
.. highlight:: text
9 c71a1a3d Iustin Pop
10 28e15341 Iustin Pop
Introduction
11 28e15341 Iustin Pop
------------
12 28e15341 Iustin Pop
13 28e15341 Iustin Pop
Ganeti is a cluster virtualization management system based on Xen or
14 c71a1a3d Iustin Pop
KVM. This document explains how to bootstrap a Ganeti node (Xen *dom0*,
15 c71a1a3d Iustin Pop
the host Linux system for KVM), create a running cluster and install
16 c71a1a3d Iustin Pop
virtual instances (Xen *domUs*, KVM guests).  You need to repeat most of
17 c71a1a3d Iustin Pop
the steps in this document for every node you want to install, but of
18 c71a1a3d Iustin Pop
course we recommend creating some semi-automatic procedure if you plan
19 c71a1a3d Iustin Pop
to deploy Ganeti on a medium/large scale.
20 28e15341 Iustin Pop
21 28e15341 Iustin Pop
A basic Ganeti terminology glossary is provided in the introductory
22 c71a1a3d Iustin Pop
section of the :doc:`admin`. Please refer to that document if you are
23 c71a1a3d Iustin Pop
uncertain about the terms we are using.
24 28e15341 Iustin Pop
25 c71a1a3d Iustin Pop
Ganeti has been developed for Linux and should be distribution-agnostic.
26 28e15341 Iustin Pop
This documentation will use Debian Lenny as an example system but the
27 c71a1a3d Iustin Pop
examples can be translated to any other distribution. You are expected
28 c71a1a3d Iustin Pop
to be familiar with your distribution, its package management system,
29 c71a1a3d Iustin Pop
and Xen or KVM before trying to use Ganeti.
30 28e15341 Iustin Pop
31 28e15341 Iustin Pop
This document is divided into two main sections:
32 28e15341 Iustin Pop
33 28e15341 Iustin Pop
- Installation of the base system and base components
34 28e15341 Iustin Pop
35 28e15341 Iustin Pop
- Configuration of the environment for Ganeti
36 28e15341 Iustin Pop
37 28e15341 Iustin Pop
Each of these is divided into sub-sections. While a full Ganeti system
38 c71a1a3d Iustin Pop
will need all of the steps specified, some are not strictly required for
39 c71a1a3d Iustin Pop
every environment. Which ones they are, and why, is specified in the
40 c71a1a3d Iustin Pop
corresponding sections.
41 28e15341 Iustin Pop
42 28e15341 Iustin Pop
Installing the base system and base components
43 28e15341 Iustin Pop
----------------------------------------------
44 28e15341 Iustin Pop
45 28e15341 Iustin Pop
Hardware requirements
46 28e15341 Iustin Pop
+++++++++++++++++++++
47 28e15341 Iustin Pop
48 c71a1a3d Iustin Pop
Any system supported by your Linux distribution is fine. 64-bit systems
49 c71a1a3d Iustin Pop
are better as they can support more memory.
50 28e15341 Iustin Pop
51 c71a1a3d Iustin Pop
Any disk drive recognized by Linux (``IDE``/``SCSI``/``SATA``/etc.) is
52 c71a1a3d Iustin Pop
supported in Ganeti. Note that no shared storage (e.g. ``SAN``) is
53 c71a1a3d Iustin Pop
needed to get high-availability features (but of course, one can be used
54 c71a1a3d Iustin Pop
to store the images). It is highly recommended to use more than one disk
55 c71a1a3d Iustin Pop
drive to improve speed. But Ganeti also works with one disk per machine.
56 28e15341 Iustin Pop
57 28e15341 Iustin Pop
Installing the base system
58 28e15341 Iustin Pop
++++++++++++++++++++++++++
59 28e15341 Iustin Pop
60 28e15341 Iustin Pop
**Mandatory** on all nodes.
61 28e15341 Iustin Pop
62 28e15341 Iustin Pop
It is advised to start with a clean, minimal install of the operating
63 c71a1a3d Iustin Pop
system. The only requirement you need to be aware of at this stage is to
64 c71a1a3d Iustin Pop
partition leaving enough space for a big (**minimum** 20GiB) LVM volume
65 c71a1a3d Iustin Pop
group which will then host your instance filesystems, if you want to use
66 c71a1a3d Iustin Pop
all Ganeti features. The volume group name Ganeti uses (by default) is
67 c71a1a3d Iustin Pop
``xenvg``.
68 28e15341 Iustin Pop
69 c71a1a3d Iustin Pop
You can also use file-based storage only, without LVM, but this setup is
70 c71a1a3d Iustin Pop
not detailed in this document.
71 28e15341 Iustin Pop
72 28e15341 Iustin Pop
While you can use an existing system, please note that the Ganeti
73 28e15341 Iustin Pop
installation is intrusive in terms of changes to the system
74 28e15341 Iustin Pop
configuration, and it's best to use a newly-installed system without
75 28e15341 Iustin Pop
important data on it.
76 28e15341 Iustin Pop
77 28e15341 Iustin Pop
Also, for best results, it's advised that the nodes have as much as
78 28e15341 Iustin Pop
possible the same hardware and software configuration. This will make
79 28e15341 Iustin Pop
administration much easier.
80 28e15341 Iustin Pop
81 28e15341 Iustin Pop
Hostname issues
82 28e15341 Iustin Pop
~~~~~~~~~~~~~~~
83 28e15341 Iustin Pop
84 28e15341 Iustin Pop
Note that Ganeti requires the hostnames of the systems (i.e. what the
85 28e15341 Iustin Pop
``hostname`` command outputs to be a fully-qualified name, not a short
86 28e15341 Iustin Pop
name. In other words, you should use *node1.example.com* as a hostname
87 28e15341 Iustin Pop
and not just *node1*.
88 28e15341 Iustin Pop
89 28e15341 Iustin Pop
.. admonition:: Debian
90 28e15341 Iustin Pop
91 28e15341 Iustin Pop
   Debian Lenny and Etch configures the hostname differently than you
92 28e15341 Iustin Pop
   need it for Ganeti. For example, this is what Etch puts in
93 28e15341 Iustin Pop
   ``/etc/hosts`` in certain situations::
94 28e15341 Iustin Pop
95 28e15341 Iustin Pop
     127.0.0.1       localhost
96 28e15341 Iustin Pop
     127.0.1.1       node1.example.com node1
97 28e15341 Iustin Pop
98 28e15341 Iustin Pop
   but for Ganeti you need to have::
99 28e15341 Iustin Pop
100 28e15341 Iustin Pop
     127.0.0.1       localhost
101 28e15341 Iustin Pop
     192.168.1.1     node1.example.com node1
102 28e15341 Iustin Pop
103 28e15341 Iustin Pop
   replacing ``192.168.1.1`` with your node's address. Also, the file
104 28e15341 Iustin Pop
   ``/etc/hostname`` which configures the hostname of the system
105 28e15341 Iustin Pop
   should contain ``node1.example.com`` and not just ``node1`` (you
106 28e15341 Iustin Pop
   need to run the command ``/etc/init.d/hostname.sh start`` after
107 28e15341 Iustin Pop
   changing the file).
108 28e15341 Iustin Pop
109 1232284c Guido Trotter
.. admonition:: Why a fully qualified host name
110 1232284c Guido Trotter
111 7faf5110 Michael Hanselmann
   Although most distributions use only the short name in the
112 7faf5110 Michael Hanselmann
   /etc/hostname file, we still think Ganeti nodes should use the full
113 7faf5110 Michael Hanselmann
   name. The reason for this is that calling 'hostname --fqdn' requires
114 7faf5110 Michael Hanselmann
   the resolver library to work and is a 'guess' via heuristics at what
115 7faf5110 Michael Hanselmann
   is your domain name. Since Ganeti can be used among other things to
116 7faf5110 Michael Hanselmann
   host DNS servers, we don't want to depend on them as much as
117 7faf5110 Michael Hanselmann
   possible, and we'd rather have the uname() syscall return the full
118 7faf5110 Michael Hanselmann
   node name.
119 7faf5110 Michael Hanselmann
120 7faf5110 Michael Hanselmann
   We haven't ever found any breakage in using a full hostname on a
121 7faf5110 Michael Hanselmann
   Linux system, and anyway we recommend to have only a minimal
122 7faf5110 Michael Hanselmann
   installation on Ganeti nodes, and to use instances (or other
123 7faf5110 Michael Hanselmann
   dedicated machines) to run the rest of your network services. By
124 7faf5110 Michael Hanselmann
   doing this you can change the /etc/hostname file to contain an FQDN
125 7faf5110 Michael Hanselmann
   without the fear of breaking anything unrelated.
126 1232284c Guido Trotter
127 1232284c Guido Trotter
128 756d5ec3 Guido Trotter
Installing The Hypervisor
129 756d5ec3 Guido Trotter
+++++++++++++++++++++++++
130 28e15341 Iustin Pop
131 28e15341 Iustin Pop
**Mandatory** on all nodes.
132 28e15341 Iustin Pop
133 756d5ec3 Guido Trotter
While Ganeti is developed with the ability to modularly run on different
134 7faf5110 Michael Hanselmann
virtualization environments in mind the only two currently useable on a
135 7faf5110 Michael Hanselmann
live system are Xen and KVM. Supported Xen versions are: 3.0.3, 3.0.4
136 7faf5110 Michael Hanselmann
and 3.1.  Supported KVM version are 72 and above.
137 28e15341 Iustin Pop
138 c71a1a3d Iustin Pop
Please follow your distribution's recommended way to install and set up
139 c71a1a3d Iustin Pop
Xen, or install Xen from the upstream source, if you wish, following
140 c71a1a3d Iustin Pop
their manual. For KVM, make sure you have a KVM-enabled kernel and the
141 c71a1a3d Iustin Pop
KVM tools.
142 28e15341 Iustin Pop
143 756d5ec3 Guido Trotter
After installing Xen, you need to reboot into your new system. On some
144 7faf5110 Michael Hanselmann
distributions this might involve configuring GRUB appropriately, whereas
145 7faf5110 Michael Hanselmann
others will configure it automatically when you install the respective
146 7faf5110 Michael Hanselmann
kernels. For KVM no reboot should be necessary.
147 28e15341 Iustin Pop
148 756d5ec3 Guido Trotter
.. admonition:: Xen on Debian
149 28e15341 Iustin Pop
150 c71a1a3d Iustin Pop
   Under Lenny or Etch you can install the relevant ``xen-linux-system``
151 c71a1a3d Iustin Pop
   package, which will pull in both the hypervisor and the relevant
152 c71a1a3d Iustin Pop
   kernel. Also, if you are installing a 32-bit Lenny/Etch, you should
153 c71a1a3d Iustin Pop
   install the ``libc6-xen`` package (run ``apt-get install
154 c71a1a3d Iustin Pop
   libc6-xen``).
155 28e15341 Iustin Pop
156 28e15341 Iustin Pop
Xen settings
157 28e15341 Iustin Pop
~~~~~~~~~~~~
158 28e15341 Iustin Pop
159 28e15341 Iustin Pop
It's recommended that dom0 is restricted to a low amount of memory
160 c71a1a3d Iustin Pop
(512MiB or 1GiB is reasonable) and that memory ballooning is disabled in
161 c71a1a3d Iustin Pop
the file ``/etc/xen/xend-config.sxp`` by setting the value
162 c71a1a3d Iustin Pop
``dom0-min-mem`` to 0, like this::
163 28e15341 Iustin Pop
164 28e15341 Iustin Pop
  (dom0-min-mem 0)
165 28e15341 Iustin Pop
166 28e15341 Iustin Pop
For optimum performance when running both CPU and I/O intensive
167 c71a1a3d Iustin Pop
instances, it's also recommended that the dom0 is restricted to one CPU
168 c71a1a3d Iustin Pop
only, for example by booting with the kernel parameter ``nosmp``.
169 28e15341 Iustin Pop
170 28e15341 Iustin Pop
It is recommended that you disable xen's automatic save of virtual
171 28e15341 Iustin Pop
machines at system shutdown and subsequent restore of them at reboot.
172 28e15341 Iustin Pop
To obtain this make sure the variable ``XENDOMAINS_SAVE`` in the file
173 28e15341 Iustin Pop
``/etc/default/xendomains`` is set to an empty value.
174 28e15341 Iustin Pop
175 aeaa2ea2 Guido Trotter
If you want to use live migration make sure you have, in the xen config
176 aeaa2ea2 Guido Trotter
file, something that allows the nodes to migrate instances between each
177 aeaa2ea2 Guido Trotter
other. For example::
178 8ab90d80 Guido Trotter
179 8ab90d80 Guido Trotter
  (xend-relocation-server yes)
180 8ab90d80 Guido Trotter
  (xend-relocation-port 8002)
181 8ab90d80 Guido Trotter
  (xend-relocation-address '')
182 8ab90d80 Guido Trotter
  (xend-relocation-hosts-allow '^192\\.168\\.3\\.[0-9]+$')
183 8ab90d80 Guido Trotter
184 8ab90d80 Guido Trotter
The last line assumes that all your nodes have secondary IPs in the
185 8ab90d80 Guido Trotter
192.168.3.0/24 network, adjust it accordingly to your setup.
186 8ab90d80 Guido Trotter
187 28e15341 Iustin Pop
.. admonition:: Debian
188 28e15341 Iustin Pop
189 28e15341 Iustin Pop
   Besides the ballooning change which you need to set in
190 28e15341 Iustin Pop
   ``/etc/xen/xend-config.sxp``, you need to set the memory and nosmp
191 28e15341 Iustin Pop
   parameters in the file ``/boot/grub/menu.lst``. You need to modify
192 28e15341 Iustin Pop
   the variable ``xenhopt`` to add ``dom0_mem=1024M`` like this::
193 28e15341 Iustin Pop
194 28e15341 Iustin Pop
     ## Xen hypervisor options to use with the default Xen boot option
195 28e15341 Iustin Pop
     # xenhopt=dom0_mem=1024M
196 28e15341 Iustin Pop
197 c71a1a3d Iustin Pop
   and the ``xenkopt`` needs to include the ``nosmp`` option like this::
198 28e15341 Iustin Pop
199 28e15341 Iustin Pop
     ## Xen Linux kernel options to use with the default Xen boot option
200 28e15341 Iustin Pop
     # xenkopt=nosmp
201 28e15341 Iustin Pop
202 28e15341 Iustin Pop
   Any existing parameters can be left in place: it's ok to have
203 28e15341 Iustin Pop
   ``xenkopt=console=tty0 nosmp``, for example. After modifying the
204 28e15341 Iustin Pop
   files, you need to run::
205 28e15341 Iustin Pop
206 28e15341 Iustin Pop
     /sbin/update-grub
207 28e15341 Iustin Pop
208 c71a1a3d Iustin Pop
If you want to run HVM instances too with Ganeti and want VNC access to
209 c71a1a3d Iustin Pop
the console of your instances, set the following two entries in
210 28e15341 Iustin Pop
``/etc/xen/xend-config.sxp``::
211 28e15341 Iustin Pop
212 28e15341 Iustin Pop
  (vnc-listen '0.0.0.0') (vncpasswd '')
213 28e15341 Iustin Pop
214 28e15341 Iustin Pop
You need to restart the Xen daemon for these settings to take effect::
215 28e15341 Iustin Pop
216 28e15341 Iustin Pop
  /etc/init.d/xend restart
217 28e15341 Iustin Pop
218 28e15341 Iustin Pop
Selecting the instance kernel
219 28e15341 Iustin Pop
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
220 28e15341 Iustin Pop
221 28e15341 Iustin Pop
After you have installed Xen, you need to tell Ganeti exactly what
222 c71a1a3d Iustin Pop
kernel to use for the instances it will create. This is done by creating
223 c71a1a3d Iustin Pop
a symlink from your actual kernel to ``/boot/vmlinuz-2.6-xenU``, and one
224 c71a1a3d Iustin Pop
from your initrd to ``/boot/initrd-2.6-xenU`` [#defkernel]_. Note that
225 c71a1a3d Iustin Pop
if you don't use an initrd for the domU kernel, you don't need to create
226 c71a1a3d Iustin Pop
the initrd symlink.
227 28e15341 Iustin Pop
228 28e15341 Iustin Pop
.. admonition:: Debian
229 28e15341 Iustin Pop
230 28e15341 Iustin Pop
   After installation of the ``xen-linux-system`` package, you need to
231 28e15341 Iustin Pop
   run (replace the exact version number with the one you have)::
232 28e15341 Iustin Pop
233 28e15341 Iustin Pop
     cd /boot
234 28e15341 Iustin Pop
     ln -s vmlinuz-2.6.26-1-xen-amd64 vmlinuz-2.6-xenU
235 28e15341 Iustin Pop
     ln -s initrd.img-2.6.26-1-xen-amd64 initrd-2.6-xenU
236 28e15341 Iustin Pop
237 28e15341 Iustin Pop
Installing DRBD
238 28e15341 Iustin Pop
+++++++++++++++
239 28e15341 Iustin Pop
240 c71a1a3d Iustin Pop
Recommended on all nodes: DRBD_ is required if you want to use the high
241 c71a1a3d Iustin Pop
availability (HA) features of Ganeti, but optional if you don't require
242 c71a1a3d Iustin Pop
them or only run Ganeti on single-node clusters. You can upgrade a
243 c71a1a3d Iustin Pop
non-HA cluster to an HA one later, but you might need to export and
244 28e15341 Iustin Pop
re-import all your instances to take advantage of the new features.
245 28e15341 Iustin Pop
246 28e15341 Iustin Pop
.. _DRBD: http://www.drbd.org/
247 28e15341 Iustin Pop
248 c71a1a3d Iustin Pop
Supported DRBD versions: 8.0+. It's recommended to have at least version
249 c71a1a3d Iustin Pop
8.0.12. Note that for version 8.2 and newer it is needed to pass the
250 c71a1a3d Iustin Pop
``usermode_helper=/bin/true`` parameter to the module, either by
251 c71a1a3d Iustin Pop
configuring ``/etc/modules`` or when inserting it manually.
252 28e15341 Iustin Pop
253 28e15341 Iustin Pop
Now the bad news: unless your distribution already provides it
254 c71a1a3d Iustin Pop
installing DRBD might involve recompiling your kernel or anyway fiddling
255 c71a1a3d Iustin Pop
with it. Hopefully at least the Xen-ified kernel source to start from
256 c71a1a3d Iustin Pop
will be provided (if you intend to use Xen).
257 28e15341 Iustin Pop
258 28e15341 Iustin Pop
The good news is that you don't need to configure DRBD at all. Ganeti
259 c71a1a3d Iustin Pop
will do it for you for every instance you set up.  If you have the DRBD
260 c71a1a3d Iustin Pop
utils installed and the module in your kernel you're fine. Please check
261 c71a1a3d Iustin Pop
that your system is configured to load the module at every boot, and
262 c71a1a3d Iustin Pop
that it passes the following option to the module:
263 c71a1a3d Iustin Pop
``minor_count=NUMBER``. We recommend that you use 128 as the value of
264 c71a1a3d Iustin Pop
the minor_count - this will allow you to use up to 64 instances in total
265 c71a1a3d Iustin Pop
per node (both primary and secondary, when using only one disk per
266 c71a1a3d Iustin Pop
instance). You can increase the number up to 255 if you need more
267 c71a1a3d Iustin Pop
instances on a node.
268 c71a1a3d Iustin Pop
269 28e15341 Iustin Pop
270 28e15341 Iustin Pop
.. admonition:: Debian
271 28e15341 Iustin Pop
272 c71a1a3d Iustin Pop
   On Debian, you can just install (build) the DRBD module with the
273 c71a1a3d Iustin Pop
   following commands, making sure you are running the target (Xen or
274 c71a1a3d Iustin Pop
   KVM) kernel::
275 28e15341 Iustin Pop
276 28e15341 Iustin Pop
     apt-get install drbd8-source drbd8-utils
277 28e15341 Iustin Pop
     m-a update
278 28e15341 Iustin Pop
     m-a a-i drbd8
279 c71a1a3d Iustin Pop
     echo drbd minor_count=128 usermode_helper=/bin/true >> /etc/modules
280 28e15341 Iustin Pop
     depmod -a
281 c71a1a3d Iustin Pop
     modprobe drbd minor_count=128 usermode_helper=/bin/true
282 28e15341 Iustin Pop
283 c71a1a3d Iustin Pop
   It is also recommended that you comment out the default resources in
284 c71a1a3d Iustin Pop
   the ``/etc/drbd.conf`` file, so that the init script doesn't try to
285 c71a1a3d Iustin Pop
   configure any drbd devices. You can do this by prefixing all
286 28e15341 Iustin Pop
   *resource* lines in the file with the keyword *skip*, like this::
287 28e15341 Iustin Pop
288 28e15341 Iustin Pop
     skip resource r0 {
289 28e15341 Iustin Pop
       ...
290 28e15341 Iustin Pop
     }
291 28e15341 Iustin Pop
292 28e15341 Iustin Pop
     skip resource "r1" {
293 28e15341 Iustin Pop
       ...
294 28e15341 Iustin Pop
     }
295 28e15341 Iustin Pop
296 28e15341 Iustin Pop
Other required software
297 28e15341 Iustin Pop
+++++++++++++++++++++++
298 28e15341 Iustin Pop
299 28e15341 Iustin Pop
Besides Xen and DRBD, you will need to install the following (on all
300 28e15341 Iustin Pop
nodes):
301 28e15341 Iustin Pop
302 28e15341 Iustin Pop
- LVM version 2, `<http://sourceware.org/lvm2/>`_
303 28e15341 Iustin Pop
304 28e15341 Iustin Pop
- OpenSSL, `<http://www.openssl.org/>`_
305 28e15341 Iustin Pop
306 28e15341 Iustin Pop
- OpenSSH, `<http://www.openssh.com/portable.html>`_
307 28e15341 Iustin Pop
308 28e15341 Iustin Pop
- bridge utilities, `<http://bridge.sourceforge.net/>`_
309 28e15341 Iustin Pop
310 28e15341 Iustin Pop
- iproute2, `<http://developer.osdl.org/dev/iproute2>`_
311 28e15341 Iustin Pop
312 28e15341 Iustin Pop
- arping (part of iputils package),
313 28e15341 Iustin Pop
  `<ftp://ftp.inr.ac.ru/ip-routing/iputils-current.tar.gz>`_
314 28e15341 Iustin Pop
315 28e15341 Iustin Pop
- Python version 2.4 or 2.5, `<http://www.python.org>`_
316 28e15341 Iustin Pop
317 28e15341 Iustin Pop
- Python OpenSSL bindings, `<http://pyopenssl.sourceforge.net/>`_
318 28e15341 Iustin Pop
319 28e15341 Iustin Pop
- simplejson Python module, `<http://www.undefined.org/python/#simplejson>`_
320 28e15341 Iustin Pop
321 28e15341 Iustin Pop
- pyparsing Python module, `<http://pyparsing.wikispaces.com/>`_
322 28e15341 Iustin Pop
323 0f18ee6d Guido Trotter
- pyinotify Python module, `<http://trac.dbzteam.org/pyinotify>`_
324 0f18ee6d Guido Trotter
325 28e15341 Iustin Pop
These programs are supplied as part of most Linux distributions, so
326 28e15341 Iustin Pop
usually they can be installed via apt or similar methods. Also many of
327 28e15341 Iustin Pop
them will already be installed on a standard machine.
328 28e15341 Iustin Pop
329 28e15341 Iustin Pop
330 28e15341 Iustin Pop
.. admonition:: Debian
331 28e15341 Iustin Pop
332 28e15341 Iustin Pop
   You can use this command line to install all needed packages::
333 28e15341 Iustin Pop
334 28e15341 Iustin Pop
     # apt-get install lvm2 ssh bridge-utils iproute iputils-arping \
335 7faf5110 Michael Hanselmann
     python python-pyopenssl openssl python-pyparsing \
336 7faf5110 Michael Hanselmann
     python-simplejson python-pyinotify
337 28e15341 Iustin Pop
338 28e15341 Iustin Pop
Setting up the environment for Ganeti
339 28e15341 Iustin Pop
-------------------------------------
340 28e15341 Iustin Pop
341 28e15341 Iustin Pop
Configuring the network
342 28e15341 Iustin Pop
+++++++++++++++++++++++
343 28e15341 Iustin Pop
344 28e15341 Iustin Pop
**Mandatory** on all nodes.
345 28e15341 Iustin Pop
346 7faf5110 Michael Hanselmann
You can run Ganeti either in "bridge mode" or in "routed mode". In
347 7faf5110 Michael Hanselmann
bridge mode, the default, the instances network interfaces will be
348 7faf5110 Michael Hanselmann
attached to a software bridge running in dom0. Xen by default creates
349 7faf5110 Michael Hanselmann
such a bridge at startup, but your distribution might have a different
350 7faf5110 Michael Hanselmann
way to do things, and you'll definitely need to manually set it up under
351 7faf5110 Michael Hanselmann
KVM.
352 28e15341 Iustin Pop
353 c71a1a3d Iustin Pop
Beware that the default name Ganeti uses is ``xen-br0`` (which was used
354 c71a1a3d Iustin Pop
in Xen 2.0) while Xen 3.0 uses ``xenbr0`` by default. The default bridge
355 c71a1a3d Iustin Pop
your Ganeti cluster will use for new instances can be specified at
356 c71a1a3d Iustin Pop
cluster initialization time.
357 28e15341 Iustin Pop
358 7faf5110 Michael Hanselmann
If you want to run in "routing mode" you need to specify that at cluster
359 7faf5110 Michael Hanselmann
init time (using the --nicparam option), and then no bridge will be
360 7faf5110 Michael Hanselmann
needed. In this mode instance traffic will be routed by dom0, instead of
361 7faf5110 Michael Hanselmann
bridged.
362 9f83899a Guido Trotter
363 7faf5110 Michael Hanselmann
In order to use "routing mode" under Xen, you'll need to change the
364 7faf5110 Michael Hanselmann
relevant parameters in the Xen config file. Under KVM instead, no config
365 7faf5110 Michael Hanselmann
change is necessary, but you still need to set up your network
366 7faf5110 Michael Hanselmann
interfaces correctly.
367 9f83899a Guido Trotter
368 9f83899a Guido Trotter
By default, under KVM, the "link" parameter you specify per-nic will
369 7faf5110 Michael Hanselmann
represent, if non-empty, a different routing table name or number to use
370 7faf5110 Michael Hanselmann
for your instances. This allows insulation between different instance
371 7faf5110 Michael Hanselmann
groups, and different routing policies between node traffic and instance
372 7faf5110 Michael Hanselmann
traffic.
373 9f83899a Guido Trotter
374 7faf5110 Michael Hanselmann
You will need to configure your routing table basic routes and rules
375 7faf5110 Michael Hanselmann
outside of ganeti. The vif scripts will only add /32 routes to your
376 7faf5110 Michael Hanselmann
instances, through their interface, in the table you specified (under
377 7faf5110 Michael Hanselmann
KVM, and in the main table under Xen).
378 9f83899a Guido Trotter
379 9f83899a Guido Trotter
.. admonition:: Bridging under Debian
380 28e15341 Iustin Pop
381 28e15341 Iustin Pop
   The recommended way to configure the Xen bridge is to edit your
382 28e15341 Iustin Pop
   ``/etc/network/interfaces`` file and substitute your normal
383 28e15341 Iustin Pop
   Ethernet stanza with the following snippet::
384 28e15341 Iustin Pop
385 28e15341 Iustin Pop
     auto xen-br0
386 28e15341 Iustin Pop
     iface xen-br0 inet static
387 28e15341 Iustin Pop
        address YOUR_IP_ADDRESS
388 28e15341 Iustin Pop
        netmask YOUR_NETMASK
389 28e15341 Iustin Pop
        network YOUR_NETWORK
390 28e15341 Iustin Pop
        broadcast YOUR_BROADCAST_ADDRESS
391 28e15341 Iustin Pop
        gateway YOUR_GATEWAY
392 28e15341 Iustin Pop
        bridge_ports eth0
393 28e15341 Iustin Pop
        bridge_stp off
394 28e15341 Iustin Pop
        bridge_fd 0
395 28e15341 Iustin Pop
396 28e15341 Iustin Pop
The following commands need to be executed on the local console:
397 28e15341 Iustin Pop
398 28e15341 Iustin Pop
  ifdown eth0
399 28e15341 Iustin Pop
  ifup xen-br0
400 28e15341 Iustin Pop
401 28e15341 Iustin Pop
To check if the bridge is setup, use the ``ip`` and ``brctl show``
402 28e15341 Iustin Pop
commands::
403 28e15341 Iustin Pop
404 28e15341 Iustin Pop
  # ip a show xen-br0
405 28e15341 Iustin Pop
  9: xen-br0: <BROADCAST,MULTICAST,UP,10000> mtu 1500 qdisc noqueue
406 28e15341 Iustin Pop
      link/ether 00:20:fc:1e:d5:5d brd ff:ff:ff:ff:ff:ff
407 28e15341 Iustin Pop
      inet 10.1.1.200/24 brd 10.1.1.255 scope global xen-br0
408 28e15341 Iustin Pop
      inet6 fe80::220:fcff:fe1e:d55d/64 scope link
409 28e15341 Iustin Pop
         valid_lft forever preferred_lft forever
410 28e15341 Iustin Pop
411 28e15341 Iustin Pop
  # brctl show xen-br0
412 28e15341 Iustin Pop
  bridge name     bridge id               STP enabled     interfaces
413 28e15341 Iustin Pop
  xen-br0         8000.0020fc1ed55d       no              eth0
414 28e15341 Iustin Pop
415 c71a1a3d Iustin Pop
.. _configure-lvm-label:
416 c71a1a3d Iustin Pop
417 28e15341 Iustin Pop
Configuring LVM
418 28e15341 Iustin Pop
+++++++++++++++
419 28e15341 Iustin Pop
420 28e15341 Iustin Pop
**Mandatory** on all nodes.
421 28e15341 Iustin Pop
422 28e15341 Iustin Pop
The volume group is required to be at least 20GiB.
423 28e15341 Iustin Pop
424 c71a1a3d Iustin Pop
If you haven't configured your LVM volume group at install time you need
425 c71a1a3d Iustin Pop
to do it before trying to initialize the Ganeti cluster. This is done by
426 c71a1a3d Iustin Pop
formatting the devices/partitions you want to use for it and then adding
427 c71a1a3d Iustin Pop
them to the relevant volume group::
428 28e15341 Iustin Pop
429 28e15341 Iustin Pop
  pvcreate /dev/sda3
430 28e15341 Iustin Pop
  vgcreate xenvg /dev/sda3
431 28e15341 Iustin Pop
432 28e15341 Iustin Pop
or::
433 28e15341 Iustin Pop
434 28e15341 Iustin Pop
  pvcreate /dev/sdb1
435 28e15341 Iustin Pop
  pvcreate /dev/sdc1
436 28e15341 Iustin Pop
  vgcreate xenvg /dev/sdb1 /dev/sdc1
437 28e15341 Iustin Pop
438 28e15341 Iustin Pop
If you want to add a device later you can do so with the *vgextend*
439 28e15341 Iustin Pop
command::
440 28e15341 Iustin Pop
441 28e15341 Iustin Pop
  pvcreate /dev/sdd1
442 28e15341 Iustin Pop
  vgextend xenvg /dev/sdd1
443 28e15341 Iustin Pop
444 28e15341 Iustin Pop
Optional: it is recommended to configure LVM not to scan the DRBD
445 28e15341 Iustin Pop
devices for physical volumes. This can be accomplished by editing
446 c71a1a3d Iustin Pop
``/etc/lvm/lvm.conf`` and adding the ``/dev/drbd[0-9]+`` regular
447 c71a1a3d Iustin Pop
expression to the ``filter`` variable, like this::
448 28e15341 Iustin Pop
449 28e15341 Iustin Pop
  filter = ["r|/dev/cdrom|", "r|/dev/drbd[0-9]+|" ]
450 28e15341 Iustin Pop
451 c71a1a3d Iustin Pop
Note that with Ganeti a helper script is provided - ``lvmstrap`` which
452 c71a1a3d Iustin Pop
will erase and configure as LVM any not in-use disk on your system. This
453 c71a1a3d Iustin Pop
is dangerous and it's recommended to read its ``--help`` output if you
454 c71a1a3d Iustin Pop
want to use it.
455 c71a1a3d Iustin Pop
456 28e15341 Iustin Pop
Installing Ganeti
457 28e15341 Iustin Pop
+++++++++++++++++
458 28e15341 Iustin Pop
459 28e15341 Iustin Pop
**Mandatory** on all nodes.
460 28e15341 Iustin Pop
461 28e15341 Iustin Pop
It's now time to install the Ganeti software itself.  Download the
462 28e15341 Iustin Pop
source from the project page at `<http://code.google.com/p/ganeti/>`_,
463 28e15341 Iustin Pop
and install it (replace 2.0.0 with the latest version)::
464 28e15341 Iustin Pop
465 28e15341 Iustin Pop
  tar xvzf ganeti-2.0.0.tar.gz
466 28e15341 Iustin Pop
  cd ganeti-2.0.0
467 28e15341 Iustin Pop
  ./configure --localstatedir=/var --sysconfdir=/etc
468 28e15341 Iustin Pop
  make
469 28e15341 Iustin Pop
  make install
470 28e15341 Iustin Pop
  mkdir /srv/ganeti/ /srv/ganeti/os /srv/ganeti/export
471 28e15341 Iustin Pop
472 c71a1a3d Iustin Pop
You also need to copy the file ``doc/examples/ganeti.initd`` from the
473 c71a1a3d Iustin Pop
source archive to ``/etc/init.d/ganeti`` and register it with your
474 28e15341 Iustin Pop
distribution's startup scripts, for example in Debian::
475 28e15341 Iustin Pop
476 28e15341 Iustin Pop
  update-rc.d ganeti defaults 20 80
477 28e15341 Iustin Pop
478 c71a1a3d Iustin Pop
In order to automatically restart failed instances, you need to setup a
479 c71a1a3d Iustin Pop
cron job run the *ganeti-watcher* command. A sample cron file is
480 c71a1a3d Iustin Pop
provided in the source at ``doc/examples/ganeti.cron`` and you can copy
481 c71a1a3d Iustin Pop
that (eventually altering the path) to ``/etc/cron.d/ganeti``.
482 c71a1a3d Iustin Pop
483 c71a1a3d Iustin Pop
What gets installed
484 c71a1a3d Iustin Pop
~~~~~~~~~~~~~~~~~~~
485 c71a1a3d Iustin Pop
486 c71a1a3d Iustin Pop
The above ``make install`` invocation, or installing via your
487 c71a1a3d Iustin Pop
distribution mechanisms, will install on the system:
488 c71a1a3d Iustin Pop
489 c71a1a3d Iustin Pop
- a set of python libraries under the *ganeti* namespace (depending on
490 c71a1a3d Iustin Pop
  the python version this can be located in either
491 c71a1a3d Iustin Pop
  ``lib/python-$ver/site-packages`` or various other locations)
492 c71a1a3d Iustin Pop
- a set of programs under ``/usr/local/sbin`` or ``/usr/sbin``
493 c71a1a3d Iustin Pop
- man pages for the above programs
494 c71a1a3d Iustin Pop
- a set of tools under the ``lib/ganeti/tools`` directory
495 c71a1a3d Iustin Pop
- an example iallocator script (see the admin guide for details) under
496 c71a1a3d Iustin Pop
  ``lib/ganeti/iallocators``
497 c71a1a3d Iustin Pop
- a cron job that is needed for cluster maintenance
498 c71a1a3d Iustin Pop
- an init script for automatic startup of Ganeti daemons
499 c71a1a3d Iustin Pop
- provided but not installed automatically by ``make install`` is a bash
500 c71a1a3d Iustin Pop
  completion script that hopefully will ease working with the many
501 c71a1a3d Iustin Pop
  cluster commands
502 28e15341 Iustin Pop
503 28e15341 Iustin Pop
Installing the Operating System support packages
504 28e15341 Iustin Pop
++++++++++++++++++++++++++++++++++++++++++++++++
505 28e15341 Iustin Pop
506 28e15341 Iustin Pop
**Mandatory** on all nodes.
507 28e15341 Iustin Pop
508 28e15341 Iustin Pop
To be able to install instances you need to have an Operating System
509 28e15341 Iustin Pop
installation script. An example OS that works under Debian and can
510 28e15341 Iustin Pop
install Debian and Ubuntu instace OSes is provided on the project web
511 c71a1a3d Iustin Pop
site.  Download it from the project page and follow the instructions in
512 c71a1a3d Iustin Pop
the ``README`` file.  Here is the installation procedure (replace 0.7
513 c71a1a3d Iustin Pop
with the latest version that is compatible with your ganeti version)::
514 28e15341 Iustin Pop
515 28e15341 Iustin Pop
  cd /usr/local/src/
516 28e15341 Iustin Pop
  wget http://ganeti.googlecode.com/files/ganeti-instance-debootstrap-0.7.tar.gz
517 28e15341 Iustin Pop
  tar xzf ganeti-instance-debootstrap-0.7.tar.gz
518 28e15341 Iustin Pop
  cd ganeti-instance-debootstrap-0.7
519 28e15341 Iustin Pop
  ./configure
520 28e15341 Iustin Pop
  make
521 28e15341 Iustin Pop
  make install
522 28e15341 Iustin Pop
523 28e15341 Iustin Pop
In order to use this OS definition, you need to have internet access
524 28e15341 Iustin Pop
from your nodes and have the *debootstrap*, *dump* and *restore*
525 28e15341 Iustin Pop
commands installed on all nodes. Also, if the OS is configured to
526 28e15341 Iustin Pop
partition the instance's disk in
527 28e15341 Iustin Pop
``/etc/default/ganeti-instance-debootstrap``, you will need *kpartx*
528 28e15341 Iustin Pop
installed.
529 28e15341 Iustin Pop
530 28e15341 Iustin Pop
.. admonition:: Debian
531 28e15341 Iustin Pop
532 28e15341 Iustin Pop
   Use this command on all nodes to install the required packages::
533 28e15341 Iustin Pop
534 28e15341 Iustin Pop
     apt-get install debootstrap dump kpartx
535 28e15341 Iustin Pop
536 28e15341 Iustin Pop
Alternatively, you can create your own OS definitions. See the manpage
537 fd07c6b3 Iustin Pop
:manpage:`ganeti-os-interface`.
538 28e15341 Iustin Pop
539 28e15341 Iustin Pop
Initializing the cluster
540 28e15341 Iustin Pop
++++++++++++++++++++++++
541 28e15341 Iustin Pop
542 c71a1a3d Iustin Pop
**Mandatory** once per cluster, on the first node.
543 28e15341 Iustin Pop
544 c71a1a3d Iustin Pop
The last step is to initialize the cluster. After you have repeated the
545 28e15341 Iustin Pop
above process on all of your nodes, choose one as the master, and
546 28e15341 Iustin Pop
execute::
547 28e15341 Iustin Pop
548 28e15341 Iustin Pop
  gnt-cluster init <CLUSTERNAME>
549 28e15341 Iustin Pop
550 c71a1a3d Iustin Pop
The *CLUSTERNAME* is a hostname, which must be resolvable (e.g. it must
551 c71a1a3d Iustin Pop
exist in DNS or in ``/etc/hosts``) by all the nodes in the cluster. You
552 c71a1a3d Iustin Pop
must choose a name different from any of the nodes names for a
553 c71a1a3d Iustin Pop
multi-node cluster. In general the best choice is to have a unique name
554 c71a1a3d Iustin Pop
for a cluster, even if it consists of only one machine, as you will be
555 c71a1a3d Iustin Pop
able to expand it later without any problems. Please note that the
556 c71a1a3d Iustin Pop
hostname used for this must resolve to an IP address reserved
557 28e15341 Iustin Pop
**exclusively** for this purpose, and cannot be the name of the first
558 28e15341 Iustin Pop
(master) node.
559 28e15341 Iustin Pop
560 7faf5110 Michael Hanselmann
If you want to use a bridge which is not ``xen-br0``, or no bridge at
561 7faf5110 Michael Hanselmann
all, use ``--nicparams``.
562 9f83899a Guido Trotter
563 28e15341 Iustin Pop
If the bridge name you are using is not ``xen-br0``, use the *-b
564 28e15341 Iustin Pop
<BRIDGENAME>* option to specify the bridge name. In this case, you
565 c71a1a3d Iustin Pop
should also use the *--master-netdev <BRIDGENAME>* option with the same
566 c71a1a3d Iustin Pop
BRIDGENAME argument.
567 28e15341 Iustin Pop
568 28e15341 Iustin Pop
You can use a different name than ``xenvg`` for the volume group (but
569 28e15341 Iustin Pop
note that the name must be identical on all nodes). In this case you
570 c71a1a3d Iustin Pop
need to specify it by passing the *-g <VGNAME>* option to ``gnt-cluster
571 c71a1a3d Iustin Pop
init``.
572 28e15341 Iustin Pop
573 c71a1a3d Iustin Pop
To set up the cluster as an Xen HVM cluster, use the
574 28e15341 Iustin Pop
``--enabled-hypervisors=xen-hvm`` option to enable the HVM hypervisor
575 c71a1a3d Iustin Pop
(you can also add ``,xen-pvm`` to enable the PVM one too). You will also
576 c71a1a3d Iustin Pop
need to create the VNC cluster password file
577 28e15341 Iustin Pop
``/etc/ganeti/vnc-cluster-password`` which contains one line with the
578 28e15341 Iustin Pop
default VNC password for the cluster.
579 28e15341 Iustin Pop
580 28e15341 Iustin Pop
To setup the cluster for KVM-only usage (KVM and Xen cannot be mixed),
581 28e15341 Iustin Pop
pass ``--enabled-hypervisors=kvm`` to the init command.
582 28e15341 Iustin Pop
583 28e15341 Iustin Pop
You can also invoke the command with the ``--help`` option in order to
584 28e15341 Iustin Pop
see all the possibilities.
585 28e15341 Iustin Pop
586 28e15341 Iustin Pop
Joining the nodes to the cluster
587 28e15341 Iustin Pop
++++++++++++++++++++++++++++++++
588 28e15341 Iustin Pop
589 28e15341 Iustin Pop
**Mandatory** for all the other nodes.
590 28e15341 Iustin Pop
591 c71a1a3d Iustin Pop
After you have initialized your cluster you need to join the other nodes
592 c71a1a3d Iustin Pop
to it. You can do so by executing the following command on the master
593 c71a1a3d Iustin Pop
node::
594 28e15341 Iustin Pop
595 28e15341 Iustin Pop
  gnt-node add <NODENAME>
596 28e15341 Iustin Pop
597 28e15341 Iustin Pop
Separate replication network
598 28e15341 Iustin Pop
++++++++++++++++++++++++++++
599 28e15341 Iustin Pop
600 28e15341 Iustin Pop
**Optional**
601 28e15341 Iustin Pop
602 28e15341 Iustin Pop
Ganeti uses DRBD to mirror the disk of the virtual instances between
603 28e15341 Iustin Pop
nodes. To use a dedicated network interface for this (in order to
604 28e15341 Iustin Pop
improve performance or to enhance security) you need to configure an
605 28e15341 Iustin Pop
additional interface for each node.  Use the *-s* option with
606 28e15341 Iustin Pop
``gnt-cluster init`` and ``gnt-node add`` to specify the IP address of
607 28e15341 Iustin Pop
this secondary interface to use for each node. Note that if you
608 c71a1a3d Iustin Pop
specified this option at cluster setup time, you must afterwards use it
609 c71a1a3d Iustin Pop
for every node add operation.
610 28e15341 Iustin Pop
611 28e15341 Iustin Pop
Testing the setup
612 28e15341 Iustin Pop
+++++++++++++++++
613 28e15341 Iustin Pop
614 c71a1a3d Iustin Pop
Execute the ``gnt-node list`` command to see all nodes in the cluster::
615 28e15341 Iustin Pop
616 28e15341 Iustin Pop
  # gnt-node list
617 28e15341 Iustin Pop
  Node              DTotal  DFree MTotal MNode MFree Pinst Sinst
618 28e15341 Iustin Pop
  node1.example.com 197404 197404   2047  1896   125     0     0
619 28e15341 Iustin Pop
620 c71a1a3d Iustin Pop
The above shows a couple of things:
621 28e15341 Iustin Pop
622 c71a1a3d Iustin Pop
- The various Ganeti daemons can talk to each other
623 c71a1a3d Iustin Pop
- Ganeti can examine the storage of the node (DTotal/DFree)
624 c71a1a3d Iustin Pop
- Ganeti can talk to the selected hypervisor (MTotal/MNode/MFree)
625 28e15341 Iustin Pop
626 c71a1a3d Iustin Pop
Cluster burnin
627 c71a1a3d Iustin Pop
~~~~~~~~~~~~~~
628 28e15341 Iustin Pop
629 c71a1a3d Iustin Pop
With Ganeti a tool called :command:`burnin` is provided that can test
630 c71a1a3d Iustin Pop
most of the Ganeti functionality. The tool is installed under the
631 c71a1a3d Iustin Pop
``lib/ganeti/tools`` directory (either under ``/usr`` or ``/usr/local``
632 c71a1a3d Iustin Pop
based on the installation method). See more details under
633 c71a1a3d Iustin Pop
:ref:`burnin-label`.
634 28e15341 Iustin Pop
635 c71a1a3d Iustin Pop
Further steps
636 c71a1a3d Iustin Pop
-------------
637 28e15341 Iustin Pop
638 c71a1a3d Iustin Pop
You can now proceed either to the :doc:`admin`, or read the manpages of
639 c71a1a3d Iustin Pop
the various commands (:manpage:`ganeti(7)`, :manpage:`gnt-cluster(8)`,
640 c71a1a3d Iustin Pop
:manpage:`gnt-node(8)`, :manpage:`gnt-instance(8)`,
641 c71a1a3d Iustin Pop
:manpage:`gnt-job(8)`).
642 28e15341 Iustin Pop
643 c71a1a3d Iustin Pop
.. rubric:: Footnotes
644 28e15341 Iustin Pop
645 c71a1a3d Iustin Pop
.. [#defkernel] The kernel and initrd paths can be changed at either
646 c71a1a3d Iustin Pop
   cluster level (which changes the default for all instances) or at
647 c71a1a3d Iustin Pop
   instance level.
648 558fd122 Michael Hanselmann
649 558fd122 Michael Hanselmann
.. vim: set textwidth=72 :
650 c71a1a3d Iustin Pop
.. Local Variables:
651 c71a1a3d Iustin Pop
.. mode: rst
652 c71a1a3d Iustin Pop
.. fill-column: 72
653 c71a1a3d Iustin Pop
.. End: