root / doc / install.rst @ a295eb80
History | View | Annotate | Download (28.7 kB)
1 |
Ganeti installation tutorial |
---|---|
2 |
============================ |
3 |
|
4 |
Documents Ganeti version |version| |
5 |
|
6 |
.. contents:: |
7 |
|
8 |
.. highlight:: shell-example |
9 |
|
10 |
Introduction |
11 |
------------ |
12 |
|
13 |
Ganeti is a cluster virtualization management system based on Xen or |
14 |
KVM. This document explains how to bootstrap a Ganeti node (Xen *dom0*, |
15 |
the host Linux system for KVM), create a running cluster and install |
16 |
virtual instances (Xen *domUs*, KVM guests). You need to repeat most of |
17 |
the steps in this document for every node you want to install, but of |
18 |
course we recommend creating some semi-automatic procedure if you plan |
19 |
to deploy Ganeti on a medium/large scale. |
20 |
|
21 |
A basic Ganeti terminology glossary is provided in the introductory |
22 |
section of the :doc:`admin`. Please refer to that document if you are |
23 |
uncertain about the terms we are using. |
24 |
|
25 |
Ganeti has been developed for Linux and should be distribution-agnostic. |
26 |
This documentation will use Debian Squeeze as an example system but the |
27 |
examples can be translated to any other distribution. You are expected |
28 |
to be familiar with your distribution, its package management system, |
29 |
and Xen or KVM before trying to use Ganeti. |
30 |
|
31 |
This document is divided into two main sections: |
32 |
|
33 |
- Installation of the base system and base components |
34 |
|
35 |
- Configuration of the environment for Ganeti |
36 |
|
37 |
Each of these is divided into sub-sections. While a full Ganeti system |
38 |
will need all of the steps specified, some are not strictly required for |
39 |
every environment. Which ones they are, and why, is specified in the |
40 |
corresponding sections. |
41 |
|
42 |
Installing the base system and base components |
43 |
---------------------------------------------- |
44 |
|
45 |
Hardware requirements |
46 |
+++++++++++++++++++++ |
47 |
|
48 |
Any system supported by your Linux distribution is fine. 64-bit systems |
49 |
are better as they can support more memory. |
50 |
|
51 |
Any disk drive recognized by Linux (``IDE``/``SCSI``/``SATA``/etc.) is |
52 |
supported in Ganeti. Note that no shared storage (e.g. ``SAN``) is |
53 |
needed to get high-availability features (but of course, one can be used |
54 |
to store the images). Whilte it is highly recommended to use more than |
55 |
one disk drive in order to improve speed, Ganeti also works with one |
56 |
disk per machine. |
57 |
|
58 |
Installing the base system |
59 |
++++++++++++++++++++++++++ |
60 |
|
61 |
**Mandatory** on all nodes. |
62 |
|
63 |
It is advised to start with a clean, minimal install of the operating |
64 |
system. The only requirement you need to be aware of at this stage is to |
65 |
partition leaving enough space for a big (**minimum** 20GiB) LVM volume |
66 |
group which will then host your instance filesystems, if you want to use |
67 |
all Ganeti features. The volume group name Ganeti uses (by default) is |
68 |
``xenvg``. |
69 |
|
70 |
You can also use file-based storage only, without LVM, but this setup is |
71 |
not detailed in this document. |
72 |
|
73 |
If you choose to use RBD-based instances, there's no need for LVM |
74 |
provisioning. However, this feature is experimental, and is not yet |
75 |
recommended for production clusters. |
76 |
|
77 |
While you can use an existing system, please note that the Ganeti |
78 |
installation is intrusive in terms of changes to the system |
79 |
configuration, and it's best to use a newly-installed system without |
80 |
important data on it. |
81 |
|
82 |
Also, for best results, it's advised that the nodes have as much as |
83 |
possible the same hardware and software configuration. This will make |
84 |
administration much easier. |
85 |
|
86 |
Hostname issues |
87 |
~~~~~~~~~~~~~~~ |
88 |
|
89 |
Note that Ganeti requires the hostnames of the systems (i.e. what the |
90 |
``hostname`` command outputs to be a fully-qualified name, not a short |
91 |
name. In other words, you should use *node1.example.com* as a hostname |
92 |
and not just *node1*. |
93 |
|
94 |
.. admonition:: Debian |
95 |
|
96 |
Debian usually configures the hostname differently than you need it |
97 |
for Ganeti. For example, this is what it puts in ``/etc/hosts`` in |
98 |
certain situations:: |
99 |
|
100 |
127.0.0.1 localhost |
101 |
127.0.1.1 node1.example.com node1 |
102 |
|
103 |
but for Ganeti you need to have:: |
104 |
|
105 |
127.0.0.1 localhost |
106 |
192.0.2.1 node1.example.com node1 |
107 |
|
108 |
replacing ``192.0.2.1`` with your node's address. Also, the file |
109 |
``/etc/hostname`` which configures the hostname of the system |
110 |
should contain ``node1.example.com`` and not just ``node1`` (you |
111 |
need to run the command ``/etc/init.d/hostname.sh start`` after |
112 |
changing the file). |
113 |
|
114 |
.. admonition:: Why a fully qualified host name |
115 |
|
116 |
Although most distributions use only the short name in the |
117 |
/etc/hostname file, we still think Ganeti nodes should use the full |
118 |
name. The reason for this is that calling 'hostname --fqdn' requires |
119 |
the resolver library to work and is a 'guess' via heuristics at what |
120 |
is your domain name. Since Ganeti can be used among other things to |
121 |
host DNS servers, we don't want to depend on them as much as |
122 |
possible, and we'd rather have the uname() syscall return the full |
123 |
node name. |
124 |
|
125 |
We haven't ever found any breakage in using a full hostname on a |
126 |
Linux system, and anyway we recommend to have only a minimal |
127 |
installation on Ganeti nodes, and to use instances (or other |
128 |
dedicated machines) to run the rest of your network services. By |
129 |
doing this you can change the /etc/hostname file to contain an FQDN |
130 |
without the fear of breaking anything unrelated. |
131 |
|
132 |
|
133 |
Installing The Hypervisor |
134 |
+++++++++++++++++++++++++ |
135 |
|
136 |
**Mandatory** on all nodes. |
137 |
|
138 |
While Ganeti is developed with the ability to modularly run on different |
139 |
virtualization environments in mind the only two currently useable on a |
140 |
live system are Xen and KVM. Supported Xen versions are: 3.0.3 and later |
141 |
3.x versions, and 4.x (tested up to 4.1). Supported KVM versions are 72 |
142 |
and above. |
143 |
|
144 |
Please follow your distribution's recommended way to install and set up |
145 |
Xen, or install Xen from the upstream source, if you wish, following |
146 |
their manual. For KVM, make sure you have a KVM-enabled kernel and the |
147 |
KVM tools. |
148 |
|
149 |
After installing Xen, you need to reboot into your new system. On some |
150 |
distributions this might involve configuring GRUB appropriately, whereas |
151 |
others will configure it automatically when you install the respective |
152 |
kernels. For KVM no reboot should be necessary. |
153 |
|
154 |
.. admonition:: Xen on Debian |
155 |
|
156 |
Under Debian you can install the relevant ``xen-linux-system`` |
157 |
package, which will pull in both the hypervisor and the relevant |
158 |
kernel. Also, if you are installing a 32-bit system, you should |
159 |
install the ``libc6-xen`` package (run ``apt-get install |
160 |
libc6-xen``). |
161 |
|
162 |
Xen settings |
163 |
~~~~~~~~~~~~ |
164 |
|
165 |
It's recommended that dom0 is restricted to a low amount of memory |
166 |
(512MiB or 1GiB is reasonable) and that memory ballooning is disabled in |
167 |
the file ``/etc/xen/xend-config.sxp`` by setting the value |
168 |
``dom0-min-mem`` to 0, like this:: |
169 |
|
170 |
(dom0-min-mem 0) |
171 |
|
172 |
For optimum performance when running both CPU and I/O intensive |
173 |
instances, it's also recommended that the dom0 is restricted to one CPU |
174 |
only. For example you can add ``dom0_max_vcpus=1,dom0_vcpus_pin`` to your |
175 |
kernels boot command line and set ``dom0-cpus`` in |
176 |
``/etc/xen/xend-config.sxp`` like this:: |
177 |
|
178 |
(dom0-cpus 1) |
179 |
|
180 |
It is recommended that you disable xen's automatic save of virtual |
181 |
machines at system shutdown and subsequent restore of them at reboot. |
182 |
To obtain this make sure the variable ``XENDOMAINS_SAVE`` in the file |
183 |
``/etc/default/xendomains`` is set to an empty value. |
184 |
|
185 |
If you want to use live migration make sure you have, in the xen config |
186 |
file, something that allows the nodes to migrate instances between each |
187 |
other. For example: |
188 |
|
189 |
.. code-block:: text |
190 |
|
191 |
(xend-relocation-server yes) |
192 |
(xend-relocation-port 8002) |
193 |
(xend-relocation-address '') |
194 |
(xend-relocation-hosts-allow '^192\\.0\\.2\\.[0-9]+$') |
195 |
|
196 |
|
197 |
The second line assumes that the hypervisor parameter |
198 |
``migration_port`` is set 8002, otherwise modify it to match. The last |
199 |
line assumes that all your nodes have secondary IPs in the |
200 |
192.0.2.0/24 network, adjust it accordingly to your setup. |
201 |
|
202 |
If you want to run HVM instances too with Ganeti and want VNC access to |
203 |
the console of your instances, set the following two entries in |
204 |
``/etc/xen/xend-config.sxp``: |
205 |
|
206 |
.. code-block:: text |
207 |
|
208 |
(vnc-listen '0.0.0.0') (vncpasswd '') |
209 |
|
210 |
You need to restart the Xen daemon for these settings to take effect:: |
211 |
|
212 |
$ /etc/init.d/xend restart |
213 |
|
214 |
Selecting the instance kernel |
215 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
216 |
|
217 |
After you have installed Xen, you need to tell Ganeti exactly what |
218 |
kernel to use for the instances it will create. This is done by creating |
219 |
a symlink from your actual kernel to ``/boot/vmlinuz-3-xenU``, and one |
220 |
from your initrd to ``/boot/initrd-3-xenU`` [#defkernel]_. Note that |
221 |
if you don't use an initrd for the domU kernel, you don't need to create |
222 |
the initrd symlink. |
223 |
|
224 |
.. admonition:: Debian |
225 |
|
226 |
After installation of the ``xen-linux-system`` package, you need to |
227 |
run (replace the exact version number with the one you have):: |
228 |
|
229 |
$ cd /boot |
230 |
$ ln -s vmlinuz-%2.6.26-1%-xen-amd64 vmlinuz-3-xenU |
231 |
$ ln -s initrd.img-%2.6.26-1%-xen-amd64 initrd-3-xenU |
232 |
|
233 |
By default, the initrd doesn't contain the Xen block drivers needed |
234 |
to mount the root device, so it is recommended to update the initrd |
235 |
by following these two steps: |
236 |
|
237 |
- edit ``/etc/initramfs-tools/modules`` and add ``xen_blkfront`` |
238 |
- run ``update-initramfs -u`` |
239 |
|
240 |
Installing DRBD |
241 |
+++++++++++++++ |
242 |
|
243 |
Recommended on all nodes: DRBD_ is required if you want to use the high |
244 |
availability (HA) features of Ganeti, but optional if you don't require |
245 |
them or only run Ganeti on single-node clusters. You can upgrade a |
246 |
non-HA cluster to an HA one later, but you might need to convert all |
247 |
your instances to DRBD to take advantage of the new features. |
248 |
|
249 |
.. _DRBD: http://www.drbd.org/ |
250 |
|
251 |
Supported DRBD versions: 8.0-8.3. It's recommended to have at least |
252 |
version 8.0.12. Note that for version 8.2 and newer it is needed to pass |
253 |
the ``usermode_helper=/bin/true`` parameter to the module, either by |
254 |
configuring ``/etc/modules`` or when inserting it manually. |
255 |
|
256 |
Now the bad news: unless your distribution already provides it |
257 |
installing DRBD might involve recompiling your kernel or anyway fiddling |
258 |
with it. Hopefully at least the Xen-ified kernel source to start from |
259 |
will be provided (if you intend to use Xen). |
260 |
|
261 |
The good news is that you don't need to configure DRBD at all. Ganeti |
262 |
will do it for you for every instance you set up. If you have the DRBD |
263 |
utils installed and the module in your kernel you're fine. Please check |
264 |
that your system is configured to load the module at every boot, and |
265 |
that it passes the following option to the module: |
266 |
``minor_count=NUMBER``. We recommend that you use 128 as the value of |
267 |
the minor_count - this will allow you to use up to 64 instances in total |
268 |
per node (both primary and secondary, when using only one disk per |
269 |
instance). You can increase the number up to 255 if you need more |
270 |
instances on a node. |
271 |
|
272 |
|
273 |
.. admonition:: Debian |
274 |
|
275 |
On Debian, you can just install (build) the DRBD module with the |
276 |
following commands, making sure you are running the target (Xen or |
277 |
KVM) kernel:: |
278 |
|
279 |
$ apt-get install drbd8-source drbd8-utils |
280 |
$ m-a update |
281 |
$ m-a a-i drbd8 |
282 |
|
283 |
Or on newer versions, if the kernel already has modules: |
284 |
|
285 |
$ apt-get install drbd8-utils |
286 |
|
287 |
Then to configure it for Ganeti:: |
288 |
|
289 |
$ echo drbd minor_count=128 usermode_helper=/bin/true >> /etc/modules |
290 |
$ depmod -a |
291 |
$ modprobe drbd minor_count=128 usermode_helper=/bin/true |
292 |
|
293 |
It is also recommended that you comment out the default resources (if any) |
294 |
in the ``/etc/drbd.conf`` file, so that the init script doesn't try to |
295 |
configure any drbd devices. You can do this by prefixing all |
296 |
*resource* lines in the file with the keyword *skip*, like this: |
297 |
|
298 |
.. code-block:: text |
299 |
|
300 |
skip { |
301 |
resource r0 { |
302 |
... |
303 |
} |
304 |
} |
305 |
|
306 |
skip { |
307 |
resource "r1" { |
308 |
... |
309 |
} |
310 |
} |
311 |
|
312 |
Installing RBD |
313 |
++++++++++++++ |
314 |
|
315 |
Recommended on all nodes: RBD_ is required if you want to create |
316 |
instances with RBD disks residing inside a RADOS cluster (make use of |
317 |
the rbd disk template). RBD-based instances can failover or migrate to |
318 |
any other node in the ganeti cluster, enabling you to exploit of all |
319 |
Ganeti's high availabilily (HA) features. |
320 |
|
321 |
.. attention:: |
322 |
Be careful though: rbd is still experimental! For now it is |
323 |
recommended only for testing purposes. No sensitive data should be |
324 |
stored there. |
325 |
|
326 |
.. _RBD: http://ceph.newdream.net/ |
327 |
|
328 |
You will need the ``rbd`` and ``libceph`` kernel modules, the RBD/Ceph |
329 |
userspace utils (ceph-common Debian package) and an appropriate |
330 |
Ceph/RADOS configuration file on every VM-capable node. |
331 |
|
332 |
You will also need a working RADOS Cluster accessible by the above |
333 |
nodes. |
334 |
|
335 |
RADOS Cluster |
336 |
~~~~~~~~~~~~~ |
337 |
|
338 |
You will need a working RADOS Cluster accesible by all VM-capable nodes |
339 |
to use the RBD template. For more information on setting up a RADOS |
340 |
Cluster, refer to the `official docs <http://ceph.newdream.net/>`_. |
341 |
|
342 |
If you want to use a pool for storing RBD disk images other than the |
343 |
default (``rbd``), you should first create the pool in the RADOS |
344 |
Cluster, and then set the corresponding rbd disk parameter named |
345 |
``pool``. |
346 |
|
347 |
Kernel Modules |
348 |
~~~~~~~~~~~~~~ |
349 |
|
350 |
Unless your distribution already provides it, you might need to compile |
351 |
the ``rbd`` and ``libceph`` modules from source. You will need Linux |
352 |
Kernel 3.2 or above for the kernel modules. Alternatively you will have |
353 |
to build them as external modules (from Linux Kernel source 3.2 or |
354 |
above), if you want to run a less recent kernel, or your kernel doesn't |
355 |
include them. |
356 |
|
357 |
Userspace Utils |
358 |
~~~~~~~~~~~~~~~ |
359 |
|
360 |
The RBD template has been tested with ``ceph-common`` v0.38 and |
361 |
above. We recommend using the latest version of ``ceph-common``. |
362 |
|
363 |
.. admonition:: Debian |
364 |
|
365 |
On Debian, you can just install the RBD/Ceph userspace utils with |
366 |
the following command:: |
367 |
|
368 |
$ apt-get install ceph-common |
369 |
|
370 |
Configuration file |
371 |
~~~~~~~~~~~~~~~~~~ |
372 |
|
373 |
You should also provide an appropriate configuration file |
374 |
(``ceph.conf``) in ``/etc/ceph``. For the rbd userspace utils, you'll |
375 |
only need to specify the IP addresses of the RADOS Cluster monitors. |
376 |
|
377 |
.. admonition:: ceph.conf |
378 |
|
379 |
Sample configuration file: |
380 |
|
381 |
.. code-block:: text |
382 |
|
383 |
[mon.a] |
384 |
host = example_monitor_host1 |
385 |
mon addr = 1.2.3.4:6789 |
386 |
[mon.b] |
387 |
host = example_monitor_host2 |
388 |
mon addr = 1.2.3.5:6789 |
389 |
[mon.c] |
390 |
host = example_monitor_host3 |
391 |
mon addr = 1.2.3.6:6789 |
392 |
|
393 |
For more information, please see the `Ceph Docs |
394 |
<http://ceph.newdream.net/docs/latest/>`_ |
395 |
|
396 |
Other required software |
397 |
+++++++++++++++++++++++ |
398 |
|
399 |
Please install all software requirements mentioned in :doc:`install-quick`. |
400 |
If you want to build Ganeti from source, don't forget to follow the steps |
401 |
required for that as well. |
402 |
|
403 |
Setting up the environment for Ganeti |
404 |
------------------------------------- |
405 |
|
406 |
Configuring the network |
407 |
+++++++++++++++++++++++ |
408 |
|
409 |
**Mandatory** on all nodes. |
410 |
|
411 |
You can run Ganeti either in "bridged mode", "routed mode" or |
412 |
"openvswitch mode". In bridged mode, the default, the instances network |
413 |
interfaces will be attached to a software bridge running in dom0. Xen by |
414 |
default creates such a bridge at startup, but your distribution might |
415 |
have a different way to do things, and you'll definitely need to |
416 |
manually set it up under KVM. |
417 |
|
418 |
Beware that the default name Ganeti uses is ``xen-br0`` (which was used |
419 |
in Xen 2.0) while Xen 3.0 uses ``xenbr0`` by default. See the |
420 |
`Initializing the cluster`_ section to learn how to choose a different |
421 |
bridge, or not to use one at all and use "routed mode". |
422 |
|
423 |
In order to use "routed mode" under Xen, you'll need to change the |
424 |
relevant parameters in the Xen config file. Under KVM instead, no config |
425 |
change is necessary, but you still need to set up your network |
426 |
interfaces correctly. |
427 |
|
428 |
By default, under KVM, the "link" parameter you specify per-nic will |
429 |
represent, if non-empty, a different routing table name or number to use |
430 |
for your instances. This allows isolation between different instance |
431 |
groups, and different routing policies between node traffic and instance |
432 |
traffic. |
433 |
|
434 |
You will need to configure your routing table basic routes and rules |
435 |
outside of ganeti. The vif scripts will only add /32 routes to your |
436 |
instances, through their interface, in the table you specified (under |
437 |
KVM, and in the main table under Xen). |
438 |
|
439 |
Also for "openvswitch mode" under Xen a custom network script is needed. |
440 |
Under KVM everything should work, but you'll need to configure your |
441 |
switches outside of Ganeti (as for bridges). |
442 |
|
443 |
.. admonition:: Bridging issues with certain kernels |
444 |
|
445 |
Some kernel versions (e.g. 2.6.32) have an issue where the bridge |
446 |
will automatically change its ``MAC`` address to the lower-numbered |
447 |
slave on port addition and removal. This means that, depending on |
448 |
the ``MAC`` address of the actual NIC on the node and the addresses |
449 |
of the instances, it could be that starting, stopping or migrating |
450 |
instances will lead to timeouts due to the address of the bridge |
451 |
(and thus node itself) changing. |
452 |
|
453 |
To prevent this, it's enough to set the bridge manually to a |
454 |
specific ``MAC`` address, which will disable this automatic address |
455 |
change. In Debian, this can be done as follows in the bridge |
456 |
configuration snippet:: |
457 |
|
458 |
up ip link set addr $(cat /sys/class/net/$IFACE/address) dev $IFACE |
459 |
|
460 |
which will "set" the bridge address to the initial one, disallowing |
461 |
changes. |
462 |
|
463 |
.. admonition:: Bridging under Debian |
464 |
|
465 |
The recommended way to configure the Xen bridge is to edit your |
466 |
``/etc/network/interfaces`` file and substitute your normal |
467 |
Ethernet stanza with the following snippet:: |
468 |
|
469 |
auto xen-br0 |
470 |
iface xen-br0 inet static |
471 |
address %YOUR_IP_ADDRESS% |
472 |
netmask %YOUR_NETMASK% |
473 |
network %YOUR_NETWORK% |
474 |
broadcast %YOUR_BROADCAST_ADDRESS% |
475 |
gateway %YOUR_GATEWAY% |
476 |
bridge_ports eth0 |
477 |
bridge_stp off |
478 |
bridge_fd 0 |
479 |
# example for setting manually the bridge address to the eth0 NIC |
480 |
up ip link set addr $(cat /sys/class/net/eth0/address) dev $IFACE |
481 |
|
482 |
The following commands need to be executed on the local console:: |
483 |
|
484 |
$ ifdown eth0 |
485 |
$ ifup xen-br0 |
486 |
|
487 |
To check if the bridge is setup, use the ``ip`` and ``brctl show`` |
488 |
commands:: |
489 |
|
490 |
$ ip a show xen-br0 |
491 |
9: xen-br0: <BROADCAST,MULTICAST,UP,10000> mtu 1500 qdisc noqueue |
492 |
link/ether 00:20:fc:1e:d5:5d brd ff:ff:ff:ff:ff:ff |
493 |
inet 10.1.1.200/24 brd 10.1.1.255 scope global xen-br0 |
494 |
inet6 fe80::220:fcff:fe1e:d55d/64 scope link |
495 |
valid_lft forever preferred_lft forever |
496 |
|
497 |
$ brctl show xen-br0 |
498 |
bridge name bridge id STP enabled interfaces |
499 |
xen-br0 8000.0020fc1ed55d no eth0 |
500 |
|
501 |
.. _configure-lvm-label: |
502 |
|
503 |
Configuring LVM |
504 |
+++++++++++++++ |
505 |
|
506 |
**Mandatory** on all nodes. |
507 |
|
508 |
The volume group is required to be at least 20GiB. |
509 |
|
510 |
If you haven't configured your LVM volume group at install time you need |
511 |
to do it before trying to initialize the Ganeti cluster. This is done by |
512 |
formatting the devices/partitions you want to use for it and then adding |
513 |
them to the relevant volume group:: |
514 |
|
515 |
$ pvcreate /dev/%sda3% |
516 |
$ vgcreate xenvg /dev/%sda3% |
517 |
|
518 |
or:: |
519 |
|
520 |
$ pvcreate /dev/%sdb1% |
521 |
$ pvcreate /dev/%sdc1% |
522 |
$ vgcreate xenvg /dev/%sdb1% /dev/%sdc1% |
523 |
|
524 |
If you want to add a device later you can do so with the *vgextend* |
525 |
command:: |
526 |
|
527 |
$ pvcreate /dev/%sdd1% |
528 |
$ vgextend xenvg /dev/%sdd1% |
529 |
|
530 |
Optional: it is recommended to configure LVM not to scan the DRBD |
531 |
devices for physical volumes. This can be accomplished by editing |
532 |
``/etc/lvm/lvm.conf`` and adding the ``/dev/drbd[0-9]+`` regular |
533 |
expression to the ``filter`` variable, like this: |
534 |
|
535 |
.. code-block:: text |
536 |
|
537 |
filter = ["r|/dev/cdrom|", "r|/dev/drbd[0-9]+|" ] |
538 |
|
539 |
Note that with Ganeti a helper script is provided - ``lvmstrap`` which |
540 |
will erase and configure as LVM any not in-use disk on your system. This |
541 |
is dangerous and it's recommended to read its ``--help`` output if you |
542 |
want to use it. |
543 |
|
544 |
Installing Ganeti |
545 |
+++++++++++++++++ |
546 |
|
547 |
**Mandatory** on all nodes. |
548 |
|
549 |
It's now time to install the Ganeti software itself. Download the |
550 |
source from the project page at `<http://code.google.com/p/ganeti/>`_, |
551 |
and install it (replace 2.6.0 with the latest version):: |
552 |
|
553 |
$ tar xvzf ganeti-%2.6.0%.tar.gz |
554 |
$ cd ganeti-%2.6.0% |
555 |
$ ./configure --localstatedir=/var --sysconfdir=/etc |
556 |
$ make |
557 |
$ make install |
558 |
$ mkdir /srv/ganeti/ /srv/ganeti/os /srv/ganeti/export |
559 |
|
560 |
You also need to copy the file ``doc/examples/ganeti.initd`` from the |
561 |
source archive to ``/etc/init.d/ganeti`` and register it with your |
562 |
distribution's startup scripts, for example in Debian:: |
563 |
|
564 |
$ chmod +x /etc/init.d/ganeti |
565 |
$ update-rc.d ganeti defaults 20 80 |
566 |
|
567 |
In order to automatically restart failed instances, you need to setup a |
568 |
cron job run the *ganeti-watcher* command. A sample cron file is |
569 |
provided in the source at ``doc/examples/ganeti.cron`` and you can copy |
570 |
that (eventually altering the path) to ``/etc/cron.d/ganeti``. |
571 |
|
572 |
What gets installed |
573 |
~~~~~~~~~~~~~~~~~~~ |
574 |
|
575 |
The above ``make install`` invocation, or installing via your |
576 |
distribution mechanisms, will install on the system: |
577 |
|
578 |
- a set of python libraries under the *ganeti* namespace (depending on |
579 |
the python version this can be located in either |
580 |
``lib/python-$ver/site-packages`` or various other locations) |
581 |
- a set of programs under ``/usr/local/sbin`` or ``/usr/sbin`` |
582 |
- if the htools component was enabled, a set of programs unde |
583 |
``/usr/local/bin`` or ``/usr/bin/`` |
584 |
- man pages for the above programs |
585 |
- a set of tools under the ``lib/ganeti/tools`` directory |
586 |
- an example iallocator script (see the admin guide for details) under |
587 |
``lib/ganeti/iallocators`` |
588 |
- a cron job that is needed for cluster maintenance |
589 |
- an init script for automatic startup of Ganeti daemons |
590 |
- provided but not installed automatically by ``make install`` is a bash |
591 |
completion script that hopefully will ease working with the many |
592 |
cluster commands |
593 |
|
594 |
Installing the Operating System support packages |
595 |
++++++++++++++++++++++++++++++++++++++++++++++++ |
596 |
|
597 |
**Mandatory** on all nodes. |
598 |
|
599 |
To be able to install instances you need to have an Operating System |
600 |
installation script. An example OS that works under Debian and can |
601 |
install Debian and Ubuntu instace OSes is provided on the project web |
602 |
site. Download it from the project page and follow the instructions in |
603 |
the ``README`` file. Here is the installation procedure (replace 0.12 |
604 |
with the latest version that is compatible with your ganeti version):: |
605 |
|
606 |
$ cd /usr/local/src/ |
607 |
$ wget http://ganeti.googlecode.com/files/ganeti-instance-debootstrap-%0.12%.tar.gz |
608 |
$ tar xzf ganeti-instance-debootstrap-%0.12%.tar.gz |
609 |
$ cd ganeti-instance-debootstrap-%0.12% |
610 |
$ ./configure --with-os-dir=/srv/ganeti/os |
611 |
$ make |
612 |
$ make install |
613 |
|
614 |
In order to use this OS definition, you need to have internet access |
615 |
from your nodes and have the *debootstrap*, *dump* and *restore* |
616 |
commands installed on all nodes. Also, if the OS is configured to |
617 |
partition the instance's disk in |
618 |
``/etc/default/ganeti-instance-debootstrap``, you will need *kpartx* |
619 |
installed. |
620 |
|
621 |
.. admonition:: Debian |
622 |
|
623 |
Use this command on all nodes to install the required packages:: |
624 |
|
625 |
$ apt-get install debootstrap dump kpartx |
626 |
|
627 |
Or alternatively install the OS definition from the Debian package:: |
628 |
|
629 |
$ apt-get install ganeti-instance-debootstrap |
630 |
|
631 |
.. admonition:: KVM |
632 |
|
633 |
In order for debootstrap instances to be able to shutdown cleanly |
634 |
they must install have basic ACPI support inside the instance. Which |
635 |
packages are needed depend on the exact flavor of Debian or Ubuntu |
636 |
which you're installing, but the example defaults file has a |
637 |
commented out configuration line that works for Debian Lenny and |
638 |
Squeeze:: |
639 |
|
640 |
EXTRA_PKGS="acpi-support-base,console-tools,udev" |
641 |
|
642 |
``kbd`` can be used instead of ``console-tools``, and more packages |
643 |
can be added, of course, if needed. |
644 |
|
645 |
Please refer to the ``README`` file of ``ganeti-instance-debootstrap`` for |
646 |
further documentation. |
647 |
|
648 |
Alternatively, you can create your own OS definitions. See the manpage |
649 |
:manpage:`ganeti-os-interface(7)`. |
650 |
|
651 |
Initializing the cluster |
652 |
++++++++++++++++++++++++ |
653 |
|
654 |
**Mandatory** once per cluster, on the first node. |
655 |
|
656 |
The last step is to initialize the cluster. After you have repeated the |
657 |
above process on all of your nodes and choose one as the master. Make sure |
658 |
there is a SSH key pair on the master node (optionally generating one using |
659 |
``ssh-keygen``). Finally execute:: |
660 |
|
661 |
$ gnt-cluster init %CLUSTERNAME% |
662 |
|
663 |
The *CLUSTERNAME* is a hostname, which must be resolvable (e.g. it must |
664 |
exist in DNS or in ``/etc/hosts``) by all the nodes in the cluster. You |
665 |
must choose a name different from any of the nodes names for a |
666 |
multi-node cluster. In general the best choice is to have a unique name |
667 |
for a cluster, even if it consists of only one machine, as you will be |
668 |
able to expand it later without any problems. Please note that the |
669 |
hostname used for this must resolve to an IP address reserved |
670 |
**exclusively** for this purpose, and cannot be the name of the first |
671 |
(master) node. |
672 |
|
673 |
If you want to use a bridge which is not ``xen-br0``, or no bridge at |
674 |
all, change it with the ``--nic-parameters`` option. For example to |
675 |
bridge on br0 you can add:: |
676 |
|
677 |
--nic-parameters link=br0 |
678 |
|
679 |
Or to not bridge at all, and use a separate routing table:: |
680 |
|
681 |
--nic-parameters mode=routed,link=100 |
682 |
|
683 |
If you don't have a ``xen-br0`` interface you also have to specify a |
684 |
different network interface which will get the cluster IP, on the master |
685 |
node, by using the ``--master-netdev <device>`` option. |
686 |
|
687 |
You can use a different name than ``xenvg`` for the volume group (but |
688 |
note that the name must be identical on all nodes). In this case you |
689 |
need to specify it by passing the *--vg-name <VGNAME>* option to |
690 |
``gnt-cluster init``. |
691 |
|
692 |
To set up the cluster as an Xen HVM cluster, use the |
693 |
``--enabled-hypervisors=xen-hvm`` option to enable the HVM hypervisor |
694 |
(you can also add ``,xen-pvm`` to enable the PVM one too). You will also |
695 |
need to create the VNC cluster password file |
696 |
``/etc/ganeti/vnc-cluster-password`` which contains one line with the |
697 |
default VNC password for the cluster. |
698 |
|
699 |
To setup the cluster for KVM-only usage (KVM and Xen cannot be mixed), |
700 |
pass ``--enabled-hypervisors=kvm`` to the init command. |
701 |
|
702 |
You can also invoke the command with the ``--help`` option in order to |
703 |
see all the possibilities. |
704 |
|
705 |
Hypervisor/Network/Cluster parameters |
706 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
707 |
|
708 |
Please note that the default hypervisor/network/cluster parameters may |
709 |
not be the correct one for your environment. Carefully check them, and |
710 |
change them either at cluster init time, or later with ``gnt-cluster |
711 |
modify``. |
712 |
|
713 |
Your instance types, networking environment, hypervisor type and version |
714 |
may all affect what kind of parameters should be used on your cluster. |
715 |
|
716 |
.. admonition:: KVM |
717 |
|
718 |
Instances are by default configured to use a host kernel, and to be |
719 |
reached via serial console, which works nice for Linux paravirtualized |
720 |
instances. If you want fully virtualized instances you may want to |
721 |
handle their kernel inside the instance, and to use VNC. |
722 |
|
723 |
Some versions of KVM have a bug that will make an instance hang when |
724 |
configured to use the serial console (which is the default) unless a |
725 |
connection is made to it within about 2 seconds of the instance's |
726 |
startup. For such case it's recommended to disable the |
727 |
``serial_console`` option. |
728 |
|
729 |
|
730 |
Joining the nodes to the cluster |
731 |
++++++++++++++++++++++++++++++++ |
732 |
|
733 |
**Mandatory** for all the other nodes. |
734 |
|
735 |
After you have initialized your cluster you need to join the other nodes |
736 |
to it. You can do so by executing the following command on the master |
737 |
node:: |
738 |
|
739 |
$ gnt-node add %NODENAME% |
740 |
|
741 |
Separate replication network |
742 |
++++++++++++++++++++++++++++ |
743 |
|
744 |
**Optional** |
745 |
|
746 |
Ganeti uses DRBD to mirror the disk of the virtual instances between |
747 |
nodes. To use a dedicated network interface for this (in order to |
748 |
improve performance or to enhance security) you need to configure an |
749 |
additional interface for each node. Use the *-s* option with |
750 |
``gnt-cluster init`` and ``gnt-node add`` to specify the IP address of |
751 |
this secondary interface to use for each node. Note that if you |
752 |
specified this option at cluster setup time, you must afterwards use it |
753 |
for every node add operation. |
754 |
|
755 |
Testing the setup |
756 |
+++++++++++++++++ |
757 |
|
758 |
Execute the ``gnt-node list`` command to see all nodes in the cluster:: |
759 |
|
760 |
$ gnt-node list |
761 |
Node DTotal DFree MTotal MNode MFree Pinst Sinst |
762 |
node1.example.com 197404 197404 2047 1896 125 0 0 |
763 |
|
764 |
The above shows a couple of things: |
765 |
|
766 |
- The various Ganeti daemons can talk to each other |
767 |
- Ganeti can examine the storage of the node (DTotal/DFree) |
768 |
- Ganeti can talk to the selected hypervisor (MTotal/MNode/MFree) |
769 |
|
770 |
Cluster burnin |
771 |
~~~~~~~~~~~~~~ |
772 |
|
773 |
With Ganeti a tool called :command:`burnin` is provided that can test |
774 |
most of the Ganeti functionality. The tool is installed under the |
775 |
``lib/ganeti/tools`` directory (either under ``/usr`` or ``/usr/local`` |
776 |
based on the installation method). See more details under |
777 |
:ref:`burnin-label`. |
778 |
|
779 |
Further steps |
780 |
------------- |
781 |
|
782 |
You can now proceed either to the :doc:`admin`, or read the manpages of |
783 |
the various commands (:manpage:`ganeti(7)`, :manpage:`gnt-cluster(8)`, |
784 |
:manpage:`gnt-node(8)`, :manpage:`gnt-instance(8)`, |
785 |
:manpage:`gnt-job(8)`). |
786 |
|
787 |
.. rubric:: Footnotes |
788 |
|
789 |
.. [#defkernel] The kernel and initrd paths can be changed at either |
790 |
cluster level (which changes the default for all instances) or at |
791 |
instance level. |
792 |
|
793 |
.. vim: set textwidth=72 : |
794 |
.. Local Variables: |
795 |
.. mode: rst |
796 |
.. fill-column: 72 |
797 |
.. End: |