1 Ganeti installation tutorial
2 ============================
4 Documents Ganeti version |version|
13 Ganeti is a cluster virtualization management system based on Xen or
14 KVM. This document explains how to bootstrap a Ganeti node (Xen *dom0*,
15 the host Linux system for KVM), create a running cluster and install
16 virtual instances (Xen *domUs*, KVM guests). You need to repeat most of
17 the steps in this document for every node you want to install, but of
18 course we recommend creating some semi-automatic procedure if you plan
19 to deploy Ganeti on a medium/large scale.
21 A basic Ganeti terminology glossary is provided in the introductory
22 section of the :doc:`admin`. Please refer to that document if you are
23 uncertain about the terms we are using.
25 Ganeti has been developed for Linux and should be distribution-agnostic.
26 This documentation will use Debian Lenny as an example system but the
27 examples can be translated to any other distribution. You are expected
28 to be familiar with your distribution, its package management system,
29 and Xen or KVM before trying to use Ganeti.
31 This document is divided into two main sections:
33 - Installation of the base system and base components
35 - Configuration of the environment for Ganeti
37 Each of these is divided into sub-sections. While a full Ganeti system
38 will need all of the steps specified, some are not strictly required for
39 every environment. Which ones they are, and why, is specified in the
40 corresponding sections.
42 Installing the base system and base components
43 ----------------------------------------------
48 Any system supported by your Linux distribution is fine. 64-bit systems
49 are better as they can support more memory.
51 Any disk drive recognized by Linux (``IDE``/``SCSI``/``SATA``/etc.) is
52 supported in Ganeti. Note that no shared storage (e.g. ``SAN``) is
53 needed to get high-availability features (but of course, one can be used
54 to store the images). It is highly recommended to use more than one disk
55 drive to improve speed. But Ganeti also works with one disk per machine.
57 Installing the base system
58 ++++++++++++++++++++++++++
60 **Mandatory** on all nodes.
62 It is advised to start with a clean, minimal install of the operating
63 system. The only requirement you need to be aware of at this stage is to
64 partition leaving enough space for a big (**minimum** 20GiB) LVM volume
65 group which will then host your instance filesystems, if you want to use
66 all Ganeti features. The volume group name Ganeti uses (by default) is
69 You can also use file-based storage only, without LVM, but this setup is
70 not detailed in this document.
72 While you can use an existing system, please note that the Ganeti
73 installation is intrusive in terms of changes to the system
74 configuration, and it's best to use a newly-installed system without
77 Also, for best results, it's advised that the nodes have as much as
78 possible the same hardware and software configuration. This will make
79 administration much easier.
84 Note that Ganeti requires the hostnames of the systems (i.e. what the
85 ``hostname`` command outputs to be a fully-qualified name, not a short
86 name. In other words, you should use *node1.example.com* as a hostname
89 .. admonition:: Debian
91 Debian Lenny and Etch configures the hostname differently than you
92 need it for Ganeti. For example, this is what Etch puts in
93 ``/etc/hosts`` in certain situations::
96 127.0.1.1 node1.example.com node1
98 but for Ganeti you need to have::
101 192.168.1.1 node1.example.com node1
103 replacing ``192.168.1.1`` with your node's address. Also, the file
104 ``/etc/hostname`` which configures the hostname of the system
105 should contain ``node1.example.com`` and not just ``node1`` (you
106 need to run the command ``/etc/init.d/hostname.sh start`` after
109 .. admonition:: Why a fully qualified host name
111 Although most distributions use only the short name in the
112 /etc/hostname file, we still think Ganeti nodes should use the full
113 name. The reason for this is that calling 'hostname --fqdn' requires
114 the resolver library to work and is a 'guess' via heuristics at what
115 is your domain name. Since Ganeti can be used among other things to
116 host DNS servers, we don't want to depend on them as much as
117 possible, and we'd rather have the uname() syscall return the full
120 We haven't ever found any breakage in using a full hostname on a
121 Linux system, and anyway we recommend to have only a minimal
122 installation on Ganeti nodes, and to use instances (or other
123 dedicated machines) to run the rest of your network services. By
124 doing this you can change the /etc/hostname file to contain an FQDN
125 without the fear of breaking anything unrelated.
128 Installing The Hypervisor
129 +++++++++++++++++++++++++
131 **Mandatory** on all nodes.
133 While Ganeti is developed with the ability to modularly run on different
134 virtualization environments in mind the only two currently useable on a
135 live system are Xen and KVM. Supported Xen versions are: 3.0.3, 3.0.4
136 and 3.1. Supported KVM version are 72 and above.
138 Please follow your distribution's recommended way to install and set up
139 Xen, or install Xen from the upstream source, if you wish, following
140 their manual. For KVM, make sure you have a KVM-enabled kernel and the
143 After installing Xen, you need to reboot into your new system. On some
144 distributions this might involve configuring GRUB appropriately, whereas
145 others will configure it automatically when you install the respective
146 kernels. For KVM no reboot should be necessary.
148 .. admonition:: Xen on Debian
150 Under Lenny or Etch you can install the relevant ``xen-linux-system``
151 package, which will pull in both the hypervisor and the relevant
152 kernel. Also, if you are installing a 32-bit Lenny/Etch, you should
153 install the ``libc6-xen`` package (run ``apt-get install
159 It's recommended that dom0 is restricted to a low amount of memory
160 (512MiB or 1GiB is reasonable) and that memory ballooning is disabled in
161 the file ``/etc/xen/xend-config.sxp`` by setting the value
162 ``dom0-min-mem`` to 0, like this::
166 For optimum performance when running both CPU and I/O intensive
167 instances, it's also recommended that the dom0 is restricted to one CPU
168 only, for example by booting with the kernel parameter ``nosmp``.
170 It is recommended that you disable xen's automatic save of virtual
171 machines at system shutdown and subsequent restore of them at reboot.
172 To obtain this make sure the variable ``XENDOMAINS_SAVE`` in the file
173 ``/etc/default/xendomains`` is set to an empty value.
175 If you want to use live migration make sure you have, in the xen config
176 file, something that allows the nodes to migrate instances between each
179 (xend-relocation-server yes)
180 (xend-relocation-port 8002)
181 (xend-relocation-address '')
182 (xend-relocation-hosts-allow '^192\\.168\\.3\\.[0-9]+$')
185 The second line assumess that the hypervisor parameter
186 ``migration_port`` is set 8002, otherwise modify it to match. The last
187 line assumes that all your nodes have secondary IPs in the
188 192.168.3.0/24 network, adjust it accordingly to your setup.
190 .. admonition:: Debian
192 Besides the ballooning change which you need to set in
193 ``/etc/xen/xend-config.sxp``, you need to set the memory and nosmp
194 parameters in the file ``/boot/grub/menu.lst``. You need to modify
195 the variable ``xenhopt`` to add ``dom0_mem=1024M`` like this::
197 ## Xen hypervisor options to use with the default Xen boot option
198 # xenhopt=dom0_mem=1024M
200 and the ``xenkopt`` needs to include the ``nosmp`` option like this::
202 ## Xen Linux kernel options to use with the default Xen boot option
205 Any existing parameters can be left in place: it's ok to have
206 ``xenkopt=console=tty0 nosmp``, for example. After modifying the
207 files, you need to run::
211 If you want to run HVM instances too with Ganeti and want VNC access to
212 the console of your instances, set the following two entries in
213 ``/etc/xen/xend-config.sxp``::
215 (vnc-listen '0.0.0.0') (vncpasswd '')
217 You need to restart the Xen daemon for these settings to take effect::
219 /etc/init.d/xend restart
221 Selecting the instance kernel
222 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
224 After you have installed Xen, you need to tell Ganeti exactly what
225 kernel to use for the instances it will create. This is done by creating
226 a symlink from your actual kernel to ``/boot/vmlinuz-2.6-xenU``, and one
227 from your initrd to ``/boot/initrd-2.6-xenU`` [#defkernel]_. Note that
228 if you don't use an initrd for the domU kernel, you don't need to create
231 .. admonition:: Debian
233 After installation of the ``xen-linux-system`` package, you need to
234 run (replace the exact version number with the one you have)::
237 ln -s vmlinuz-2.6.26-1-xen-amd64 vmlinuz-2.6-xenU
238 ln -s initrd.img-2.6.26-1-xen-amd64 initrd-2.6-xenU
243 Recommended on all nodes: DRBD_ is required if you want to use the high
244 availability (HA) features of Ganeti, but optional if you don't require
245 them or only run Ganeti on single-node clusters. You can upgrade a
246 non-HA cluster to an HA one later, but you might need to export and
247 re-import all your instances to take advantage of the new features.
249 .. _DRBD: http://www.drbd.org/
251 Supported DRBD versions: 8.0+. It's recommended to have at least version
252 8.0.12. Note that for version 8.2 and newer it is needed to pass the
253 ``usermode_helper=/bin/true`` parameter to the module, either by
254 configuring ``/etc/modules`` or when inserting it manually.
256 Now the bad news: unless your distribution already provides it
257 installing DRBD might involve recompiling your kernel or anyway fiddling
258 with it. Hopefully at least the Xen-ified kernel source to start from
259 will be provided (if you intend to use Xen).
261 The good news is that you don't need to configure DRBD at all. Ganeti
262 will do it for you for every instance you set up. If you have the DRBD
263 utils installed and the module in your kernel you're fine. Please check
264 that your system is configured to load the module at every boot, and
265 that it passes the following option to the module:
266 ``minor_count=NUMBER``. We recommend that you use 128 as the value of
267 the minor_count - this will allow you to use up to 64 instances in total
268 per node (both primary and secondary, when using only one disk per
269 instance). You can increase the number up to 255 if you need more
273 .. admonition:: Debian
275 On Debian, you can just install (build) the DRBD module with the
276 following commands, making sure you are running the target (Xen or
279 apt-get install drbd8-source drbd8-utils
282 echo drbd minor_count=128 usermode_helper=/bin/true >> /etc/modules
284 modprobe drbd minor_count=128 usermode_helper=/bin/true
286 It is also recommended that you comment out the default resources in
287 the ``/etc/drbd.conf`` file, so that the init script doesn't try to
288 configure any drbd devices. You can do this by prefixing all
289 *resource* lines in the file with the keyword *skip*, like this::
299 Other required software
300 +++++++++++++++++++++++
302 See :doc:`install-quick`.
304 Setting up the environment for Ganeti
305 -------------------------------------
307 Configuring the network
308 +++++++++++++++++++++++
310 **Mandatory** on all nodes.
312 You can run Ganeti either in "bridge mode" or in "routed mode". In
313 bridge mode, the default, the instances network interfaces will be
314 attached to a software bridge running in dom0. Xen by default creates
315 such a bridge at startup, but your distribution might have a different
316 way to do things, and you'll definitely need to manually set it up under
319 Beware that the default name Ganeti uses is ``xen-br0`` (which was used
320 in Xen 2.0) while Xen 3.0 uses ``xenbr0`` by default. The default bridge
321 your Ganeti cluster will use for new instances can be specified at
322 cluster initialization time.
324 If you want to run in "routing mode" you need to specify that at cluster
325 init time (using the --nicparam option), and then no bridge will be
326 needed. In this mode instance traffic will be routed by dom0, instead of
329 In order to use "routing mode" under Xen, you'll need to change the
330 relevant parameters in the Xen config file. Under KVM instead, no config
331 change is necessary, but you still need to set up your network
332 interfaces correctly.
334 By default, under KVM, the "link" parameter you specify per-nic will
335 represent, if non-empty, a different routing table name or number to use
336 for your instances. This allows insulation between different instance
337 groups, and different routing policies between node traffic and instance
340 You will need to configure your routing table basic routes and rules
341 outside of ganeti. The vif scripts will only add /32 routes to your
342 instances, through their interface, in the table you specified (under
343 KVM, and in the main table under Xen).
345 .. admonition:: Bridging under Debian
347 The recommended way to configure the Xen bridge is to edit your
348 ``/etc/network/interfaces`` file and substitute your normal
349 Ethernet stanza with the following snippet::
352 iface xen-br0 inet static
353 address YOUR_IP_ADDRESS
356 broadcast YOUR_BROADCAST_ADDRESS
362 The following commands need to be executed on the local console:
367 To check if the bridge is setup, use the ``ip`` and ``brctl show``
371 9: xen-br0: <BROADCAST,MULTICAST,UP,10000> mtu 1500 qdisc noqueue
372 link/ether 00:20:fc:1e:d5:5d brd ff:ff:ff:ff:ff:ff
373 inet 10.1.1.200/24 brd 10.1.1.255 scope global xen-br0
374 inet6 fe80::220:fcff:fe1e:d55d/64 scope link
375 valid_lft forever preferred_lft forever
378 bridge name bridge id STP enabled interfaces
379 xen-br0 8000.0020fc1ed55d no eth0
381 .. _configure-lvm-label:
386 **Mandatory** on all nodes.
388 The volume group is required to be at least 20GiB.
390 If you haven't configured your LVM volume group at install time you need
391 to do it before trying to initialize the Ganeti cluster. This is done by
392 formatting the devices/partitions you want to use for it and then adding
393 them to the relevant volume group::
396 vgcreate xenvg /dev/sda3
402 vgcreate xenvg /dev/sdb1 /dev/sdc1
404 If you want to add a device later you can do so with the *vgextend*
408 vgextend xenvg /dev/sdd1
410 Optional: it is recommended to configure LVM not to scan the DRBD
411 devices for physical volumes. This can be accomplished by editing
412 ``/etc/lvm/lvm.conf`` and adding the ``/dev/drbd[0-9]+`` regular
413 expression to the ``filter`` variable, like this::
415 filter = ["r|/dev/cdrom|", "r|/dev/drbd[0-9]+|" ]
417 Note that with Ganeti a helper script is provided - ``lvmstrap`` which
418 will erase and configure as LVM any not in-use disk on your system. This
419 is dangerous and it's recommended to read its ``--help`` output if you
425 **Mandatory** on all nodes.
427 It's now time to install the Ganeti software itself. Download the
428 source from the project page at `<http://code.google.com/p/ganeti/>`_,
429 and install it (replace 2.0.0 with the latest version)::
431 tar xvzf ganeti-2.0.0.tar.gz
433 ./configure --localstatedir=/var --sysconfdir=/etc
436 mkdir /srv/ganeti/ /srv/ganeti/os /srv/ganeti/export
438 You also need to copy the file ``doc/examples/ganeti.initd`` from the
439 source archive to ``/etc/init.d/ganeti`` and register it with your
440 distribution's startup scripts, for example in Debian::
442 update-rc.d ganeti defaults 20 80
444 In order to automatically restart failed instances, you need to setup a
445 cron job run the *ganeti-watcher* command. A sample cron file is
446 provided in the source at ``doc/examples/ganeti.cron`` and you can copy
447 that (eventually altering the path) to ``/etc/cron.d/ganeti``.
452 The above ``make install`` invocation, or installing via your
453 distribution mechanisms, will install on the system:
455 - a set of python libraries under the *ganeti* namespace (depending on
456 the python version this can be located in either
457 ``lib/python-$ver/site-packages`` or various other locations)
458 - a set of programs under ``/usr/local/sbin`` or ``/usr/sbin``
459 - man pages for the above programs
460 - a set of tools under the ``lib/ganeti/tools`` directory
461 - an example iallocator script (see the admin guide for details) under
462 ``lib/ganeti/iallocators``
463 - a cron job that is needed for cluster maintenance
464 - an init script for automatic startup of Ganeti daemons
465 - provided but not installed automatically by ``make install`` is a bash
466 completion script that hopefully will ease working with the many
469 Installing the Operating System support packages
470 ++++++++++++++++++++++++++++++++++++++++++++++++
472 **Mandatory** on all nodes.
474 To be able to install instances you need to have an Operating System
475 installation script. An example OS that works under Debian and can
476 install Debian and Ubuntu instace OSes is provided on the project web
477 site. Download it from the project page and follow the instructions in
478 the ``README`` file. Here is the installation procedure (replace 0.7
479 with the latest version that is compatible with your ganeti version)::
482 wget http://ganeti.googlecode.com/files/ganeti-instance-debootstrap-0.7.tar.gz
483 tar xzf ganeti-instance-debootstrap-0.7.tar.gz
484 cd ganeti-instance-debootstrap-0.7
489 In order to use this OS definition, you need to have internet access
490 from your nodes and have the *debootstrap*, *dump* and *restore*
491 commands installed on all nodes. Also, if the OS is configured to
492 partition the instance's disk in
493 ``/etc/default/ganeti-instance-debootstrap``, you will need *kpartx*
496 .. admonition:: Debian
498 Use this command on all nodes to install the required packages::
500 apt-get install debootstrap dump kpartx
502 Alternatively, you can create your own OS definitions. See the manpage
503 :manpage:`ganeti-os-interface`.
505 Initializing the cluster
506 ++++++++++++++++++++++++
508 **Mandatory** once per cluster, on the first node.
510 The last step is to initialize the cluster. After you have repeated the
511 above process on all of your nodes, choose one as the master, and
514 gnt-cluster init <CLUSTERNAME>
516 The *CLUSTERNAME* is a hostname, which must be resolvable (e.g. it must
517 exist in DNS or in ``/etc/hosts``) by all the nodes in the cluster. You
518 must choose a name different from any of the nodes names for a
519 multi-node cluster. In general the best choice is to have a unique name
520 for a cluster, even if it consists of only one machine, as you will be
521 able to expand it later without any problems. Please note that the
522 hostname used for this must resolve to an IP address reserved
523 **exclusively** for this purpose, and cannot be the name of the first
526 If you want to use a bridge which is not ``xen-br0``, or no bridge at
527 all, use ``--nicparams``.
529 If the bridge name you are using is not ``xen-br0``, use the *-b
530 <BRIDGENAME>* option to specify the bridge name. In this case, you
531 should also use the *--master-netdev <BRIDGENAME>* option with the same
534 You can use a different name than ``xenvg`` for the volume group (but
535 note that the name must be identical on all nodes). In this case you
536 need to specify it by passing the *-g <VGNAME>* option to ``gnt-cluster
539 To set up the cluster as an Xen HVM cluster, use the
540 ``--enabled-hypervisors=xen-hvm`` option to enable the HVM hypervisor
541 (you can also add ``,xen-pvm`` to enable the PVM one too). You will also
542 need to create the VNC cluster password file
543 ``/etc/ganeti/vnc-cluster-password`` which contains one line with the
544 default VNC password for the cluster.
546 To setup the cluster for KVM-only usage (KVM and Xen cannot be mixed),
547 pass ``--enabled-hypervisors=kvm`` to the init command.
549 You can also invoke the command with the ``--help`` option in order to
550 see all the possibilities.
552 Joining the nodes to the cluster
553 ++++++++++++++++++++++++++++++++
555 **Mandatory** for all the other nodes.
557 After you have initialized your cluster you need to join the other nodes
558 to it. You can do so by executing the following command on the master
561 gnt-node add <NODENAME>
563 Separate replication network
564 ++++++++++++++++++++++++++++
568 Ganeti uses DRBD to mirror the disk of the virtual instances between
569 nodes. To use a dedicated network interface for this (in order to
570 improve performance or to enhance security) you need to configure an
571 additional interface for each node. Use the *-s* option with
572 ``gnt-cluster init`` and ``gnt-node add`` to specify the IP address of
573 this secondary interface to use for each node. Note that if you
574 specified this option at cluster setup time, you must afterwards use it
575 for every node add operation.
580 Execute the ``gnt-node list`` command to see all nodes in the cluster::
583 Node DTotal DFree MTotal MNode MFree Pinst Sinst
584 node1.example.com 197404 197404 2047 1896 125 0 0
586 The above shows a couple of things:
588 - The various Ganeti daemons can talk to each other
589 - Ganeti can examine the storage of the node (DTotal/DFree)
590 - Ganeti can talk to the selected hypervisor (MTotal/MNode/MFree)
595 With Ganeti a tool called :command:`burnin` is provided that can test
596 most of the Ganeti functionality. The tool is installed under the
597 ``lib/ganeti/tools`` directory (either under ``/usr`` or ``/usr/local``
598 based on the installation method). See more details under
604 You can now proceed either to the :doc:`admin`, or read the manpages of
605 the various commands (:manpage:`ganeti(7)`, :manpage:`gnt-cluster(8)`,
606 :manpage:`gnt-node(8)`, :manpage:`gnt-instance(8)`,
607 :manpage:`gnt-job(8)`).
609 .. rubric:: Footnotes
611 .. [#defkernel] The kernel and initrd paths can be changed at either
612 cluster level (which changes the default for all instances) or at
615 .. vim: set textwidth=72 :