1 Ganeti installation tutorial
2 ============================
4 Documents Ganeti version |version|
11 Ganeti is a cluster virtualization management system based on Xen or
12 KVM. This document explains how to bootstrap a Ganeti node (Xen
13 *dom0*), create a running cluster and install virtual instance (Xen
14 *domU*). You need to repeat most of the steps in this document for
15 every node you want to install, but of course we recommend creating
16 some semi-automatic procedure if you plan to deploy Ganeti on a
19 A basic Ganeti terminology glossary is provided in the introductory
20 section of the *Ganeti administrator's guide*. Please refer to that
21 document if you are uncertain about the terms we are using.
23 Ganeti has been developed for Linux and is distribution-agnostic.
24 This documentation will use Debian Lenny as an example system but the
25 examples can easily be translated to any other distribution. ou are
26 expected to be familiar with your distribution, its package management
27 system, and Xen or KVM before trying to use Ganeti.
29 This document is divided into two main sections:
31 - Installation of the base system and base components
33 - Configuration of the environment for Ganeti
35 Each of these is divided into sub-sections. While a full Ganeti system
36 will need all of the steps specified, some are not strictly required
37 for every environment. Which ones they are, and why, is specified in
38 the corresponding sections.
40 Installing the base system and base components
41 ----------------------------------------------
46 Any system supported by your Linux distribution is fine. 64-bit
47 systems are better as they can support more memory.
49 Any disk drive recognized by Linux (``IDE``/``SCSI``/``SATA``/etc.)
50 is supported in Ganeti. Note that no shared storage (e.g. ``SAN``) is
51 needed to get high-availability features (but of course, one can be
52 used to store the images). It is highly recommended to use more than
53 one disk drive to improve speed. But Ganeti also works with one disk
56 Installing the base system
57 ++++++++++++++++++++++++++
59 **Mandatory** on all nodes.
61 It is advised to start with a clean, minimal install of the operating
62 system. The only requirement you need to be aware of at this stage is
63 to partition leaving enough space for a big (**minimum** 20GiB) LVM
64 volume group which will then host your instance filesystems, if you
65 want to use all Ganeti features. The volume group name Ganeti 2.0 uses
66 (by default) is ``xenvg``.
68 You can also use file-based storage only, without LVM, but this setup
69 is not detailed in this document.
72 While you can use an existing system, please note that the Ganeti
73 installation is intrusive in terms of changes to the system
74 configuration, and it's best to use a newly-installed system without
77 Also, for best results, it's advised that the nodes have as much as
78 possible the same hardware and software configuration. This will make
79 administration much easier.
84 Note that Ganeti requires the hostnames of the systems (i.e. what the
85 ``hostname`` command outputs to be a fully-qualified name, not a short
86 name. In other words, you should use *node1.example.com* as a hostname
89 .. admonition:: Debian
91 Debian Lenny and Etch configures the hostname differently than you
92 need it for Ganeti. For example, this is what Etch puts in
93 ``/etc/hosts`` in certain situations::
96 127.0.1.1 node1.example.com node1
98 but for Ganeti you need to have::
101 192.168.1.1 node1.example.com node1
103 replacing ``192.168.1.1`` with your node's address. Also, the file
104 ``/etc/hostname`` which configures the hostname of the system
105 should contain ``node1.example.com`` and not just ``node1`` (you
106 need to run the command ``/etc/init.d/hostname.sh start`` after
109 .. admonition:: Why a fully qualified host name
111 Although most distributions use only the short name in the
112 /etc/hostname file, we still think Ganeti nodes should use the full
113 name. The reason for this is that calling 'hostname --fqdn' requires
114 the resolver library to work and is a 'guess' via heuristics at what
115 is your domain name. Since Ganeti can be used among other things to
116 host DNS servers, we don't want to depend on them as much as
117 possible, and we'd rather have the uname() syscall return the full
120 We haven't ever found any breakage in using a full hostname on a
121 Linux system, and anyway we recommend to have only a minimal
122 installation on Ganeti nodes, and to use instances (or other
123 dedicated machines) to run the rest of your network services. By
124 doing this you can change the /etc/hostname file to contain an FQDN
125 without the fear of breaking anything unrelated.
128 Installing The Hypervisor
129 +++++++++++++++++++++++++
131 **Mandatory** on all nodes.
133 While Ganeti is developed with the ability to modularly run on different
134 virtualization environments in mind the only two currently useable on a
135 live system are Xen and KVM. Supported Xen versions are: 3.0.3, 3.0.4
136 and 3.1. Supported KVM version are 72 and above.
138 Please follow your distribution's recommended way to install and set
139 up Xen, or install Xen from the upstream source, if you wish,
140 following their manual. For KVM, make sure you have a KVM-enabled
141 kernel and the KVM tools.
143 After installing Xen, you need to reboot into your new system. On some
144 distributions this might involve configuring GRUB appropriately, whereas
145 others will configure it automatically when you install the respective
146 kernels. For KVM no reboot should be necessary.
148 .. admonition:: Xen on Debian
150 Under Lenny or Etch you can install the relevant
151 ``xen-linux-system`` package, which will pull in both the
152 hypervisor and the relevant kernel. Also, if you are installing a
153 32-bit Lenny/Etch, you should install the ``libc6-xen`` package
154 (run ``apt-get install libc6-xen``).
159 It's recommended that dom0 is restricted to a low amount of memory
160 (512MiB or 1GiB is reasonable) and that memory ballooning is disabled
161 in the file ``/etc/xen/xend-config.sxp`` by setting
162 the value ``dom0-min-mem`` to 0,
167 For optimum performance when running both CPU and I/O intensive
168 instances, it's also recommended that the dom0 is restricted to one
169 CPU only, for example by booting with the kernel parameter ``nosmp``.
171 It is recommended that you disable xen's automatic save of virtual
172 machines at system shutdown and subsequent restore of them at reboot.
173 To obtain this make sure the variable ``XENDOMAINS_SAVE`` in the file
174 ``/etc/default/xendomains`` is set to an empty value.
176 .. admonition:: Debian
178 Besides the ballooning change which you need to set in
179 ``/etc/xen/xend-config.sxp``, you need to set the memory and nosmp
180 parameters in the file ``/boot/grub/menu.lst``. You need to modify
181 the variable ``xenhopt`` to add ``dom0_mem=1024M`` like this::
183 ## Xen hypervisor options to use with the default Xen boot option
184 # xenhopt=dom0_mem=1024M
186 and the ``xenkopt`` needs to include the ``nosmp`` option like
189 ## Xen Linux kernel options to use with the default Xen boot option
192 Any existing parameters can be left in place: it's ok to have
193 ``xenkopt=console=tty0 nosmp``, for example. After modifying the
194 files, you need to run::
198 If you want to run HVM instances too with Ganeti and want VNC access
199 to the console of your instances, set the following two entries in
200 ``/etc/xen/xend-config.sxp``::
202 (vnc-listen '0.0.0.0') (vncpasswd '')
204 You need to restart the Xen daemon for these settings to take effect::
206 /etc/init.d/xend restart
208 Selecting the instance kernel
209 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
211 After you have installed Xen, you need to tell Ganeti exactly what
212 kernel to use for the instances it will create. This is done by
213 creating a symlink from your actual kernel to
214 ``/boot/vmlinuz-2.6-xenU``, and one from your initrd
215 to ``/boot/initrd-2.6-xenU``. Note that if you don't
216 use an initrd for the domU kernel, you don't need
217 to create the initrd symlink.
219 .. admonition:: Debian
221 After installation of the ``xen-linux-system`` package, you need to
222 run (replace the exact version number with the one you have)::
225 ln -s vmlinuz-2.6.26-1-xen-amd64 vmlinuz-2.6-xenU
226 ln -s initrd.img-2.6.26-1-xen-amd64 initrd-2.6-xenU
231 Recommended on all nodes: DRBD_ is required if you want to use the
232 high availability (HA) features of Ganeti, but optional if you don't
233 require HA or only run Ganeti on single-node clusters. You can upgrade
234 a non-HA cluster to an HA one later, but you might need to export and
235 re-import all your instances to take advantage of the new features.
237 .. _DRBD: http://www.drbd.org/
239 Supported DRBD versions: 8.0.x. It's recommended to have at least
242 Now the bad news: unless your distribution already provides it
243 installing DRBD might involve recompiling your kernel or anyway
244 fiddling with it. Hopefully at least the Xen-ified kernel source to
245 start from will be provided.
247 The good news is that you don't need to configure DRBD at all. Ganeti
248 will do it for you for every instance you set up. If you have the
249 DRBD utils installed and the module in your kernel you're fine. Please
250 check that your system is configured to load the module at every boot,
251 and that it passes the following option to the module
252 ``minor_count=255``. This will allow you to use up to 128 instances
253 per node (for most clusters 128 should be enough, though).
255 .. admonition:: Debian
257 On Debian, you can just install (build) the DRBD 8.0.x module with
258 the following commands (make sure you are running the Xen kernel)::
260 apt-get install drbd8-source drbd8-utils
263 echo drbd minor_count=128 >> /etc/modules
265 modprobe drbd minor_count=128
267 It is also recommended that you comment out the default resources
268 in the ``/etc/drbd.conf`` file, so that the init script doesn't try
269 to configure any drbd devices. You can do this by prefixing all
270 *resource* lines in the file with the keyword *skip*, like this::
280 Other required software
281 +++++++++++++++++++++++
283 Besides Xen and DRBD, you will need to install the following (on all
286 - LVM version 2, `<http://sourceware.org/lvm2/>`_
288 - OpenSSL, `<http://www.openssl.org/>`_
290 - OpenSSH, `<http://www.openssh.com/portable.html>`_
292 - bridge utilities, `<http://bridge.sourceforge.net/>`_
294 - iproute2, `<http://developer.osdl.org/dev/iproute2>`_
296 - arping (part of iputils package),
297 `<ftp://ftp.inr.ac.ru/ip-routing/iputils-current.tar.gz>`_
299 - Python version 2.4 or 2.5, `<http://www.python.org>`_
301 - Python OpenSSL bindings, `<http://pyopenssl.sourceforge.net/>`_
303 - simplejson Python module, `<http://www.undefined.org/python/#simplejson>`_
305 - pyparsing Python module, `<http://pyparsing.wikispaces.com/>`_
307 - pyinotify Python module, `<http://trac.dbzteam.org/pyinotify>`_
309 These programs are supplied as part of most Linux distributions, so
310 usually they can be installed via apt or similar methods. Also many of
311 them will already be installed on a standard machine.
314 .. admonition:: Debian
316 You can use this command line to install all needed packages::
318 # apt-get install lvm2 ssh bridge-utils iproute iputils-arping \
319 python python-pyopenssl openssl python-pyparsing \
320 python-simplejson python-pyinotify
322 Setting up the environment for Ganeti
323 -------------------------------------
325 Configuring the network
326 +++++++++++++++++++++++
328 **Mandatory** on all nodes.
330 You can run Ganeti either in "bridge mode" or in "routed mode". In
331 bridge mode, the default, the instances network interfaces will be
332 attached to a software bridge running in dom0. Xen by default creates
333 such a bridge at startup, but your distribution might have a different
334 way to do things, and you'll definitely need to manually set it up under
337 Beware that the default name Ganeti uses is ``xen-br0`` (which was
338 used in Xen 2.0) while Xen 3.0 uses ``xenbr0`` by default. The default
339 bridge your Ganeti cluster will use for new instances can be specified
340 at cluster initialization time.
342 If you want to run in "routing mode" you need to specify that at cluster
343 init time (using the --nicparam option), and then no bridge will be
344 needed. In this mode instance traffic will be routed by dom0, instead of
347 In order to use "routing mode" under Xen, you'll need to change the
348 relevant parameters in the Xen config file. Under KVM instead, no config
349 change is necessary, but you still need to set up your network
350 interfaces correctly.
352 By default, under KVM, the "link" parameter you specify per-nic will
353 represent, if non-empty, a different routing table name or number to use
354 for your instances. This allows insulation between different instance
355 groups, and different routing policies between node traffic and instance
358 You will need to configure your routing table basic routes and rules
359 outside of ganeti. The vif scripts will only add /32 routes to your
360 instances, through their interface, in the table you specified (under
361 KVM, and in the main table under Xen).
363 .. admonition:: Bridging under Debian
365 The recommended way to configure the Xen bridge is to edit your
366 ``/etc/network/interfaces`` file and substitute your normal
367 Ethernet stanza with the following snippet::
370 iface xen-br0 inet static
371 address YOUR_IP_ADDRESS
374 broadcast YOUR_BROADCAST_ADDRESS
380 The following commands need to be executed on the local console:
385 To check if the bridge is setup, use the ``ip`` and ``brctl show``
389 9: xen-br0: <BROADCAST,MULTICAST,UP,10000> mtu 1500 qdisc noqueue
390 link/ether 00:20:fc:1e:d5:5d brd ff:ff:ff:ff:ff:ff
391 inet 10.1.1.200/24 brd 10.1.1.255 scope global xen-br0
392 inet6 fe80::220:fcff:fe1e:d55d/64 scope link
393 valid_lft forever preferred_lft forever
396 bridge name bridge id STP enabled interfaces
397 xen-br0 8000.0020fc1ed55d no eth0
402 **Mandatory** on all nodes.
404 The volume group is required to be at least 20GiB.
406 If you haven't configured your LVM volume group at install time you
407 need to do it before trying to initialize the Ganeti cluster. This is
408 done by formatting the devices/partitions you want to use for it and
409 then adding them to the relevant volume group::
412 vgcreate xenvg /dev/sda3
418 vgcreate xenvg /dev/sdb1 /dev/sdc1
420 If you want to add a device later you can do so with the *vgextend*
424 vgextend xenvg /dev/sdd1
426 Optional: it is recommended to configure LVM not to scan the DRBD
427 devices for physical volumes. This can be accomplished by editing
428 ``/etc/lvm/lvm.conf`` and adding the
429 ``/dev/drbd[0-9]+`` regular expression to the
430 ``filter`` variable, like this::
432 filter = ["r|/dev/cdrom|", "r|/dev/drbd[0-9]+|" ]
437 **Mandatory** on all nodes.
439 It's now time to install the Ganeti software itself. Download the
440 source from the project page at `<http://code.google.com/p/ganeti/>`_,
441 and install it (replace 2.0.0 with the latest version)::
443 tar xvzf ganeti-2.0.0.tar.gz
445 ./configure --localstatedir=/var --sysconfdir=/etc
448 mkdir /srv/ganeti/ /srv/ganeti/os /srv/ganeti/export
450 You also need to copy the file
451 ``doc/examples/ganeti.initd`` from the source archive
452 to ``/etc/init.d/ganeti`` and register it with your
453 distribution's startup scripts, for example in Debian::
455 update-rc.d ganeti defaults 20 80
457 In order to automatically restart failed instances, you need to setup
458 a cron job run the *ganeti-watcher* command. A sample cron file is
459 provided in the source at ``doc/examples/ganeti.cron`` and you can
460 copy that (eventually altering the path) to ``/etc/cron.d/ganeti``.
462 Installing the Operating System support packages
463 ++++++++++++++++++++++++++++++++++++++++++++++++
465 **Mandatory** on all nodes.
467 To be able to install instances you need to have an Operating System
468 installation script. An example OS that works under Debian and can
469 install Debian and Ubuntu instace OSes is provided on the project web
470 site. Download it from the project page and follow the instructions
471 in the ``README`` file. Here is the installation procedure (replace
472 0.7 with the latest version that is compatible with your ganeti
476 wget http://ganeti.googlecode.com/files/ganeti-instance-debootstrap-0.7.tar.gz
477 tar xzf ganeti-instance-debootstrap-0.7.tar.gz
478 cd ganeti-instance-debootstrap-0.7
483 In order to use this OS definition, you need to have internet access
484 from your nodes and have the *debootstrap*, *dump* and *restore*
485 commands installed on all nodes. Also, if the OS is configured to
486 partition the instance's disk in
487 ``/etc/default/ganeti-instance-debootstrap``, you will need *kpartx*
490 .. admonition:: Debian
492 Use this command on all nodes to install the required packages::
494 apt-get install debootstrap dump kpartx
496 Alternatively, you can create your own OS definitions. See the manpage
497 :manpage:`ganeti-os-interface`.
499 Initializing the cluster
500 ++++++++++++++++++++++++
502 **Mandatory** on one node per cluster.
504 The last step is to initialize the cluster. After you've repeated the
505 above process on all of your nodes, choose one as the master, and
508 gnt-cluster init <CLUSTERNAME>
510 The *CLUSTERNAME* is a hostname, which must be resolvable (e.g. it
511 must exist in DNS or in ``/etc/hosts``) by all the nodes in the
512 cluster. You must choose a name different from any of the nodes names
513 for a multi-node cluster. In general the best choice is to have a
514 unique name for a cluster, even if it consists of only one machine, as
515 you will be able to expand it later without any problems. Please note
516 that the hostname used for this must resolve to an IP address reserved
517 **exclusively** for this purpose, and cannot be the name of the first
520 If you want to use a bridge which is not ``xen-br0``, or no bridge at
521 all, use ``--nicparams``.
523 If the bridge name you are using is not ``xen-br0``, use the *-b
524 <BRIDGENAME>* option to specify the bridge name. In this case, you
525 should also use the *--master-netdev <BRIDGENAME>* option with the
526 same BRIDGENAME argument.
528 You can use a different name than ``xenvg`` for the volume group (but
529 note that the name must be identical on all nodes). In this case you
530 need to specify it by passing the *-g <VGNAME>* option to
531 ``gnt-cluster init``.
533 To set up the cluster as an HVM cluster, use the
534 ``--enabled-hypervisors=xen-hvm`` option to enable the HVM hypervisor
535 (you can also add ``,xen-pvm`` to enable the PVM one too). You will
536 also need to create the VNC cluster password file
537 ``/etc/ganeti/vnc-cluster-password`` which contains one line with the
538 default VNC password for the cluster.
540 To setup the cluster for KVM-only usage (KVM and Xen cannot be mixed),
541 pass ``--enabled-hypervisors=kvm`` to the init command.
543 You can also invoke the command with the ``--help`` option in order to
544 see all the possibilities.
546 Joining the nodes to the cluster
547 ++++++++++++++++++++++++++++++++
549 **Mandatory** for all the other nodes.
551 After you have initialized your cluster you need to join the other
552 nodes to it. You can do so by executing the following command on the
555 gnt-node add <NODENAME>
557 Separate replication network
558 ++++++++++++++++++++++++++++
562 Ganeti uses DRBD to mirror the disk of the virtual instances between
563 nodes. To use a dedicated network interface for this (in order to
564 improve performance or to enhance security) you need to configure an
565 additional interface for each node. Use the *-s* option with
566 ``gnt-cluster init`` and ``gnt-node add`` to specify the IP address of
567 this secondary interface to use for each node. Note that if you
568 specified this option at cluster setup time, you must afterwards use
569 it for every node add operation.
574 Execute the ``gnt-node list`` command to see all nodes in the
578 Node DTotal DFree MTotal MNode MFree Pinst Sinst
579 node1.example.com 197404 197404 2047 1896 125 0 0
581 Setting up and managing virtual instances
582 -----------------------------------------
584 Setting up virtual instances
585 ++++++++++++++++++++++++++++
587 This step shows how to setup a virtual instance with either
588 non-mirrored disks (``plain``) or with network mirrored disks
589 (``drbd``). All commands need to be executed on the Ganeti master
590 node (the one on which ``gnt-cluster init`` was run). Verify that the
591 OS scripts are present on all cluster nodes with ``gnt-os list``.
594 To create a virtual instance, you need a hostname which is resolvable
595 (DNS or ``/etc/hosts`` on all nodes). The following command will
596 create a non-mirrored instance for you::
598 gnt-instance add -t plain -s 1G -n node1 -o debootstrap instance1.example.com
599 * creating instance disks...
600 adding instance instance1.example.com to cluster config
601 - INFO: Waiting for instance instance1.example.com to sync disks.
602 - INFO: Instance instance1.example.com's disks are in sync.
603 creating os for instance instance1.example.com on node node1.example.com
604 * running the instance OS create scripts...
605 * starting instance...
607 The above instance will have no network interface enabled. You can
608 access it over the virtual console with ``gnt-instance console
609 inst1``. There is no password for root. As this is a Debian instance,
610 you can modify the ``/etc/network/interfaces`` file to setup the
611 network interface (eth0 is the name of the interface provided to the
614 To create a network mirrored instance, change the argument to the *-t*
615 option from ``plain`` to ``drbd`` and specify the node on which the
616 mirror should reside with the second value of the *--node* option,
617 like this (note that the command output includes timestamps which have
618 been removed for clarity)::
620 # gnt-instance add -t drbd -s 1G -n node1:node2 -o debootstrap instance2
621 * creating instance disks...
622 adding instance instance2.example.com to cluster config
623 - INFO: Waiting for instance instance2.example.com to sync disks.
624 - INFO: - device disk/0: 35.50% done, 11 estimated seconds remaining
625 - INFO: - device disk/0: 100.00% done, 0 estimated seconds remaining
626 - INFO: Instance instance2.example.com's disks are in sync.
627 creating os for instance instance2.example.com on node node1.example.com
628 * running the instance OS create scripts...
629 * starting instance...
631 Managing virtual instances
632 ++++++++++++++++++++++++++
634 All commands need to be executed on the Ganeti master node.
636 To access the console of an instance, run::
638 gnt-instance console INSTANCENAME
640 To shutdown an instance, run::
642 gnt-instance shutdown INSTANCENAME
644 To startup an instance, run::
646 gnt-instance startup INSTANCENAME
648 To failover an instance to its secondary node (only possible with
649 ``drbd`` disk templates), run::
651 gnt-instance failover INSTANCENAME
653 For more instance and cluster administration details, see the
654 *Ganeti administrator's guide*.
656 .. vim: set textwidth=72 :