root / doc / install.rst @ d98b37a1
History | View | Annotate | Download (21 kB)
1 | 28e15341 | Iustin Pop | Ganeti installation tutorial |
---|---|---|---|
2 | 28e15341 | Iustin Pop | ============================ |
3 | 28e15341 | Iustin Pop | |
4 | 77031881 | Iustin Pop | Documents Ganeti version 2.0 |
5 | 28e15341 | Iustin Pop | |
6 | 28e15341 | Iustin Pop | .. contents:: |
7 | 28e15341 | Iustin Pop | |
8 | 28e15341 | Iustin Pop | Introduction |
9 | 28e15341 | Iustin Pop | ------------ |
10 | 28e15341 | Iustin Pop | |
11 | 28e15341 | Iustin Pop | Ganeti is a cluster virtualization management system based on Xen or |
12 | 28e15341 | Iustin Pop | KVM. This document explains how to bootstrap a Ganeti node (Xen |
13 | 28e15341 | Iustin Pop | *dom0*), create a running cluster and install virtual instance (Xen |
14 | 28e15341 | Iustin Pop | *domU*). You need to repeat most of the steps in this document for |
15 | 28e15341 | Iustin Pop | every node you want to install, but of course we recommend creating |
16 | 28e15341 | Iustin Pop | some semi-automatic procedure if you plan to deploy Ganeti on a |
17 | 28e15341 | Iustin Pop | medium/large scale. |
18 | 28e15341 | Iustin Pop | |
19 | 28e15341 | Iustin Pop | A basic Ganeti terminology glossary is provided in the introductory |
20 | 28e15341 | Iustin Pop | section of the *Ganeti administrator's guide*. Please refer to that |
21 | 28e15341 | Iustin Pop | document if you are uncertain about the terms we are using. |
22 | 28e15341 | Iustin Pop | |
23 | 28e15341 | Iustin Pop | Ganeti has been developed for Linux and is distribution-agnostic. |
24 | 28e15341 | Iustin Pop | This documentation will use Debian Lenny as an example system but the |
25 | 28e15341 | Iustin Pop | examples can easily be translated to any other distribution. ou are |
26 | 28e15341 | Iustin Pop | expected to be familiar with your distribution, its package management |
27 | 28e15341 | Iustin Pop | system, and Xen or KVM before trying to use Ganeti. |
28 | 28e15341 | Iustin Pop | |
29 | 28e15341 | Iustin Pop | This document is divided into two main sections: |
30 | 28e15341 | Iustin Pop | |
31 | 28e15341 | Iustin Pop | - Installation of the base system and base components |
32 | 28e15341 | Iustin Pop | |
33 | 28e15341 | Iustin Pop | - Configuration of the environment for Ganeti |
34 | 28e15341 | Iustin Pop | |
35 | 28e15341 | Iustin Pop | Each of these is divided into sub-sections. While a full Ganeti system |
36 | 28e15341 | Iustin Pop | will need all of the steps specified, some are not strictly required |
37 | 28e15341 | Iustin Pop | for every environment. Which ones they are, and why, is specified in |
38 | 28e15341 | Iustin Pop | the corresponding sections. |
39 | 28e15341 | Iustin Pop | |
40 | 28e15341 | Iustin Pop | Installing the base system and base components |
41 | 28e15341 | Iustin Pop | ---------------------------------------------- |
42 | 28e15341 | Iustin Pop | |
43 | 28e15341 | Iustin Pop | Hardware requirements |
44 | 28e15341 | Iustin Pop | +++++++++++++++++++++ |
45 | 28e15341 | Iustin Pop | |
46 | 28e15341 | Iustin Pop | Any system supported by your Linux distribution is fine. 64-bit |
47 | 28e15341 | Iustin Pop | systems are better as they can support more memory. |
48 | 28e15341 | Iustin Pop | |
49 | 28e15341 | Iustin Pop | Any disk drive recognized by Linux (``IDE``/``SCSI``/``SATA``/etc.) |
50 | 28e15341 | Iustin Pop | is supported in Ganeti. Note that no shared storage (e.g. ``SAN``) is |
51 | 28e15341 | Iustin Pop | needed to get high-availability features (but of course, one can be |
52 | 28e15341 | Iustin Pop | used to store the images). It is highly recommended to use more than |
53 | 28e15341 | Iustin Pop | one disk drive to improve speed. But Ganeti also works with one disk |
54 | 28e15341 | Iustin Pop | per machine. |
55 | 28e15341 | Iustin Pop | |
56 | 28e15341 | Iustin Pop | Installing the base system |
57 | 28e15341 | Iustin Pop | ++++++++++++++++++++++++++ |
58 | 28e15341 | Iustin Pop | |
59 | 28e15341 | Iustin Pop | **Mandatory** on all nodes. |
60 | 28e15341 | Iustin Pop | |
61 | 28e15341 | Iustin Pop | It is advised to start with a clean, minimal install of the operating |
62 | 28e15341 | Iustin Pop | system. The only requirement you need to be aware of at this stage is |
63 | 28e15341 | Iustin Pop | to partition leaving enough space for a big (**minimum** 20GiB) LVM |
64 | 28e15341 | Iustin Pop | volume group which will then host your instance filesystems, if you |
65 | 28e15341 | Iustin Pop | want to use all Ganeti features. The volume group name Ganeti 2.0 uses |
66 | 28e15341 | Iustin Pop | (by default) is ``xenvg``. |
67 | 28e15341 | Iustin Pop | |
68 | 28e15341 | Iustin Pop | You can also use file-based storage only, without LVM, but this setup |
69 | 28e15341 | Iustin Pop | is not detailed in this document. |
70 | 28e15341 | Iustin Pop | |
71 | 28e15341 | Iustin Pop | |
72 | 28e15341 | Iustin Pop | While you can use an existing system, please note that the Ganeti |
73 | 28e15341 | Iustin Pop | installation is intrusive in terms of changes to the system |
74 | 28e15341 | Iustin Pop | configuration, and it's best to use a newly-installed system without |
75 | 28e15341 | Iustin Pop | important data on it. |
76 | 28e15341 | Iustin Pop | |
77 | 28e15341 | Iustin Pop | Also, for best results, it's advised that the nodes have as much as |
78 | 28e15341 | Iustin Pop | possible the same hardware and software configuration. This will make |
79 | 28e15341 | Iustin Pop | administration much easier. |
80 | 28e15341 | Iustin Pop | |
81 | 28e15341 | Iustin Pop | Hostname issues |
82 | 28e15341 | Iustin Pop | ~~~~~~~~~~~~~~~ |
83 | 28e15341 | Iustin Pop | |
84 | 28e15341 | Iustin Pop | Note that Ganeti requires the hostnames of the systems (i.e. what the |
85 | 28e15341 | Iustin Pop | ``hostname`` command outputs to be a fully-qualified name, not a short |
86 | 28e15341 | Iustin Pop | name. In other words, you should use *node1.example.com* as a hostname |
87 | 28e15341 | Iustin Pop | and not just *node1*. |
88 | 28e15341 | Iustin Pop | |
89 | 28e15341 | Iustin Pop | .. admonition:: Debian |
90 | 28e15341 | Iustin Pop | |
91 | 28e15341 | Iustin Pop | Debian Lenny and Etch configures the hostname differently than you |
92 | 28e15341 | Iustin Pop | need it for Ganeti. For example, this is what Etch puts in |
93 | 28e15341 | Iustin Pop | ``/etc/hosts`` in certain situations:: |
94 | 28e15341 | Iustin Pop | |
95 | 28e15341 | Iustin Pop | 127.0.0.1 localhost |
96 | 28e15341 | Iustin Pop | 127.0.1.1 node1.example.com node1 |
97 | 28e15341 | Iustin Pop | |
98 | 28e15341 | Iustin Pop | but for Ganeti you need to have:: |
99 | 28e15341 | Iustin Pop | |
100 | 28e15341 | Iustin Pop | 127.0.0.1 localhost |
101 | 28e15341 | Iustin Pop | 192.168.1.1 node1.example.com node1 |
102 | 28e15341 | Iustin Pop | |
103 | 28e15341 | Iustin Pop | replacing ``192.168.1.1`` with your node's address. Also, the file |
104 | 28e15341 | Iustin Pop | ``/etc/hostname`` which configures the hostname of the system |
105 | 28e15341 | Iustin Pop | should contain ``node1.example.com`` and not just ``node1`` (you |
106 | 28e15341 | Iustin Pop | need to run the command ``/etc/init.d/hostname.sh start`` after |
107 | 28e15341 | Iustin Pop | changing the file). |
108 | 28e15341 | Iustin Pop | |
109 | 28e15341 | Iustin Pop | Installing Xen |
110 | 28e15341 | Iustin Pop | ++++++++++++++ |
111 | 28e15341 | Iustin Pop | |
112 | 28e15341 | Iustin Pop | **Mandatory** on all nodes. |
113 | 28e15341 | Iustin Pop | |
114 | 28e15341 | Iustin Pop | While Ganeti is developed with the ability to modularly run on |
115 | 28e15341 | Iustin Pop | different virtualization environments in mind the only two currently |
116 | 28e15341 | Iustin Pop | useable on a live system are Xen and KVM. Supported |
117 | 28e15341 | Iustin Pop | Xen versions are: 3.0.3, 3.0.4 and 3.1. |
118 | 28e15341 | Iustin Pop | |
119 | 28e15341 | Iustin Pop | Please follow your distribution's recommended way to install and set |
120 | 28e15341 | Iustin Pop | up Xen, or install Xen from the upstream source, if you wish, |
121 | 28e15341 | Iustin Pop | following their manual. For KVM, make sure you have a KVM-enabled |
122 | 28e15341 | Iustin Pop | kernel and the KVM tools. |
123 | 28e15341 | Iustin Pop | |
124 | 28e15341 | Iustin Pop | After installing either hypervisor, you need to reboot into your new |
125 | 28e15341 | Iustin Pop | system. On some distributions this might involve configuring GRUB |
126 | 28e15341 | Iustin Pop | appropriately, whereas others will configure it automatically when you |
127 | 28e15341 | Iustin Pop | install the respective kernels. |
128 | 28e15341 | Iustin Pop | |
129 | 28e15341 | Iustin Pop | .. admonition:: Debian |
130 | 28e15341 | Iustin Pop | |
131 | 28e15341 | Iustin Pop | Under Lenny or Etch you can install the relevant |
132 | 28e15341 | Iustin Pop | ``xen-linux-system`` package, which will pull in both the |
133 | 28e15341 | Iustin Pop | hypervisor and the relevant kernel. Also, if you are installing a |
134 | 28e15341 | Iustin Pop | 32-bit Lenny/Etch, you should install the ``libc6-xen`` package |
135 | 28e15341 | Iustin Pop | (run ``apt-get install libc6-xen``). |
136 | 28e15341 | Iustin Pop | |
137 | 28e15341 | Iustin Pop | Xen settings |
138 | 28e15341 | Iustin Pop | ~~~~~~~~~~~~ |
139 | 28e15341 | Iustin Pop | |
140 | 28e15341 | Iustin Pop | It's recommended that dom0 is restricted to a low amount of memory |
141 | 28e15341 | Iustin Pop | (512MiB or 1GiB is reasonable) and that memory ballooning is disabled |
142 | 28e15341 | Iustin Pop | in the file ``/etc/xen/xend-config.sxp`` by setting |
143 | 28e15341 | Iustin Pop | the value ``dom0-min-mem`` to 0, |
144 | 28e15341 | Iustin Pop | like this:: |
145 | 28e15341 | Iustin Pop | |
146 | 28e15341 | Iustin Pop | (dom0-min-mem 0) |
147 | 28e15341 | Iustin Pop | |
148 | 28e15341 | Iustin Pop | For optimum performance when running both CPU and I/O intensive |
149 | 28e15341 | Iustin Pop | instances, it's also recommended that the dom0 is restricted to one |
150 | 28e15341 | Iustin Pop | CPU only, for example by booting with the kernel parameter ``nosmp``. |
151 | 28e15341 | Iustin Pop | |
152 | 28e15341 | Iustin Pop | It is recommended that you disable xen's automatic save of virtual |
153 | 28e15341 | Iustin Pop | machines at system shutdown and subsequent restore of them at reboot. |
154 | 28e15341 | Iustin Pop | To obtain this make sure the variable ``XENDOMAINS_SAVE`` in the file |
155 | 28e15341 | Iustin Pop | ``/etc/default/xendomains`` is set to an empty value. |
156 | 28e15341 | Iustin Pop | |
157 | 28e15341 | Iustin Pop | .. admonition:: Debian |
158 | 28e15341 | Iustin Pop | |
159 | 28e15341 | Iustin Pop | Besides the ballooning change which you need to set in |
160 | 28e15341 | Iustin Pop | ``/etc/xen/xend-config.sxp``, you need to set the memory and nosmp |
161 | 28e15341 | Iustin Pop | parameters in the file ``/boot/grub/menu.lst``. You need to modify |
162 | 28e15341 | Iustin Pop | the variable ``xenhopt`` to add ``dom0_mem=1024M`` like this:: |
163 | 28e15341 | Iustin Pop | |
164 | 28e15341 | Iustin Pop | ## Xen hypervisor options to use with the default Xen boot option |
165 | 28e15341 | Iustin Pop | # xenhopt=dom0_mem=1024M |
166 | 28e15341 | Iustin Pop | |
167 | 28e15341 | Iustin Pop | and the ``xenkopt`` needs to include the ``nosmp`` option like |
168 | 28e15341 | Iustin Pop | this:: |
169 | 28e15341 | Iustin Pop | |
170 | 28e15341 | Iustin Pop | ## Xen Linux kernel options to use with the default Xen boot option |
171 | 28e15341 | Iustin Pop | # xenkopt=nosmp |
172 | 28e15341 | Iustin Pop | |
173 | 28e15341 | Iustin Pop | Any existing parameters can be left in place: it's ok to have |
174 | 28e15341 | Iustin Pop | ``xenkopt=console=tty0 nosmp``, for example. After modifying the |
175 | 28e15341 | Iustin Pop | files, you need to run:: |
176 | 28e15341 | Iustin Pop | |
177 | 28e15341 | Iustin Pop | /sbin/update-grub |
178 | 28e15341 | Iustin Pop | |
179 | 28e15341 | Iustin Pop | If you want to run HVM instances too with Ganeti and want VNC access |
180 | 28e15341 | Iustin Pop | to the console of your instances, set the following two entries in |
181 | 28e15341 | Iustin Pop | ``/etc/xen/xend-config.sxp``:: |
182 | 28e15341 | Iustin Pop | |
183 | 28e15341 | Iustin Pop | (vnc-listen '0.0.0.0') (vncpasswd '') |
184 | 28e15341 | Iustin Pop | |
185 | 28e15341 | Iustin Pop | You need to restart the Xen daemon for these settings to take effect:: |
186 | 28e15341 | Iustin Pop | |
187 | 28e15341 | Iustin Pop | /etc/init.d/xend restart |
188 | 28e15341 | Iustin Pop | |
189 | 28e15341 | Iustin Pop | Selecting the instance kernel |
190 | 28e15341 | Iustin Pop | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
191 | 28e15341 | Iustin Pop | |
192 | 28e15341 | Iustin Pop | After you have installed Xen, you need to tell Ganeti exactly what |
193 | 28e15341 | Iustin Pop | kernel to use for the instances it will create. This is done by |
194 | 28e15341 | Iustin Pop | creating a symlink from your actual kernel to |
195 | 28e15341 | Iustin Pop | ``/boot/vmlinuz-2.6-xenU``, and one from your initrd |
196 | 28e15341 | Iustin Pop | to ``/boot/initrd-2.6-xenU``. Note that if you don't |
197 | 28e15341 | Iustin Pop | use an initrd for the domU kernel, you don't need |
198 | 28e15341 | Iustin Pop | to create the initrd symlink. |
199 | 28e15341 | Iustin Pop | |
200 | 28e15341 | Iustin Pop | .. admonition:: Debian |
201 | 28e15341 | Iustin Pop | |
202 | 28e15341 | Iustin Pop | After installation of the ``xen-linux-system`` package, you need to |
203 | 28e15341 | Iustin Pop | run (replace the exact version number with the one you have):: |
204 | 28e15341 | Iustin Pop | |
205 | 28e15341 | Iustin Pop | cd /boot |
206 | 28e15341 | Iustin Pop | ln -s vmlinuz-2.6.26-1-xen-amd64 vmlinuz-2.6-xenU |
207 | 28e15341 | Iustin Pop | ln -s initrd.img-2.6.26-1-xen-amd64 initrd-2.6-xenU |
208 | 28e15341 | Iustin Pop | |
209 | 28e15341 | Iustin Pop | Installing DRBD |
210 | 28e15341 | Iustin Pop | +++++++++++++++ |
211 | 28e15341 | Iustin Pop | |
212 | 28e15341 | Iustin Pop | Recommended on all nodes: DRBD_ is required if you want to use the |
213 | 28e15341 | Iustin Pop | high availability (HA) features of Ganeti, but optional if you don't |
214 | 28e15341 | Iustin Pop | require HA or only run Ganeti on single-node clusters. You can upgrade |
215 | 28e15341 | Iustin Pop | a non-HA cluster to an HA one later, but you might need to export and |
216 | 28e15341 | Iustin Pop | re-import all your instances to take advantage of the new features. |
217 | 28e15341 | Iustin Pop | |
218 | 28e15341 | Iustin Pop | .. _DRBD: http://www.drbd.org/ |
219 | 28e15341 | Iustin Pop | |
220 | 28e15341 | Iustin Pop | Supported DRBD versions: 8.0.x. It's recommended to have at least |
221 | 28e15341 | Iustin Pop | version 8.0.12. |
222 | 28e15341 | Iustin Pop | |
223 | 28e15341 | Iustin Pop | Now the bad news: unless your distribution already provides it |
224 | 28e15341 | Iustin Pop | installing DRBD might involve recompiling your kernel or anyway |
225 | 28e15341 | Iustin Pop | fiddling with it. Hopefully at least the Xen-ified kernel source to |
226 | 28e15341 | Iustin Pop | start from will be provided. |
227 | 28e15341 | Iustin Pop | |
228 | 28e15341 | Iustin Pop | The good news is that you don't need to configure DRBD at all. Ganeti |
229 | 28e15341 | Iustin Pop | will do it for you for every instance you set up. If you have the |
230 | 28e15341 | Iustin Pop | DRBD utils installed and the module in your kernel you're fine. Please |
231 | 28e15341 | Iustin Pop | check that your system is configured to load the module at every boot, |
232 | 28e15341 | Iustin Pop | and that it passes the following option to the module |
233 | 28e15341 | Iustin Pop | ``minor_count=255``. This will allow you to use up to 128 instances |
234 | 28e15341 | Iustin Pop | per node (for most clusters 128 should be enough, though). |
235 | 28e15341 | Iustin Pop | |
236 | 28e15341 | Iustin Pop | .. admonition:: Debian |
237 | 28e15341 | Iustin Pop | |
238 | 28e15341 | Iustin Pop | On Debian, you can just install (build) the DRBD 8.0.x module with |
239 | 28e15341 | Iustin Pop | the following commands (make sure you are running the Xen kernel):: |
240 | 28e15341 | Iustin Pop | |
241 | 28e15341 | Iustin Pop | apt-get install drbd8-source drbd8-utils |
242 | 28e15341 | Iustin Pop | m-a update |
243 | 28e15341 | Iustin Pop | m-a a-i drbd8 |
244 | 28e15341 | Iustin Pop | echo drbd minor_count=128 >> /etc/modules |
245 | 28e15341 | Iustin Pop | depmod -a |
246 | 28e15341 | Iustin Pop | modprobe drbd minor_count=128 |
247 | 28e15341 | Iustin Pop | |
248 | 28e15341 | Iustin Pop | It is also recommended that you comment out the default resources |
249 | 28e15341 | Iustin Pop | in the ``/etc/drbd.conf`` file, so that the init script doesn't try |
250 | 28e15341 | Iustin Pop | to configure any drbd devices. You can do this by prefixing all |
251 | 28e15341 | Iustin Pop | *resource* lines in the file with the keyword *skip*, like this:: |
252 | 28e15341 | Iustin Pop | |
253 | 28e15341 | Iustin Pop | skip resource r0 { |
254 | 28e15341 | Iustin Pop | ... |
255 | 28e15341 | Iustin Pop | } |
256 | 28e15341 | Iustin Pop | |
257 | 28e15341 | Iustin Pop | skip resource "r1" { |
258 | 28e15341 | Iustin Pop | ... |
259 | 28e15341 | Iustin Pop | } |
260 | 28e15341 | Iustin Pop | |
261 | 28e15341 | Iustin Pop | Other required software |
262 | 28e15341 | Iustin Pop | +++++++++++++++++++++++ |
263 | 28e15341 | Iustin Pop | |
264 | 28e15341 | Iustin Pop | Besides Xen and DRBD, you will need to install the following (on all |
265 | 28e15341 | Iustin Pop | nodes): |
266 | 28e15341 | Iustin Pop | |
267 | 28e15341 | Iustin Pop | - LVM version 2, `<http://sourceware.org/lvm2/>`_ |
268 | 28e15341 | Iustin Pop | |
269 | 28e15341 | Iustin Pop | - OpenSSL, `<http://www.openssl.org/>`_ |
270 | 28e15341 | Iustin Pop | |
271 | 28e15341 | Iustin Pop | - OpenSSH, `<http://www.openssh.com/portable.html>`_ |
272 | 28e15341 | Iustin Pop | |
273 | 28e15341 | Iustin Pop | - bridge utilities, `<http://bridge.sourceforge.net/>`_ |
274 | 28e15341 | Iustin Pop | |
275 | 28e15341 | Iustin Pop | - iproute2, `<http://developer.osdl.org/dev/iproute2>`_ |
276 | 28e15341 | Iustin Pop | |
277 | 28e15341 | Iustin Pop | - arping (part of iputils package), |
278 | 28e15341 | Iustin Pop | `<ftp://ftp.inr.ac.ru/ip-routing/iputils-current.tar.gz>`_ |
279 | 28e15341 | Iustin Pop | |
280 | 28e15341 | Iustin Pop | - Python version 2.4 or 2.5, `<http://www.python.org>`_ |
281 | 28e15341 | Iustin Pop | |
282 | 28e15341 | Iustin Pop | - Python OpenSSL bindings, `<http://pyopenssl.sourceforge.net/>`_ |
283 | 28e15341 | Iustin Pop | |
284 | 28e15341 | Iustin Pop | |
285 | 28e15341 | Iustin Pop | - simplejson Python module, `<http://www.undefined.org/python/#simplejson>`_ |
286 | 28e15341 | Iustin Pop | |
287 | 28e15341 | Iustin Pop | - pyparsing Python module, `<http://pyparsing.wikispaces.com/>`_ |
288 | 28e15341 | Iustin Pop | |
289 | 28e15341 | Iustin Pop | These programs are supplied as part of most Linux distributions, so |
290 | 28e15341 | Iustin Pop | usually they can be installed via apt or similar methods. Also many of |
291 | 28e15341 | Iustin Pop | them will already be installed on a standard machine. |
292 | 28e15341 | Iustin Pop | |
293 | 28e15341 | Iustin Pop | |
294 | 28e15341 | Iustin Pop | .. admonition:: Debian |
295 | 28e15341 | Iustin Pop | |
296 | 28e15341 | Iustin Pop | You can use this command line to install all needed packages:: |
297 | 28e15341 | Iustin Pop | |
298 | 28e15341 | Iustin Pop | # apt-get install lvm2 ssh bridge-utils iproute iputils-arping \ |
299 | 28e15341 | Iustin Pop | python python-pyopenssl openssl python-pyparsing python-simplejson |
300 | 28e15341 | Iustin Pop | |
301 | 28e15341 | Iustin Pop | Setting up the environment for Ganeti |
302 | 28e15341 | Iustin Pop | ------------------------------------- |
303 | 28e15341 | Iustin Pop | |
304 | 28e15341 | Iustin Pop | Configuring the network |
305 | 28e15341 | Iustin Pop | +++++++++++++++++++++++ |
306 | 28e15341 | Iustin Pop | |
307 | 28e15341 | Iustin Pop | **Mandatory** on all nodes. |
308 | 28e15341 | Iustin Pop | |
309 | 28e15341 | Iustin Pop | Ganeti relies on Xen running in "bridge mode", which means the |
310 | 28e15341 | Iustin Pop | instances network interfaces will be attached to a software bridge |
311 | 28e15341 | Iustin Pop | running in dom0. Xen by default creates such a bridge at startup, but |
312 | 28e15341 | Iustin Pop | your distribution might have a different way to do things. |
313 | 28e15341 | Iustin Pop | |
314 | 28e15341 | Iustin Pop | Beware that the default name Ganeti uses is ``xen-br0`` (which was |
315 | 28e15341 | Iustin Pop | used in Xen 2.0) while Xen 3.0 uses ``xenbr0`` by default. The default |
316 | 28e15341 | Iustin Pop | bridge your Ganeti cluster will use for new instances can be specified |
317 | 28e15341 | Iustin Pop | at cluster initialization time. |
318 | 28e15341 | Iustin Pop | |
319 | 28e15341 | Iustin Pop | .. admonition:: Debian |
320 | 28e15341 | Iustin Pop | |
321 | 28e15341 | Iustin Pop | The recommended way to configure the Xen bridge is to edit your |
322 | 28e15341 | Iustin Pop | ``/etc/network/interfaces`` file and substitute your normal |
323 | 28e15341 | Iustin Pop | Ethernet stanza with the following snippet:: |
324 | 28e15341 | Iustin Pop | |
325 | 28e15341 | Iustin Pop | auto xen-br0 |
326 | 28e15341 | Iustin Pop | iface xen-br0 inet static |
327 | 28e15341 | Iustin Pop | address YOUR_IP_ADDRESS |
328 | 28e15341 | Iustin Pop | netmask YOUR_NETMASK |
329 | 28e15341 | Iustin Pop | network YOUR_NETWORK |
330 | 28e15341 | Iustin Pop | broadcast YOUR_BROADCAST_ADDRESS |
331 | 28e15341 | Iustin Pop | gateway YOUR_GATEWAY |
332 | 28e15341 | Iustin Pop | bridge_ports eth0 |
333 | 28e15341 | Iustin Pop | bridge_stp off |
334 | 28e15341 | Iustin Pop | bridge_fd 0 |
335 | 28e15341 | Iustin Pop | |
336 | 28e15341 | Iustin Pop | The following commands need to be executed on the local console: |
337 | 28e15341 | Iustin Pop | |
338 | 28e15341 | Iustin Pop | ifdown eth0 |
339 | 28e15341 | Iustin Pop | ifup xen-br0 |
340 | 28e15341 | Iustin Pop | |
341 | 28e15341 | Iustin Pop | To check if the bridge is setup, use the ``ip`` and ``brctl show`` |
342 | 28e15341 | Iustin Pop | commands:: |
343 | 28e15341 | Iustin Pop | |
344 | 28e15341 | Iustin Pop | # ip a show xen-br0 |
345 | 28e15341 | Iustin Pop | 9: xen-br0: <BROADCAST,MULTICAST,UP,10000> mtu 1500 qdisc noqueue |
346 | 28e15341 | Iustin Pop | link/ether 00:20:fc:1e:d5:5d brd ff:ff:ff:ff:ff:ff |
347 | 28e15341 | Iustin Pop | inet 10.1.1.200/24 brd 10.1.1.255 scope global xen-br0 |
348 | 28e15341 | Iustin Pop | inet6 fe80::220:fcff:fe1e:d55d/64 scope link |
349 | 28e15341 | Iustin Pop | valid_lft forever preferred_lft forever |
350 | 28e15341 | Iustin Pop | |
351 | 28e15341 | Iustin Pop | # brctl show xen-br0 |
352 | 28e15341 | Iustin Pop | bridge name bridge id STP enabled interfaces |
353 | 28e15341 | Iustin Pop | xen-br0 8000.0020fc1ed55d no eth0 |
354 | 28e15341 | Iustin Pop | |
355 | 28e15341 | Iustin Pop | Configuring LVM |
356 | 28e15341 | Iustin Pop | +++++++++++++++ |
357 | 28e15341 | Iustin Pop | |
358 | 28e15341 | Iustin Pop | **Mandatory** on all nodes. |
359 | 28e15341 | Iustin Pop | |
360 | 28e15341 | Iustin Pop | The volume group is required to be at least 20GiB. |
361 | 28e15341 | Iustin Pop | |
362 | 28e15341 | Iustin Pop | If you haven't configured your LVM volume group at install time you |
363 | 28e15341 | Iustin Pop | need to do it before trying to initialize the Ganeti cluster. This is |
364 | 28e15341 | Iustin Pop | done by formatting the devices/partitions you want to use for it and |
365 | 28e15341 | Iustin Pop | then adding them to the relevant volume group:: |
366 | 28e15341 | Iustin Pop | |
367 | 28e15341 | Iustin Pop | pvcreate /dev/sda3 |
368 | 28e15341 | Iustin Pop | vgcreate xenvg /dev/sda3 |
369 | 28e15341 | Iustin Pop | |
370 | 28e15341 | Iustin Pop | or:: |
371 | 28e15341 | Iustin Pop | |
372 | 28e15341 | Iustin Pop | pvcreate /dev/sdb1 |
373 | 28e15341 | Iustin Pop | pvcreate /dev/sdc1 |
374 | 28e15341 | Iustin Pop | vgcreate xenvg /dev/sdb1 /dev/sdc1 |
375 | 28e15341 | Iustin Pop | |
376 | 28e15341 | Iustin Pop | If you want to add a device later you can do so with the *vgextend* |
377 | 28e15341 | Iustin Pop | command:: |
378 | 28e15341 | Iustin Pop | |
379 | 28e15341 | Iustin Pop | pvcreate /dev/sdd1 |
380 | 28e15341 | Iustin Pop | vgextend xenvg /dev/sdd1 |
381 | 28e15341 | Iustin Pop | |
382 | 28e15341 | Iustin Pop | Optional: it is recommended to configure LVM not to scan the DRBD |
383 | 28e15341 | Iustin Pop | devices for physical volumes. This can be accomplished by editing |
384 | 28e15341 | Iustin Pop | ``/etc/lvm/lvm.conf`` and adding the |
385 | 28e15341 | Iustin Pop | ``/dev/drbd[0-9]+`` regular expression to the |
386 | 28e15341 | Iustin Pop | ``filter`` variable, like this:: |
387 | 28e15341 | Iustin Pop | |
388 | 28e15341 | Iustin Pop | filter = ["r|/dev/cdrom|", "r|/dev/drbd[0-9]+|" ] |
389 | 28e15341 | Iustin Pop | |
390 | 28e15341 | Iustin Pop | Installing Ganeti |
391 | 28e15341 | Iustin Pop | +++++++++++++++++ |
392 | 28e15341 | Iustin Pop | |
393 | 28e15341 | Iustin Pop | **Mandatory** on all nodes. |
394 | 28e15341 | Iustin Pop | |
395 | 28e15341 | Iustin Pop | It's now time to install the Ganeti software itself. Download the |
396 | 28e15341 | Iustin Pop | source from the project page at `<http://code.google.com/p/ganeti/>`_, |
397 | 28e15341 | Iustin Pop | and install it (replace 2.0.0 with the latest version):: |
398 | 28e15341 | Iustin Pop | |
399 | 28e15341 | Iustin Pop | tar xvzf ganeti-2.0.0.tar.gz |
400 | 28e15341 | Iustin Pop | cd ganeti-2.0.0 |
401 | 28e15341 | Iustin Pop | ./configure --localstatedir=/var --sysconfdir=/etc |
402 | 28e15341 | Iustin Pop | make |
403 | 28e15341 | Iustin Pop | make install |
404 | 28e15341 | Iustin Pop | mkdir /srv/ganeti/ /srv/ganeti/os /srv/ganeti/export |
405 | 28e15341 | Iustin Pop | |
406 | 28e15341 | Iustin Pop | You also need to copy the file |
407 | 28e15341 | Iustin Pop | ``doc/examples/ganeti.initd`` from the source archive |
408 | 28e15341 | Iustin Pop | to ``/etc/init.d/ganeti`` and register it with your |
409 | 28e15341 | Iustin Pop | distribution's startup scripts, for example in Debian:: |
410 | 28e15341 | Iustin Pop | |
411 | 28e15341 | Iustin Pop | update-rc.d ganeti defaults 20 80 |
412 | 28e15341 | Iustin Pop | |
413 | 28e15341 | Iustin Pop | In order to automatically restart failed instances, you need to setup |
414 | 28e15341 | Iustin Pop | a cron job run the *ganeti-watcher* command. A sample cron file is |
415 | 28e15341 | Iustin Pop | provided in the source at ``doc/examples/ganeti.cron`` and you can |
416 | 28e15341 | Iustin Pop | copy that (eventually altering the path) to ``/etc/cron.d/ganeti``. |
417 | 28e15341 | Iustin Pop | |
418 | 28e15341 | Iustin Pop | Installing the Operating System support packages |
419 | 28e15341 | Iustin Pop | ++++++++++++++++++++++++++++++++++++++++++++++++ |
420 | 28e15341 | Iustin Pop | |
421 | 28e15341 | Iustin Pop | **Mandatory** on all nodes. |
422 | 28e15341 | Iustin Pop | |
423 | 28e15341 | Iustin Pop | To be able to install instances you need to have an Operating System |
424 | 28e15341 | Iustin Pop | installation script. An example OS that works under Debian and can |
425 | 28e15341 | Iustin Pop | install Debian and Ubuntu instace OSes is provided on the project web |
426 | 28e15341 | Iustin Pop | site. Download it from the project page and follow the instructions |
427 | 28e15341 | Iustin Pop | in the ``README`` file. Here is the installation procedure (replace |
428 | 28e15341 | Iustin Pop | 0.7 with the latest version that is compatible with your ganeti |
429 | 28e15341 | Iustin Pop | version):: |
430 | 28e15341 | Iustin Pop | |
431 | 28e15341 | Iustin Pop | cd /usr/local/src/ |
432 | 28e15341 | Iustin Pop | wget http://ganeti.googlecode.com/files/ganeti-instance-debootstrap-0.7.tar.gz |
433 | 28e15341 | Iustin Pop | tar xzf ganeti-instance-debootstrap-0.7.tar.gz |
434 | 28e15341 | Iustin Pop | cd ganeti-instance-debootstrap-0.7 |
435 | 28e15341 | Iustin Pop | ./configure |
436 | 28e15341 | Iustin Pop | make |
437 | 28e15341 | Iustin Pop | make install |
438 | 28e15341 | Iustin Pop | |
439 | 28e15341 | Iustin Pop | In order to use this OS definition, you need to have internet access |
440 | 28e15341 | Iustin Pop | from your nodes and have the *debootstrap*, *dump* and *restore* |
441 | 28e15341 | Iustin Pop | commands installed on all nodes. Also, if the OS is configured to |
442 | 28e15341 | Iustin Pop | partition the instance's disk in |
443 | 28e15341 | Iustin Pop | ``/etc/default/ganeti-instance-debootstrap``, you will need *kpartx* |
444 | 28e15341 | Iustin Pop | installed. |
445 | 28e15341 | Iustin Pop | |
446 | 28e15341 | Iustin Pop | .. admonition:: Debian |
447 | 28e15341 | Iustin Pop | |
448 | 28e15341 | Iustin Pop | Use this command on all nodes to install the required packages:: |
449 | 28e15341 | Iustin Pop | |
450 | 28e15341 | Iustin Pop | apt-get install debootstrap dump kpartx |
451 | 28e15341 | Iustin Pop | |
452 | 28e15341 | Iustin Pop | Alternatively, you can create your own OS definitions. See the manpage |
453 | 28e15341 | Iustin Pop | *ganeti-os-interface*. |
454 | 28e15341 | Iustin Pop | |
455 | 28e15341 | Iustin Pop | Initializing the cluster |
456 | 28e15341 | Iustin Pop | ++++++++++++++++++++++++ |
457 | 28e15341 | Iustin Pop | |
458 | 28e15341 | Iustin Pop | **Mandatory** on one node per cluster. |
459 | 28e15341 | Iustin Pop | |
460 | 28e15341 | Iustin Pop | The last step is to initialize the cluster. After you've repeated the |
461 | 28e15341 | Iustin Pop | above process on all of your nodes, choose one as the master, and |
462 | 28e15341 | Iustin Pop | execute:: |
463 | 28e15341 | Iustin Pop | |
464 | 28e15341 | Iustin Pop | gnt-cluster init <CLUSTERNAME> |
465 | 28e15341 | Iustin Pop | |
466 | 28e15341 | Iustin Pop | The *CLUSTERNAME* is a hostname, which must be resolvable (e.g. it |
467 | 28e15341 | Iustin Pop | must exist in DNS or in ``/etc/hosts``) by all the nodes in the |
468 | 28e15341 | Iustin Pop | cluster. You must choose a name different from any of the nodes names |
469 | 28e15341 | Iustin Pop | for a multi-node cluster. In general the best choice is to have a |
470 | 28e15341 | Iustin Pop | unique name for a cluster, even if it consists of only one machine, as |
471 | 28e15341 | Iustin Pop | you will be able to expand it later without any problems. Please note |
472 | 28e15341 | Iustin Pop | that the hostname used for this must resolve to an IP address reserved |
473 | 28e15341 | Iustin Pop | **exclusively** for this purpose, and cannot be the name of the first |
474 | 28e15341 | Iustin Pop | (master) node. |
475 | 28e15341 | Iustin Pop | |
476 | 28e15341 | Iustin Pop | If the bridge name you are using is not ``xen-br0``, use the *-b |
477 | 28e15341 | Iustin Pop | <BRIDGENAME>* option to specify the bridge name. In this case, you |
478 | 28e15341 | Iustin Pop | should also use the *--master-netdev <BRIDGENAME>* option with the |
479 | 28e15341 | Iustin Pop | same BRIDGENAME argument. |
480 | 28e15341 | Iustin Pop | |
481 | 28e15341 | Iustin Pop | You can use a different name than ``xenvg`` for the volume group (but |
482 | 28e15341 | Iustin Pop | note that the name must be identical on all nodes). In this case you |
483 | 28e15341 | Iustin Pop | need to specify it by passing the *-g <VGNAME>* option to |
484 | 28e15341 | Iustin Pop | ``gnt-cluster init``. |
485 | 28e15341 | Iustin Pop | |
486 | 28e15341 | Iustin Pop | To set up the cluster as an HVM cluster, use the |
487 | 28e15341 | Iustin Pop | ``--enabled-hypervisors=xen-hvm`` option to enable the HVM hypervisor |
488 | 28e15341 | Iustin Pop | (you can also add ``,xen-pvm`` to enable the PVM one too). You will |
489 | 28e15341 | Iustin Pop | also need to create the VNC cluster password file |
490 | 28e15341 | Iustin Pop | ``/etc/ganeti/vnc-cluster-password`` which contains one line with the |
491 | 28e15341 | Iustin Pop | default VNC password for the cluster. |
492 | 28e15341 | Iustin Pop | |
493 | 28e15341 | Iustin Pop | To setup the cluster for KVM-only usage (KVM and Xen cannot be mixed), |
494 | 28e15341 | Iustin Pop | pass ``--enabled-hypervisors=kvm`` to the init command. |
495 | 28e15341 | Iustin Pop | |
496 | 28e15341 | Iustin Pop | You can also invoke the command with the ``--help`` option in order to |
497 | 28e15341 | Iustin Pop | see all the possibilities. |
498 | 28e15341 | Iustin Pop | |
499 | 28e15341 | Iustin Pop | Joining the nodes to the cluster |
500 | 28e15341 | Iustin Pop | ++++++++++++++++++++++++++++++++ |
501 | 28e15341 | Iustin Pop | |
502 | 28e15341 | Iustin Pop | **Mandatory** for all the other nodes. |
503 | 28e15341 | Iustin Pop | |
504 | 28e15341 | Iustin Pop | After you have initialized your cluster you need to join the other |
505 | 28e15341 | Iustin Pop | nodes to it. You can do so by executing the following command on the |
506 | 28e15341 | Iustin Pop | master node:: |
507 | 28e15341 | Iustin Pop | |
508 | 28e15341 | Iustin Pop | gnt-node add <NODENAME> |
509 | 28e15341 | Iustin Pop | |
510 | 28e15341 | Iustin Pop | Separate replication network |
511 | 28e15341 | Iustin Pop | ++++++++++++++++++++++++++++ |
512 | 28e15341 | Iustin Pop | |
513 | 28e15341 | Iustin Pop | **Optional** |
514 | 28e15341 | Iustin Pop | |
515 | 28e15341 | Iustin Pop | Ganeti uses DRBD to mirror the disk of the virtual instances between |
516 | 28e15341 | Iustin Pop | nodes. To use a dedicated network interface for this (in order to |
517 | 28e15341 | Iustin Pop | improve performance or to enhance security) you need to configure an |
518 | 28e15341 | Iustin Pop | additional interface for each node. Use the *-s* option with |
519 | 28e15341 | Iustin Pop | ``gnt-cluster init`` and ``gnt-node add`` to specify the IP address of |
520 | 28e15341 | Iustin Pop | this secondary interface to use for each node. Note that if you |
521 | 28e15341 | Iustin Pop | specified this option at cluster setup time, you must afterwards use |
522 | 28e15341 | Iustin Pop | it for every node add operation. |
523 | 28e15341 | Iustin Pop | |
524 | 28e15341 | Iustin Pop | Testing the setup |
525 | 28e15341 | Iustin Pop | +++++++++++++++++ |
526 | 28e15341 | Iustin Pop | |
527 | 28e15341 | Iustin Pop | Execute the ``gnt-node list`` command to see all nodes in the |
528 | 28e15341 | Iustin Pop | cluster:: |
529 | 28e15341 | Iustin Pop | |
530 | 28e15341 | Iustin Pop | # gnt-node list |
531 | 28e15341 | Iustin Pop | Node DTotal DFree MTotal MNode MFree Pinst Sinst |
532 | 28e15341 | Iustin Pop | node1.example.com 197404 197404 2047 1896 125 0 0 |
533 | 28e15341 | Iustin Pop | |
534 | 28e15341 | Iustin Pop | Setting up and managing virtual instances |
535 | 28e15341 | Iustin Pop | ----------------------------------------- |
536 | 28e15341 | Iustin Pop | |
537 | 28e15341 | Iustin Pop | Setting up virtual instances |
538 | 28e15341 | Iustin Pop | ++++++++++++++++++++++++++++ |
539 | 28e15341 | Iustin Pop | |
540 | 28e15341 | Iustin Pop | This step shows how to setup a virtual instance with either |
541 | 28e15341 | Iustin Pop | non-mirrored disks (``plain``) or with network mirrored disks |
542 | 28e15341 | Iustin Pop | (``drbd``). All commands need to be executed on the Ganeti master |
543 | 28e15341 | Iustin Pop | node (the one on which ``gnt-cluster init`` was run). Verify that the |
544 | 28e15341 | Iustin Pop | OS scripts are present on all cluster nodes with ``gnt-os list``. |
545 | 28e15341 | Iustin Pop | |
546 | 28e15341 | Iustin Pop | |
547 | 28e15341 | Iustin Pop | To create a virtual instance, you need a hostname which is resolvable |
548 | 28e15341 | Iustin Pop | (DNS or ``/etc/hosts`` on all nodes). The following command will |
549 | 28e15341 | Iustin Pop | create a non-mirrored instance for you:: |
550 | 28e15341 | Iustin Pop | |
551 | 28e15341 | Iustin Pop | gnt-instance add -t plain -s 1G -n node1 -o debootstrap instance1.example.com |
552 | 28e15341 | Iustin Pop | * creating instance disks... |
553 | 28e15341 | Iustin Pop | adding instance instance1.example.com to cluster config |
554 | 28e15341 | Iustin Pop | - INFO: Waiting for instance instance1.example.com to sync disks. |
555 | 28e15341 | Iustin Pop | - INFO: Instance instance1.example.com's disks are in sync. |
556 | 28e15341 | Iustin Pop | creating os for instance instance1.example.com on node node1.example.com |
557 | 28e15341 | Iustin Pop | * running the instance OS create scripts... |
558 | 28e15341 | Iustin Pop | * starting instance... |
559 | 28e15341 | Iustin Pop | |
560 | 28e15341 | Iustin Pop | The above instance will have no network interface enabled. You can |
561 | 28e15341 | Iustin Pop | access it over the virtual console with ``gnt-instance console |
562 | 28e15341 | Iustin Pop | inst1``. There is no password for root. As this is a Debian instance, |
563 | 28e15341 | Iustin Pop | you can modify the ``/etc/network/interfaces`` file to setup the |
564 | 28e15341 | Iustin Pop | network interface (eth0 is the name of the interface provided to the |
565 | 28e15341 | Iustin Pop | instance). |
566 | 28e15341 | Iustin Pop | |
567 | 28e15341 | Iustin Pop | To create a network mirrored instance, change the argument to the *-t* |
568 | 28e15341 | Iustin Pop | option from ``plain`` to ``drbd`` and specify the node on which the |
569 | 28e15341 | Iustin Pop | mirror should reside with the second value of the *--node* option, |
570 | 28e15341 | Iustin Pop | like this (note that the command output includes timestamps which have |
571 | 28e15341 | Iustin Pop | been removed for clarity):: |
572 | 28e15341 | Iustin Pop | |
573 | 28e15341 | Iustin Pop | # gnt-instance add -t drbd -s 1G -n node1:node2 -o debootstrap instance2 |
574 | 28e15341 | Iustin Pop | * creating instance disks... |
575 | 28e15341 | Iustin Pop | adding instance instance2.example.com to cluster config |
576 | 28e15341 | Iustin Pop | - INFO: Waiting for instance instance2.example.com to sync disks. |
577 | 28e15341 | Iustin Pop | - INFO: - device disk/0: 35.50% done, 11 estimated seconds remaining |
578 | 28e15341 | Iustin Pop | - INFO: - device disk/0: 100.00% done, 0 estimated seconds remaining |
579 | 28e15341 | Iustin Pop | - INFO: Instance instance2.example.com's disks are in sync. |
580 | 28e15341 | Iustin Pop | creating os for instance instance2.example.com on node node1.example.com |
581 | 28e15341 | Iustin Pop | * running the instance OS create scripts... |
582 | 28e15341 | Iustin Pop | * starting instance... |
583 | 28e15341 | Iustin Pop | |
584 | 28e15341 | Iustin Pop | Managing virtual instances |
585 | 28e15341 | Iustin Pop | ++++++++++++++++++++++++++ |
586 | 28e15341 | Iustin Pop | |
587 | 28e15341 | Iustin Pop | All commands need to be executed on the Ganeti master node. |
588 | 28e15341 | Iustin Pop | |
589 | 28e15341 | Iustin Pop | To access the console of an instance, run:: |
590 | 28e15341 | Iustin Pop | |
591 | 28e15341 | Iustin Pop | gnt-instance console INSTANCENAME |
592 | 28e15341 | Iustin Pop | |
593 | 28e15341 | Iustin Pop | To shutdown an instance, run:: |
594 | 28e15341 | Iustin Pop | |
595 | 28e15341 | Iustin Pop | gnt-instance shutdown INSTANCENAME |
596 | 28e15341 | Iustin Pop | |
597 | 28e15341 | Iustin Pop | To startup an instance, run:: |
598 | 28e15341 | Iustin Pop | |
599 | 28e15341 | Iustin Pop | gnt-instance startup INSTANCENAME |
600 | 28e15341 | Iustin Pop | |
601 | 28e15341 | Iustin Pop | To failover an instance to its secondary node (only possible with |
602 | 28e15341 | Iustin Pop | ``drbd`` disk templates), run:: |
603 | 28e15341 | Iustin Pop | |
604 | 28e15341 | Iustin Pop | gnt-instance failover INSTANCENAME |
605 | 28e15341 | Iustin Pop | |
606 | 28e15341 | Iustin Pop | For more instance and cluster administration details, see the |
607 | 28e15341 | Iustin Pop | *Ganeti administrator's guide*. |