root / doc / install.rst @ 5d94c034
History | View | Annotate | Download (29 kB)
1 | 28e15341 | Iustin Pop | Ganeti installation tutorial |
---|---|---|---|
2 | 28e15341 | Iustin Pop | ============================ |
3 | 28e15341 | Iustin Pop | |
4 | fd07c6b3 | Iustin Pop | Documents Ganeti version |version| |
5 | 28e15341 | Iustin Pop | |
6 | 28e15341 | Iustin Pop | .. contents:: |
7 | 28e15341 | Iustin Pop | |
8 | f6d62af4 | Iustin Pop | .. highlight:: shell-example |
9 | c71a1a3d | Iustin Pop | |
10 | 28e15341 | Iustin Pop | Introduction |
11 | 28e15341 | Iustin Pop | ------------ |
12 | 28e15341 | Iustin Pop | |
13 | 28e15341 | Iustin Pop | Ganeti is a cluster virtualization management system based on Xen or |
14 | c71a1a3d | Iustin Pop | KVM. This document explains how to bootstrap a Ganeti node (Xen *dom0*, |
15 | c71a1a3d | Iustin Pop | the host Linux system for KVM), create a running cluster and install |
16 | c71a1a3d | Iustin Pop | virtual instances (Xen *domUs*, KVM guests). You need to repeat most of |
17 | c71a1a3d | Iustin Pop | the steps in this document for every node you want to install, but of |
18 | c71a1a3d | Iustin Pop | course we recommend creating some semi-automatic procedure if you plan |
19 | c71a1a3d | Iustin Pop | to deploy Ganeti on a medium/large scale. |
20 | 28e15341 | Iustin Pop | |
21 | 28e15341 | Iustin Pop | A basic Ganeti terminology glossary is provided in the introductory |
22 | c71a1a3d | Iustin Pop | section of the :doc:`admin`. Please refer to that document if you are |
23 | c71a1a3d | Iustin Pop | uncertain about the terms we are using. |
24 | 28e15341 | Iustin Pop | |
25 | c71a1a3d | Iustin Pop | Ganeti has been developed for Linux and should be distribution-agnostic. |
26 | f6d62af4 | Iustin Pop | This documentation will use Debian Squeeze as an example system but the |
27 | c71a1a3d | Iustin Pop | examples can be translated to any other distribution. You are expected |
28 | c71a1a3d | Iustin Pop | to be familiar with your distribution, its package management system, |
29 | c71a1a3d | Iustin Pop | and Xen or KVM before trying to use Ganeti. |
30 | 28e15341 | Iustin Pop | |
31 | 28e15341 | Iustin Pop | This document is divided into two main sections: |
32 | 28e15341 | Iustin Pop | |
33 | 28e15341 | Iustin Pop | - Installation of the base system and base components |
34 | 28e15341 | Iustin Pop | |
35 | 28e15341 | Iustin Pop | - Configuration of the environment for Ganeti |
36 | 28e15341 | Iustin Pop | |
37 | 28e15341 | Iustin Pop | Each of these is divided into sub-sections. While a full Ganeti system |
38 | c71a1a3d | Iustin Pop | will need all of the steps specified, some are not strictly required for |
39 | c71a1a3d | Iustin Pop | every environment. Which ones they are, and why, is specified in the |
40 | c71a1a3d | Iustin Pop | corresponding sections. |
41 | 28e15341 | Iustin Pop | |
42 | 28e15341 | Iustin Pop | Installing the base system and base components |
43 | 28e15341 | Iustin Pop | ---------------------------------------------- |
44 | 28e15341 | Iustin Pop | |
45 | 28e15341 | Iustin Pop | Hardware requirements |
46 | 28e15341 | Iustin Pop | +++++++++++++++++++++ |
47 | 28e15341 | Iustin Pop | |
48 | c71a1a3d | Iustin Pop | Any system supported by your Linux distribution is fine. 64-bit systems |
49 | c71a1a3d | Iustin Pop | are better as they can support more memory. |
50 | 28e15341 | Iustin Pop | |
51 | c71a1a3d | Iustin Pop | Any disk drive recognized by Linux (``IDE``/``SCSI``/``SATA``/etc.) is |
52 | c71a1a3d | Iustin Pop | supported in Ganeti. Note that no shared storage (e.g. ``SAN``) is |
53 | c71a1a3d | Iustin Pop | needed to get high-availability features (but of course, one can be used |
54 | f6d62af4 | Iustin Pop | to store the images). Whilte it is highly recommended to use more than |
55 | f6d62af4 | Iustin Pop | one disk drive in order to improve speed, Ganeti also works with one |
56 | f6d62af4 | Iustin Pop | disk per machine. |
57 | 28e15341 | Iustin Pop | |
58 | 28e15341 | Iustin Pop | Installing the base system |
59 | 28e15341 | Iustin Pop | ++++++++++++++++++++++++++ |
60 | 28e15341 | Iustin Pop | |
61 | 28e15341 | Iustin Pop | **Mandatory** on all nodes. |
62 | 28e15341 | Iustin Pop | |
63 | 28e15341 | Iustin Pop | It is advised to start with a clean, minimal install of the operating |
64 | c71a1a3d | Iustin Pop | system. The only requirement you need to be aware of at this stage is to |
65 | c71a1a3d | Iustin Pop | partition leaving enough space for a big (**minimum** 20GiB) LVM volume |
66 | c71a1a3d | Iustin Pop | group which will then host your instance filesystems, if you want to use |
67 | c71a1a3d | Iustin Pop | all Ganeti features. The volume group name Ganeti uses (by default) is |
68 | c71a1a3d | Iustin Pop | ``xenvg``. |
69 | 28e15341 | Iustin Pop | |
70 | c71a1a3d | Iustin Pop | You can also use file-based storage only, without LVM, but this setup is |
71 | c71a1a3d | Iustin Pop | not detailed in this document. |
72 | 28e15341 | Iustin Pop | |
73 | 7ed400f0 | Stratos Psomadakis | If you choose to use RBD-based instances, there's no need for LVM |
74 | f6d62af4 | Iustin Pop | provisioning. However, this feature is experimental, and is not yet |
75 | 7ed400f0 | Stratos Psomadakis | recommended for production clusters. |
76 | 7ed400f0 | Stratos Psomadakis | |
77 | 28e15341 | Iustin Pop | While you can use an existing system, please note that the Ganeti |
78 | 28e15341 | Iustin Pop | installation is intrusive in terms of changes to the system |
79 | 28e15341 | Iustin Pop | configuration, and it's best to use a newly-installed system without |
80 | 28e15341 | Iustin Pop | important data on it. |
81 | 28e15341 | Iustin Pop | |
82 | 28e15341 | Iustin Pop | Also, for best results, it's advised that the nodes have as much as |
83 | 28e15341 | Iustin Pop | possible the same hardware and software configuration. This will make |
84 | 28e15341 | Iustin Pop | administration much easier. |
85 | 28e15341 | Iustin Pop | |
86 | 28e15341 | Iustin Pop | Hostname issues |
87 | 28e15341 | Iustin Pop | ~~~~~~~~~~~~~~~ |
88 | 28e15341 | Iustin Pop | |
89 | 28e15341 | Iustin Pop | Note that Ganeti requires the hostnames of the systems (i.e. what the |
90 | 28e15341 | Iustin Pop | ``hostname`` command outputs to be a fully-qualified name, not a short |
91 | 28e15341 | Iustin Pop | name. In other words, you should use *node1.example.com* as a hostname |
92 | 28e15341 | Iustin Pop | and not just *node1*. |
93 | 28e15341 | Iustin Pop | |
94 | 28e15341 | Iustin Pop | .. admonition:: Debian |
95 | 28e15341 | Iustin Pop | |
96 | f6d62af4 | Iustin Pop | Debian usually configures the hostname differently than you need it |
97 | f6d62af4 | Iustin Pop | for Ganeti. For example, this is what it puts in ``/etc/hosts`` in |
98 | f6d62af4 | Iustin Pop | certain situations:: |
99 | 28e15341 | Iustin Pop | |
100 | 28e15341 | Iustin Pop | 127.0.0.1 localhost |
101 | 28e15341 | Iustin Pop | 127.0.1.1 node1.example.com node1 |
102 | 28e15341 | Iustin Pop | |
103 | 28e15341 | Iustin Pop | but for Ganeti you need to have:: |
104 | 28e15341 | Iustin Pop | |
105 | 28e15341 | Iustin Pop | 127.0.0.1 localhost |
106 | f6d62af4 | Iustin Pop | 192.0.2.1 node1.example.com node1 |
107 | 28e15341 | Iustin Pop | |
108 | 926feaf1 | Manuel Franceschini | replacing ``192.0.2.1`` with your node's address. Also, the file |
109 | 28e15341 | Iustin Pop | ``/etc/hostname`` which configures the hostname of the system |
110 | 28e15341 | Iustin Pop | should contain ``node1.example.com`` and not just ``node1`` (you |
111 | 28e15341 | Iustin Pop | need to run the command ``/etc/init.d/hostname.sh start`` after |
112 | 28e15341 | Iustin Pop | changing the file). |
113 | 28e15341 | Iustin Pop | |
114 | 1232284c | Guido Trotter | .. admonition:: Why a fully qualified host name |
115 | 1232284c | Guido Trotter | |
116 | 7faf5110 | Michael Hanselmann | Although most distributions use only the short name in the |
117 | 7faf5110 | Michael Hanselmann | /etc/hostname file, we still think Ganeti nodes should use the full |
118 | 7faf5110 | Michael Hanselmann | name. The reason for this is that calling 'hostname --fqdn' requires |
119 | 7faf5110 | Michael Hanselmann | the resolver library to work and is a 'guess' via heuristics at what |
120 | 7faf5110 | Michael Hanselmann | is your domain name. Since Ganeti can be used among other things to |
121 | 7faf5110 | Michael Hanselmann | host DNS servers, we don't want to depend on them as much as |
122 | 7faf5110 | Michael Hanselmann | possible, and we'd rather have the uname() syscall return the full |
123 | 7faf5110 | Michael Hanselmann | node name. |
124 | 7faf5110 | Michael Hanselmann | |
125 | 7faf5110 | Michael Hanselmann | We haven't ever found any breakage in using a full hostname on a |
126 | 7faf5110 | Michael Hanselmann | Linux system, and anyway we recommend to have only a minimal |
127 | 7faf5110 | Michael Hanselmann | installation on Ganeti nodes, and to use instances (or other |
128 | 7faf5110 | Michael Hanselmann | dedicated machines) to run the rest of your network services. By |
129 | 7faf5110 | Michael Hanselmann | doing this you can change the /etc/hostname file to contain an FQDN |
130 | 7faf5110 | Michael Hanselmann | without the fear of breaking anything unrelated. |
131 | 1232284c | Guido Trotter | |
132 | 1232284c | Guido Trotter | |
133 | 756d5ec3 | Guido Trotter | Installing The Hypervisor |
134 | 756d5ec3 | Guido Trotter | +++++++++++++++++++++++++ |
135 | 28e15341 | Iustin Pop | |
136 | 28e15341 | Iustin Pop | **Mandatory** on all nodes. |
137 | 28e15341 | Iustin Pop | |
138 | 756d5ec3 | Guido Trotter | While Ganeti is developed with the ability to modularly run on different |
139 | 7faf5110 | Michael Hanselmann | virtualization environments in mind the only two currently useable on a |
140 | f6d62af4 | Iustin Pop | live system are Xen and KVM. Supported Xen versions are: 3.0.3 and later |
141 | f6d62af4 | Iustin Pop | 3.x versions, and 4.x (tested up to 4.1). Supported KVM versions are 72 |
142 | f6d62af4 | Iustin Pop | and above. |
143 | 28e15341 | Iustin Pop | |
144 | c71a1a3d | Iustin Pop | Please follow your distribution's recommended way to install and set up |
145 | c71a1a3d | Iustin Pop | Xen, or install Xen from the upstream source, if you wish, following |
146 | c71a1a3d | Iustin Pop | their manual. For KVM, make sure you have a KVM-enabled kernel and the |
147 | c71a1a3d | Iustin Pop | KVM tools. |
148 | 28e15341 | Iustin Pop | |
149 | 756d5ec3 | Guido Trotter | After installing Xen, you need to reboot into your new system. On some |
150 | 7faf5110 | Michael Hanselmann | distributions this might involve configuring GRUB appropriately, whereas |
151 | 7faf5110 | Michael Hanselmann | others will configure it automatically when you install the respective |
152 | 7faf5110 | Michael Hanselmann | kernels. For KVM no reboot should be necessary. |
153 | 28e15341 | Iustin Pop | |
154 | 756d5ec3 | Guido Trotter | .. admonition:: Xen on Debian |
155 | 28e15341 | Iustin Pop | |
156 | f6d62af4 | Iustin Pop | Under Debian you can install the relevant ``xen-linux-system`` |
157 | c71a1a3d | Iustin Pop | package, which will pull in both the hypervisor and the relevant |
158 | f6d62af4 | Iustin Pop | kernel. Also, if you are installing a 32-bit system, you should |
159 | c71a1a3d | Iustin Pop | install the ``libc6-xen`` package (run ``apt-get install |
160 | c71a1a3d | Iustin Pop | libc6-xen``). |
161 | 28e15341 | Iustin Pop | |
162 | 28e15341 | Iustin Pop | Xen settings |
163 | 28e15341 | Iustin Pop | ~~~~~~~~~~~~ |
164 | 28e15341 | Iustin Pop | |
165 | 28e15341 | Iustin Pop | It's recommended that dom0 is restricted to a low amount of memory |
166 | c71a1a3d | Iustin Pop | (512MiB or 1GiB is reasonable) and that memory ballooning is disabled in |
167 | c71a1a3d | Iustin Pop | the file ``/etc/xen/xend-config.sxp`` by setting the value |
168 | c71a1a3d | Iustin Pop | ``dom0-min-mem`` to 0, like this:: |
169 | 28e15341 | Iustin Pop | |
170 | 28e15341 | Iustin Pop | (dom0-min-mem 0) |
171 | 28e15341 | Iustin Pop | |
172 | 28e15341 | Iustin Pop | For optimum performance when running both CPU and I/O intensive |
173 | c71a1a3d | Iustin Pop | instances, it's also recommended that the dom0 is restricted to one CPU |
174 | f6d62af4 | Iustin Pop | only, for example by booting with the kernel parameter ``maxcpus=1``. |
175 | 28e15341 | Iustin Pop | |
176 | 28e15341 | Iustin Pop | It is recommended that you disable xen's automatic save of virtual |
177 | 28e15341 | Iustin Pop | machines at system shutdown and subsequent restore of them at reboot. |
178 | 28e15341 | Iustin Pop | To obtain this make sure the variable ``XENDOMAINS_SAVE`` in the file |
179 | 28e15341 | Iustin Pop | ``/etc/default/xendomains`` is set to an empty value. |
180 | 28e15341 | Iustin Pop | |
181 | aeaa2ea2 | Guido Trotter | If you want to use live migration make sure you have, in the xen config |
182 | aeaa2ea2 | Guido Trotter | file, something that allows the nodes to migrate instances between each |
183 | f6d62af4 | Iustin Pop | other. For example: |
184 | f6d62af4 | Iustin Pop | |
185 | f6d62af4 | Iustin Pop | .. code-block:: text |
186 | 8ab90d80 | Guido Trotter | |
187 | 8ab90d80 | Guido Trotter | (xend-relocation-server yes) |
188 | 8ab90d80 | Guido Trotter | (xend-relocation-port 8002) |
189 | 8ab90d80 | Guido Trotter | (xend-relocation-address '') |
190 | 926feaf1 | Manuel Franceschini | (xend-relocation-hosts-allow '^192\\.0\\.2\\.[0-9]+$') |
191 | 8ab90d80 | Guido Trotter | |
192 | e8a3bf18 | Iustin Pop | |
193 | 84d7362b | Andrea Spadaccini | The second line assumes that the hypervisor parameter |
194 | e8a3bf18 | Iustin Pop | ``migration_port`` is set 8002, otherwise modify it to match. The last |
195 | e8a3bf18 | Iustin Pop | line assumes that all your nodes have secondary IPs in the |
196 | 926feaf1 | Manuel Franceschini | 192.0.2.0/24 network, adjust it accordingly to your setup. |
197 | 8ab90d80 | Guido Trotter | |
198 | 28e15341 | Iustin Pop | .. admonition:: Debian |
199 | 28e15341 | Iustin Pop | |
200 | 28e15341 | Iustin Pop | Besides the ballooning change which you need to set in |
201 | 28e15341 | Iustin Pop | ``/etc/xen/xend-config.sxp``, you need to set the memory and nosmp |
202 | 28e15341 | Iustin Pop | parameters in the file ``/boot/grub/menu.lst``. You need to modify |
203 | f6d62af4 | Iustin Pop | the variable ``xenhopt`` to add ``dom0_mem=1024M`` like this: |
204 | f6d62af4 | Iustin Pop | |
205 | f6d62af4 | Iustin Pop | .. code-block:: text |
206 | 28e15341 | Iustin Pop | |
207 | 28e15341 | Iustin Pop | ## Xen hypervisor options to use with the default Xen boot option |
208 | 28e15341 | Iustin Pop | # xenhopt=dom0_mem=1024M |
209 | 28e15341 | Iustin Pop | |
210 | f6d62af4 | Iustin Pop | and the ``xenkopt`` needs to include the ``maxcpus`` option like |
211 | f6d62af4 | Iustin Pop | this: |
212 | f6d62af4 | Iustin Pop | |
213 | f6d62af4 | Iustin Pop | .. code-block:: text |
214 | 28e15341 | Iustin Pop | |
215 | 28e15341 | Iustin Pop | ## Xen Linux kernel options to use with the default Xen boot option |
216 | f6d62af4 | Iustin Pop | # xenkopt=maxcpus=1 |
217 | 28e15341 | Iustin Pop | |
218 | 28e15341 | Iustin Pop | Any existing parameters can be left in place: it's ok to have |
219 | f6d62af4 | Iustin Pop | ``xenkopt=console=tty0 maxcpus=1``, for example. After modifying the |
220 | 28e15341 | Iustin Pop | files, you need to run:: |
221 | 28e15341 | Iustin Pop | |
222 | f6d62af4 | Iustin Pop | $ /sbin/update-grub |
223 | 28e15341 | Iustin Pop | |
224 | c71a1a3d | Iustin Pop | If you want to run HVM instances too with Ganeti and want VNC access to |
225 | c71a1a3d | Iustin Pop | the console of your instances, set the following two entries in |
226 | f6d62af4 | Iustin Pop | ``/etc/xen/xend-config.sxp``: |
227 | f6d62af4 | Iustin Pop | |
228 | f6d62af4 | Iustin Pop | .. code-block:: text |
229 | 28e15341 | Iustin Pop | |
230 | 28e15341 | Iustin Pop | (vnc-listen '0.0.0.0') (vncpasswd '') |
231 | 28e15341 | Iustin Pop | |
232 | 28e15341 | Iustin Pop | You need to restart the Xen daemon for these settings to take effect:: |
233 | 28e15341 | Iustin Pop | |
234 | f6d62af4 | Iustin Pop | $ /etc/init.d/xend restart |
235 | 28e15341 | Iustin Pop | |
236 | 28e15341 | Iustin Pop | Selecting the instance kernel |
237 | 28e15341 | Iustin Pop | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
238 | 28e15341 | Iustin Pop | |
239 | 28e15341 | Iustin Pop | After you have installed Xen, you need to tell Ganeti exactly what |
240 | c71a1a3d | Iustin Pop | kernel to use for the instances it will create. This is done by creating |
241 | f6d62af4 | Iustin Pop | a symlink from your actual kernel to ``/boot/vmlinuz-3-xenU``, and one |
242 | f6d62af4 | Iustin Pop | from your initrd to ``/boot/initrd-3-xenU`` [#defkernel]_. Note that |
243 | c71a1a3d | Iustin Pop | if you don't use an initrd for the domU kernel, you don't need to create |
244 | c71a1a3d | Iustin Pop | the initrd symlink. |
245 | 28e15341 | Iustin Pop | |
246 | 28e15341 | Iustin Pop | .. admonition:: Debian |
247 | 28e15341 | Iustin Pop | |
248 | 28e15341 | Iustin Pop | After installation of the ``xen-linux-system`` package, you need to |
249 | 28e15341 | Iustin Pop | run (replace the exact version number with the one you have):: |
250 | 28e15341 | Iustin Pop | |
251 | f6d62af4 | Iustin Pop | $ cd /boot |
252 | f6d62af4 | Iustin Pop | $ ln -s vmlinuz-%2.6.26-1%-xen-amd64 vmlinuz-3-xenU |
253 | f6d62af4 | Iustin Pop | $ ln -s initrd.img-%2.6.26-1%-xen-amd64 initrd-3-xenU |
254 | f6d62af4 | Iustin Pop | |
255 | f6d62af4 | Iustin Pop | By default, the initrd doesn't contain the Xen block drivers needed |
256 | f6d62af4 | Iustin Pop | to mount the root device, so it is recommended to update the initrd |
257 | f6d62af4 | Iustin Pop | by following these two steps: |
258 | f6d62af4 | Iustin Pop | |
259 | f6d62af4 | Iustin Pop | - edit ``/etc/initramfs-tools/modules`` and add ``xen_blkfront`` |
260 | f6d62af4 | Iustin Pop | - run ``update-initramfs -u`` |
261 | 28e15341 | Iustin Pop | |
262 | 28e15341 | Iustin Pop | Installing DRBD |
263 | 28e15341 | Iustin Pop | +++++++++++++++ |
264 | 28e15341 | Iustin Pop | |
265 | c71a1a3d | Iustin Pop | Recommended on all nodes: DRBD_ is required if you want to use the high |
266 | c71a1a3d | Iustin Pop | availability (HA) features of Ganeti, but optional if you don't require |
267 | c71a1a3d | Iustin Pop | them or only run Ganeti on single-node clusters. You can upgrade a |
268 | f6d62af4 | Iustin Pop | non-HA cluster to an HA one later, but you might need to convert all |
269 | f6d62af4 | Iustin Pop | your instances to DRBD to take advantage of the new features. |
270 | 28e15341 | Iustin Pop | |
271 | 28e15341 | Iustin Pop | .. _DRBD: http://www.drbd.org/ |
272 | 28e15341 | Iustin Pop | |
273 | f6d62af4 | Iustin Pop | Supported DRBD versions: 8.0-8.3. It's recommended to have at least |
274 | f6d62af4 | Iustin Pop | version 8.0.12. Note that for version 8.2 and newer it is needed to pass |
275 | f6d62af4 | Iustin Pop | the ``usermode_helper=/bin/true`` parameter to the module, either by |
276 | c71a1a3d | Iustin Pop | configuring ``/etc/modules`` or when inserting it manually. |
277 | 28e15341 | Iustin Pop | |
278 | 28e15341 | Iustin Pop | Now the bad news: unless your distribution already provides it |
279 | c71a1a3d | Iustin Pop | installing DRBD might involve recompiling your kernel or anyway fiddling |
280 | c71a1a3d | Iustin Pop | with it. Hopefully at least the Xen-ified kernel source to start from |
281 | c71a1a3d | Iustin Pop | will be provided (if you intend to use Xen). |
282 | 28e15341 | Iustin Pop | |
283 | 28e15341 | Iustin Pop | The good news is that you don't need to configure DRBD at all. Ganeti |
284 | c71a1a3d | Iustin Pop | will do it for you for every instance you set up. If you have the DRBD |
285 | c71a1a3d | Iustin Pop | utils installed and the module in your kernel you're fine. Please check |
286 | c71a1a3d | Iustin Pop | that your system is configured to load the module at every boot, and |
287 | c71a1a3d | Iustin Pop | that it passes the following option to the module: |
288 | c71a1a3d | Iustin Pop | ``minor_count=NUMBER``. We recommend that you use 128 as the value of |
289 | c71a1a3d | Iustin Pop | the minor_count - this will allow you to use up to 64 instances in total |
290 | c71a1a3d | Iustin Pop | per node (both primary and secondary, when using only one disk per |
291 | c71a1a3d | Iustin Pop | instance). You can increase the number up to 255 if you need more |
292 | c71a1a3d | Iustin Pop | instances on a node. |
293 | c71a1a3d | Iustin Pop | |
294 | 28e15341 | Iustin Pop | |
295 | 28e15341 | Iustin Pop | .. admonition:: Debian |
296 | 28e15341 | Iustin Pop | |
297 | c71a1a3d | Iustin Pop | On Debian, you can just install (build) the DRBD module with the |
298 | c71a1a3d | Iustin Pop | following commands, making sure you are running the target (Xen or |
299 | c71a1a3d | Iustin Pop | KVM) kernel:: |
300 | 28e15341 | Iustin Pop | |
301 | f6d62af4 | Iustin Pop | $ apt-get install drbd8-source drbd8-utils |
302 | f6d62af4 | Iustin Pop | $ m-a update |
303 | f6d62af4 | Iustin Pop | $ m-a a-i drbd8 |
304 | db0e97f0 | Guido Trotter | |
305 | db0e97f0 | Guido Trotter | Or on newer versions, if the kernel already has modules: |
306 | db0e97f0 | Guido Trotter | |
307 | db0e97f0 | Guido Trotter | $ apt-get install drbd8-utils |
308 | db0e97f0 | Guido Trotter | |
309 | db0e97f0 | Guido Trotter | Then to configure it for Ganeti:: |
310 | db0e97f0 | Guido Trotter | |
311 | f6d62af4 | Iustin Pop | $ echo drbd minor_count=128 usermode_helper=/bin/true >> /etc/modules |
312 | f6d62af4 | Iustin Pop | $ depmod -a |
313 | f6d62af4 | Iustin Pop | $ modprobe drbd minor_count=128 usermode_helper=/bin/true |
314 | 28e15341 | Iustin Pop | |
315 | c71a1a3d | Iustin Pop | It is also recommended that you comment out the default resources in |
316 | c71a1a3d | Iustin Pop | the ``/etc/drbd.conf`` file, so that the init script doesn't try to |
317 | c71a1a3d | Iustin Pop | configure any drbd devices. You can do this by prefixing all |
318 | f6d62af4 | Iustin Pop | *resource* lines in the file with the keyword *skip*, like this: |
319 | f6d62af4 | Iustin Pop | |
320 | f6d62af4 | Iustin Pop | .. code-block:: text |
321 | 28e15341 | Iustin Pop | |
322 | 92c1ea55 | Iustin Pop | skip { |
323 | 92c1ea55 | Iustin Pop | resource r0 { |
324 | 92c1ea55 | Iustin Pop | ... |
325 | 92c1ea55 | Iustin Pop | } |
326 | 28e15341 | Iustin Pop | } |
327 | 28e15341 | Iustin Pop | |
328 | 92c1ea55 | Iustin Pop | skip { |
329 | 92c1ea55 | Iustin Pop | resource "r1" { |
330 | 92c1ea55 | Iustin Pop | ... |
331 | 92c1ea55 | Iustin Pop | } |
332 | 28e15341 | Iustin Pop | } |
333 | 28e15341 | Iustin Pop | |
334 | 7ed400f0 | Stratos Psomadakis | Installing RBD |
335 | f6d62af4 | Iustin Pop | ++++++++++++++ |
336 | 7ed400f0 | Stratos Psomadakis | |
337 | 7ed400f0 | Stratos Psomadakis | Recommended on all nodes: RBD_ is required if you want to create |
338 | 7ed400f0 | Stratos Psomadakis | instances with RBD disks residing inside a RADOS cluster (make use of |
339 | 7ed400f0 | Stratos Psomadakis | the rbd disk template). RBD-based instances can failover or migrate to |
340 | 7ed400f0 | Stratos Psomadakis | any other node in the ganeti cluster, enabling you to exploit of all |
341 | 7ed400f0 | Stratos Psomadakis | Ganeti's high availabilily (HA) features. |
342 | 7ed400f0 | Stratos Psomadakis | |
343 | 7ed400f0 | Stratos Psomadakis | .. attention:: |
344 | 7ed400f0 | Stratos Psomadakis | Be careful though: rbd is still experimental! For now it is |
345 | 7ed400f0 | Stratos Psomadakis | recommended only for testing purposes. No sensitive data should be |
346 | 7ed400f0 | Stratos Psomadakis | stored there. |
347 | 7ed400f0 | Stratos Psomadakis | |
348 | 7ed400f0 | Stratos Psomadakis | .. _RBD: http://ceph.newdream.net/ |
349 | 7ed400f0 | Stratos Psomadakis | |
350 | 7ed400f0 | Stratos Psomadakis | You will need the ``rbd`` and ``libceph`` kernel modules, the RBD/Ceph |
351 | 7ed400f0 | Stratos Psomadakis | userspace utils (ceph-common Debian package) and an appropriate |
352 | 7ed400f0 | Stratos Psomadakis | Ceph/RADOS configuration file on every VM-capable node. |
353 | 7ed400f0 | Stratos Psomadakis | |
354 | 7ed400f0 | Stratos Psomadakis | You will also need a working RADOS Cluster accessible by the above |
355 | 7ed400f0 | Stratos Psomadakis | nodes. |
356 | 7ed400f0 | Stratos Psomadakis | |
357 | 7ed400f0 | Stratos Psomadakis | RADOS Cluster |
358 | 7ed400f0 | Stratos Psomadakis | ~~~~~~~~~~~~~ |
359 | 7ed400f0 | Stratos Psomadakis | |
360 | 7ed400f0 | Stratos Psomadakis | You will need a working RADOS Cluster accesible by all VM-capable nodes |
361 | 7ed400f0 | Stratos Psomadakis | to use the RBD template. For more information on setting up a RADOS |
362 | 7ed400f0 | Stratos Psomadakis | Cluster, refer to the `official docs <http://ceph.newdream.net/>`_. |
363 | 7ed400f0 | Stratos Psomadakis | |
364 | 7ed400f0 | Stratos Psomadakis | If you want to use a pool for storing RBD disk images other than the |
365 | 7ed400f0 | Stratos Psomadakis | default (``rbd``), you should first create the pool in the RADOS |
366 | 7ed400f0 | Stratos Psomadakis | Cluster, and then set the corresponding rbd disk parameter named |
367 | 7ed400f0 | Stratos Psomadakis | ``pool``. |
368 | 7ed400f0 | Stratos Psomadakis | |
369 | 7ed400f0 | Stratos Psomadakis | Kernel Modules |
370 | 7ed400f0 | Stratos Psomadakis | ~~~~~~~~~~~~~~ |
371 | 7ed400f0 | Stratos Psomadakis | |
372 | 7ed400f0 | Stratos Psomadakis | Unless your distribution already provides it, you might need to compile |
373 | 7ed400f0 | Stratos Psomadakis | the ``rbd`` and ``libceph`` modules from source. You will need Linux |
374 | 7ed400f0 | Stratos Psomadakis | Kernel 3.2 or above for the kernel modules. Alternatively you will have |
375 | 7ed400f0 | Stratos Psomadakis | to build them as external modules (from Linux Kernel source 3.2 or |
376 | 7ed400f0 | Stratos Psomadakis | above), if you want to run a less recent kernel, or your kernel doesn't |
377 | 7ed400f0 | Stratos Psomadakis | include them. |
378 | 7ed400f0 | Stratos Psomadakis | |
379 | 7ed400f0 | Stratos Psomadakis | Userspace Utils |
380 | 7ed400f0 | Stratos Psomadakis | ~~~~~~~~~~~~~~~ |
381 | 7ed400f0 | Stratos Psomadakis | |
382 | 7ed400f0 | Stratos Psomadakis | The RBD template has been tested with ``ceph-common`` v0.38 and |
383 | 7ed400f0 | Stratos Psomadakis | above. We recommend using the latest version of ``ceph-common``. |
384 | 7ed400f0 | Stratos Psomadakis | |
385 | 7ed400f0 | Stratos Psomadakis | .. admonition:: Debian |
386 | 7ed400f0 | Stratos Psomadakis | |
387 | 7ed400f0 | Stratos Psomadakis | On Debian, you can just install the RBD/Ceph userspace utils with |
388 | 7ed400f0 | Stratos Psomadakis | the following command:: |
389 | 7ed400f0 | Stratos Psomadakis | |
390 | f6d62af4 | Iustin Pop | $ apt-get install ceph-common |
391 | 7ed400f0 | Stratos Psomadakis | |
392 | 7ed400f0 | Stratos Psomadakis | Configuration file |
393 | 7ed400f0 | Stratos Psomadakis | ~~~~~~~~~~~~~~~~~~ |
394 | 7ed400f0 | Stratos Psomadakis | |
395 | 7ed400f0 | Stratos Psomadakis | You should also provide an appropriate configuration file |
396 | 7ed400f0 | Stratos Psomadakis | (``ceph.conf``) in ``/etc/ceph``. For the rbd userspace utils, you'll |
397 | 7ed400f0 | Stratos Psomadakis | only need to specify the IP addresses of the RADOS Cluster monitors. |
398 | 7ed400f0 | Stratos Psomadakis | |
399 | 7ed400f0 | Stratos Psomadakis | .. admonition:: ceph.conf |
400 | 7ed400f0 | Stratos Psomadakis | |
401 | f6d62af4 | Iustin Pop | Sample configuration file: |
402 | f6d62af4 | Iustin Pop | |
403 | f6d62af4 | Iustin Pop | .. code-block:: text |
404 | 7ed400f0 | Stratos Psomadakis | |
405 | 7ed400f0 | Stratos Psomadakis | [mon.a] |
406 | 7ed400f0 | Stratos Psomadakis | host = example_monitor_host1 |
407 | 7ed400f0 | Stratos Psomadakis | mon addr = 1.2.3.4:6789 |
408 | 7ed400f0 | Stratos Psomadakis | [mon.b] |
409 | 7ed400f0 | Stratos Psomadakis | host = example_monitor_host2 |
410 | 7ed400f0 | Stratos Psomadakis | mon addr = 1.2.3.5:6789 |
411 | 7ed400f0 | Stratos Psomadakis | [mon.c] |
412 | 7ed400f0 | Stratos Psomadakis | host = example_monitor_host3 |
413 | 7ed400f0 | Stratos Psomadakis | mon addr = 1.2.3.6:6789 |
414 | 7ed400f0 | Stratos Psomadakis | |
415 | 7ed400f0 | Stratos Psomadakis | For more information, please see the `Ceph Docs |
416 | 7ed400f0 | Stratos Psomadakis | <http://ceph.newdream.net/docs/latest/>`_ |
417 | 7ed400f0 | Stratos Psomadakis | |
418 | 28e15341 | Iustin Pop | Other required software |
419 | 28e15341 | Iustin Pop | +++++++++++++++++++++++ |
420 | 28e15341 | Iustin Pop | |
421 | cbf3d64b | Michael Hanselmann | See :doc:`install-quick`. |
422 | 28e15341 | Iustin Pop | |
423 | 28e15341 | Iustin Pop | Setting up the environment for Ganeti |
424 | 28e15341 | Iustin Pop | ------------------------------------- |
425 | 28e15341 | Iustin Pop | |
426 | 28e15341 | Iustin Pop | Configuring the network |
427 | 28e15341 | Iustin Pop | +++++++++++++++++++++++ |
428 | 28e15341 | Iustin Pop | |
429 | 28e15341 | Iustin Pop | **Mandatory** on all nodes. |
430 | 28e15341 | Iustin Pop | |
431 | 57fb6fcb | Guido Trotter | You can run Ganeti either in "bridged mode", "routed mode" or |
432 | 57fb6fcb | Guido Trotter | "openvswitch mode". In bridged mode, the default, the instances network |
433 | 57fb6fcb | Guido Trotter | interfaces will be attached to a software bridge running in dom0. Xen by |
434 | 57fb6fcb | Guido Trotter | default creates such a bridge at startup, but your distribution might |
435 | 57fb6fcb | Guido Trotter | have a different way to do things, and you'll definitely need to |
436 | 57fb6fcb | Guido Trotter | manually set it up under KVM. |
437 | 28e15341 | Iustin Pop | |
438 | c71a1a3d | Iustin Pop | Beware that the default name Ganeti uses is ``xen-br0`` (which was used |
439 | e721c742 | Guido Trotter | in Xen 2.0) while Xen 3.0 uses ``xenbr0`` by default. See the |
440 | e721c742 | Guido Trotter | `Initializing the cluster`_ section to learn how to choose a different |
441 | e721c742 | Guido Trotter | bridge, or not to use one at all and use "routed mode". |
442 | 28e15341 | Iustin Pop | |
443 | e721c742 | Guido Trotter | In order to use "routed mode" under Xen, you'll need to change the |
444 | 7faf5110 | Michael Hanselmann | relevant parameters in the Xen config file. Under KVM instead, no config |
445 | 7faf5110 | Michael Hanselmann | change is necessary, but you still need to set up your network |
446 | 7faf5110 | Michael Hanselmann | interfaces correctly. |
447 | 9f83899a | Guido Trotter | |
448 | 9f83899a | Guido Trotter | By default, under KVM, the "link" parameter you specify per-nic will |
449 | 7faf5110 | Michael Hanselmann | represent, if non-empty, a different routing table name or number to use |
450 | f6d62af4 | Iustin Pop | for your instances. This allows isolation between different instance |
451 | 7faf5110 | Michael Hanselmann | groups, and different routing policies between node traffic and instance |
452 | 7faf5110 | Michael Hanselmann | traffic. |
453 | 9f83899a | Guido Trotter | |
454 | 7faf5110 | Michael Hanselmann | You will need to configure your routing table basic routes and rules |
455 | 7faf5110 | Michael Hanselmann | outside of ganeti. The vif scripts will only add /32 routes to your |
456 | 7faf5110 | Michael Hanselmann | instances, through their interface, in the table you specified (under |
457 | 7faf5110 | Michael Hanselmann | KVM, and in the main table under Xen). |
458 | 9f83899a | Guido Trotter | |
459 | 57fb6fcb | Guido Trotter | Also for "openvswitch mode" under Xen a custom network script is needed. |
460 | 57fb6fcb | Guido Trotter | Under KVM everything should work, but you'll need to configure your |
461 | 57fb6fcb | Guido Trotter | switches outside of Ganeti (as for bridges). |
462 | 57fb6fcb | Guido Trotter | |
463 | 12f9d75e | Iustin Pop | .. admonition:: Bridging issues with certain kernels |
464 | 12f9d75e | Iustin Pop | |
465 | 12f9d75e | Iustin Pop | Some kernel versions (e.g. 2.6.32) have an issue where the bridge |
466 | 12f9d75e | Iustin Pop | will automatically change its ``MAC`` address to the lower-numbered |
467 | 12f9d75e | Iustin Pop | slave on port addition and removal. This means that, depending on |
468 | 12f9d75e | Iustin Pop | the ``MAC`` address of the actual NIC on the node and the addresses |
469 | 12f9d75e | Iustin Pop | of the instances, it could be that starting, stopping or migrating |
470 | 12f9d75e | Iustin Pop | instances will lead to timeouts due to the address of the bridge |
471 | 12f9d75e | Iustin Pop | (and thus node itself) changing. |
472 | 12f9d75e | Iustin Pop | |
473 | 12f9d75e | Iustin Pop | To prevent this, it's enough to set the bridge manually to a |
474 | 12f9d75e | Iustin Pop | specific ``MAC`` address, which will disable this automatic address |
475 | 12f9d75e | Iustin Pop | change. In Debian, this can be done as follows in the bridge |
476 | 12f9d75e | Iustin Pop | configuration snippet:: |
477 | 12f9d75e | Iustin Pop | |
478 | 12f9d75e | Iustin Pop | up ip link set addr $(cat /sys/class/net/$IFACE/address) dev $IFACE |
479 | 12f9d75e | Iustin Pop | |
480 | 12f9d75e | Iustin Pop | which will "set" the bridge address to the initial one, disallowing |
481 | 12f9d75e | Iustin Pop | changes. |
482 | 12f9d75e | Iustin Pop | |
483 | 9f83899a | Guido Trotter | .. admonition:: Bridging under Debian |
484 | 28e15341 | Iustin Pop | |
485 | 28e15341 | Iustin Pop | The recommended way to configure the Xen bridge is to edit your |
486 | 28e15341 | Iustin Pop | ``/etc/network/interfaces`` file and substitute your normal |
487 | 28e15341 | Iustin Pop | Ethernet stanza with the following snippet:: |
488 | 28e15341 | Iustin Pop | |
489 | 28e15341 | Iustin Pop | auto xen-br0 |
490 | 28e15341 | Iustin Pop | iface xen-br0 inet static |
491 | f6d62af4 | Iustin Pop | address %YOUR_IP_ADDRESS% |
492 | f6d62af4 | Iustin Pop | netmask %YOUR_NETMASK% |
493 | f6d62af4 | Iustin Pop | network %YOUR_NETWORK% |
494 | f6d62af4 | Iustin Pop | broadcast %YOUR_BROADCAST_ADDRESS% |
495 | f6d62af4 | Iustin Pop | gateway %YOUR_GATEWAY% |
496 | 28e15341 | Iustin Pop | bridge_ports eth0 |
497 | 28e15341 | Iustin Pop | bridge_stp off |
498 | 28e15341 | Iustin Pop | bridge_fd 0 |
499 | 12f9d75e | Iustin Pop | # example for setting manually the bridge address to the eth0 NIC |
500 | 12f9d75e | Iustin Pop | up ip link set addr $(cat /sys/class/net/eth0/address) dev $IFACE |
501 | 28e15341 | Iustin Pop | |
502 | f6d62af4 | Iustin Pop | The following commands need to be executed on the local console:: |
503 | 28e15341 | Iustin Pop | |
504 | f6d62af4 | Iustin Pop | $ ifdown eth0 |
505 | f6d62af4 | Iustin Pop | $ ifup xen-br0 |
506 | 28e15341 | Iustin Pop | |
507 | 28e15341 | Iustin Pop | To check if the bridge is setup, use the ``ip`` and ``brctl show`` |
508 | 28e15341 | Iustin Pop | commands:: |
509 | 28e15341 | Iustin Pop | |
510 | f6d62af4 | Iustin Pop | $ ip a show xen-br0 |
511 | 28e15341 | Iustin Pop | 9: xen-br0: <BROADCAST,MULTICAST,UP,10000> mtu 1500 qdisc noqueue |
512 | 28e15341 | Iustin Pop | link/ether 00:20:fc:1e:d5:5d brd ff:ff:ff:ff:ff:ff |
513 | 28e15341 | Iustin Pop | inet 10.1.1.200/24 brd 10.1.1.255 scope global xen-br0 |
514 | 28e15341 | Iustin Pop | inet6 fe80::220:fcff:fe1e:d55d/64 scope link |
515 | 28e15341 | Iustin Pop | valid_lft forever preferred_lft forever |
516 | 28e15341 | Iustin Pop | |
517 | f6d62af4 | Iustin Pop | $ brctl show xen-br0 |
518 | 28e15341 | Iustin Pop | bridge name bridge id STP enabled interfaces |
519 | 28e15341 | Iustin Pop | xen-br0 8000.0020fc1ed55d no eth0 |
520 | 28e15341 | Iustin Pop | |
521 | c71a1a3d | Iustin Pop | .. _configure-lvm-label: |
522 | c71a1a3d | Iustin Pop | |
523 | 28e15341 | Iustin Pop | Configuring LVM |
524 | 28e15341 | Iustin Pop | +++++++++++++++ |
525 | 28e15341 | Iustin Pop | |
526 | 28e15341 | Iustin Pop | **Mandatory** on all nodes. |
527 | 28e15341 | Iustin Pop | |
528 | 28e15341 | Iustin Pop | The volume group is required to be at least 20GiB. |
529 | 28e15341 | Iustin Pop | |
530 | c71a1a3d | Iustin Pop | If you haven't configured your LVM volume group at install time you need |
531 | c71a1a3d | Iustin Pop | to do it before trying to initialize the Ganeti cluster. This is done by |
532 | c71a1a3d | Iustin Pop | formatting the devices/partitions you want to use for it and then adding |
533 | c71a1a3d | Iustin Pop | them to the relevant volume group:: |
534 | 28e15341 | Iustin Pop | |
535 | f6d62af4 | Iustin Pop | $ pvcreate /dev/%sda3% |
536 | f6d62af4 | Iustin Pop | $ vgcreate xenvg /dev/%sda3% |
537 | 28e15341 | Iustin Pop | |
538 | 28e15341 | Iustin Pop | or:: |
539 | 28e15341 | Iustin Pop | |
540 | f6d62af4 | Iustin Pop | $ pvcreate /dev/%sdb1% |
541 | f6d62af4 | Iustin Pop | $ pvcreate /dev/%sdc1% |
542 | f6d62af4 | Iustin Pop | $ vgcreate xenvg /dev/%sdb1% /dev/%sdc1% |
543 | 28e15341 | Iustin Pop | |
544 | 28e15341 | Iustin Pop | If you want to add a device later you can do so with the *vgextend* |
545 | 28e15341 | Iustin Pop | command:: |
546 | 28e15341 | Iustin Pop | |
547 | f6d62af4 | Iustin Pop | $ pvcreate /dev/%sdd1% |
548 | f6d62af4 | Iustin Pop | $ vgextend xenvg /dev/%sdd1% |
549 | 28e15341 | Iustin Pop | |
550 | 28e15341 | Iustin Pop | Optional: it is recommended to configure LVM not to scan the DRBD |
551 | 28e15341 | Iustin Pop | devices for physical volumes. This can be accomplished by editing |
552 | c71a1a3d | Iustin Pop | ``/etc/lvm/lvm.conf`` and adding the ``/dev/drbd[0-9]+`` regular |
553 | f6d62af4 | Iustin Pop | expression to the ``filter`` variable, like this: |
554 | f6d62af4 | Iustin Pop | |
555 | f6d62af4 | Iustin Pop | .. code-block:: text |
556 | 28e15341 | Iustin Pop | |
557 | 28e15341 | Iustin Pop | filter = ["r|/dev/cdrom|", "r|/dev/drbd[0-9]+|" ] |
558 | 28e15341 | Iustin Pop | |
559 | c71a1a3d | Iustin Pop | Note that with Ganeti a helper script is provided - ``lvmstrap`` which |
560 | c71a1a3d | Iustin Pop | will erase and configure as LVM any not in-use disk on your system. This |
561 | c71a1a3d | Iustin Pop | is dangerous and it's recommended to read its ``--help`` output if you |
562 | c71a1a3d | Iustin Pop | want to use it. |
563 | c71a1a3d | Iustin Pop | |
564 | 28e15341 | Iustin Pop | Installing Ganeti |
565 | 28e15341 | Iustin Pop | +++++++++++++++++ |
566 | 28e15341 | Iustin Pop | |
567 | 28e15341 | Iustin Pop | **Mandatory** on all nodes. |
568 | 28e15341 | Iustin Pop | |
569 | 28e15341 | Iustin Pop | It's now time to install the Ganeti software itself. Download the |
570 | 28e15341 | Iustin Pop | source from the project page at `<http://code.google.com/p/ganeti/>`_, |
571 | f6d62af4 | Iustin Pop | and install it (replace 2.6.0 with the latest version):: |
572 | 28e15341 | Iustin Pop | |
573 | f6d62af4 | Iustin Pop | $ tar xvzf ganeti-%2.6.0%.tar.gz |
574 | f6d62af4 | Iustin Pop | $ cd ganeti-%2.6.0% |
575 | f6d62af4 | Iustin Pop | $ ./configure --localstatedir=/var --sysconfdir=/etc |
576 | f6d62af4 | Iustin Pop | $ make |
577 | f6d62af4 | Iustin Pop | $ make install |
578 | f6d62af4 | Iustin Pop | $ mkdir /srv/ganeti/ /srv/ganeti/os /srv/ganeti/export |
579 | 28e15341 | Iustin Pop | |
580 | c71a1a3d | Iustin Pop | You also need to copy the file ``doc/examples/ganeti.initd`` from the |
581 | c71a1a3d | Iustin Pop | source archive to ``/etc/init.d/ganeti`` and register it with your |
582 | 28e15341 | Iustin Pop | distribution's startup scripts, for example in Debian:: |
583 | 28e15341 | Iustin Pop | |
584 | 0da22bc3 | Sebastian Gebhard | $ chmod +x /etc/init.d/ganeti |
585 | f6d62af4 | Iustin Pop | $ update-rc.d ganeti defaults 20 80 |
586 | 28e15341 | Iustin Pop | |
587 | c71a1a3d | Iustin Pop | In order to automatically restart failed instances, you need to setup a |
588 | c71a1a3d | Iustin Pop | cron job run the *ganeti-watcher* command. A sample cron file is |
589 | c71a1a3d | Iustin Pop | provided in the source at ``doc/examples/ganeti.cron`` and you can copy |
590 | c71a1a3d | Iustin Pop | that (eventually altering the path) to ``/etc/cron.d/ganeti``. |
591 | c71a1a3d | Iustin Pop | |
592 | c71a1a3d | Iustin Pop | What gets installed |
593 | c71a1a3d | Iustin Pop | ~~~~~~~~~~~~~~~~~~~ |
594 | c71a1a3d | Iustin Pop | |
595 | c71a1a3d | Iustin Pop | The above ``make install`` invocation, or installing via your |
596 | c71a1a3d | Iustin Pop | distribution mechanisms, will install on the system: |
597 | c71a1a3d | Iustin Pop | |
598 | c71a1a3d | Iustin Pop | - a set of python libraries under the *ganeti* namespace (depending on |
599 | c71a1a3d | Iustin Pop | the python version this can be located in either |
600 | c71a1a3d | Iustin Pop | ``lib/python-$ver/site-packages`` or various other locations) |
601 | c71a1a3d | Iustin Pop | - a set of programs under ``/usr/local/sbin`` or ``/usr/sbin`` |
602 | f6d62af4 | Iustin Pop | - if the htools component was enabled, a set of programs unde |
603 | f6d62af4 | Iustin Pop | ``/usr/local/bin`` or ``/usr/bin/`` |
604 | c71a1a3d | Iustin Pop | - man pages for the above programs |
605 | c71a1a3d | Iustin Pop | - a set of tools under the ``lib/ganeti/tools`` directory |
606 | c71a1a3d | Iustin Pop | - an example iallocator script (see the admin guide for details) under |
607 | c71a1a3d | Iustin Pop | ``lib/ganeti/iallocators`` |
608 | c71a1a3d | Iustin Pop | - a cron job that is needed for cluster maintenance |
609 | c71a1a3d | Iustin Pop | - an init script for automatic startup of Ganeti daemons |
610 | c71a1a3d | Iustin Pop | - provided but not installed automatically by ``make install`` is a bash |
611 | c71a1a3d | Iustin Pop | completion script that hopefully will ease working with the many |
612 | c71a1a3d | Iustin Pop | cluster commands |
613 | 28e15341 | Iustin Pop | |
614 | 28e15341 | Iustin Pop | Installing the Operating System support packages |
615 | 28e15341 | Iustin Pop | ++++++++++++++++++++++++++++++++++++++++++++++++ |
616 | 28e15341 | Iustin Pop | |
617 | 28e15341 | Iustin Pop | **Mandatory** on all nodes. |
618 | 28e15341 | Iustin Pop | |
619 | 28e15341 | Iustin Pop | To be able to install instances you need to have an Operating System |
620 | 28e15341 | Iustin Pop | installation script. An example OS that works under Debian and can |
621 | 28e15341 | Iustin Pop | install Debian and Ubuntu instace OSes is provided on the project web |
622 | c71a1a3d | Iustin Pop | site. Download it from the project page and follow the instructions in |
623 | a425810f | Guido Trotter | the ``README`` file. Here is the installation procedure (replace 0.9 |
624 | c71a1a3d | Iustin Pop | with the latest version that is compatible with your ganeti version):: |
625 | 28e15341 | Iustin Pop | |
626 | f6d62af4 | Iustin Pop | $ cd /usr/local/src/ |
627 | f6d62af4 | Iustin Pop | $ wget http://ganeti.googlecode.com/files/ganeti-instance-debootstrap-%0.9%.tar.gz |
628 | f6d62af4 | Iustin Pop | $ tar xzf ganeti-instance-debootstrap-%0.9%.tar.gz |
629 | f6d62af4 | Iustin Pop | $ cd ganeti-instance-debootstrap-%0.9% |
630 | f6d62af4 | Iustin Pop | $ ./configure |
631 | f6d62af4 | Iustin Pop | $ make |
632 | f6d62af4 | Iustin Pop | $ make install |
633 | 28e15341 | Iustin Pop | |
634 | 28e15341 | Iustin Pop | In order to use this OS definition, you need to have internet access |
635 | 28e15341 | Iustin Pop | from your nodes and have the *debootstrap*, *dump* and *restore* |
636 | 28e15341 | Iustin Pop | commands installed on all nodes. Also, if the OS is configured to |
637 | 28e15341 | Iustin Pop | partition the instance's disk in |
638 | 28e15341 | Iustin Pop | ``/etc/default/ganeti-instance-debootstrap``, you will need *kpartx* |
639 | 28e15341 | Iustin Pop | installed. |
640 | 28e15341 | Iustin Pop | |
641 | 28e15341 | Iustin Pop | .. admonition:: Debian |
642 | 28e15341 | Iustin Pop | |
643 | 28e15341 | Iustin Pop | Use this command on all nodes to install the required packages:: |
644 | 28e15341 | Iustin Pop | |
645 | f6d62af4 | Iustin Pop | $ apt-get install debootstrap dump kpartx |
646 | f6d62af4 | Iustin Pop | |
647 | f6d62af4 | Iustin Pop | Or alternatively install the OS definition from the Debian package:: |
648 | f6d62af4 | Iustin Pop | |
649 | f6d62af4 | Iustin Pop | $ apt-get install ganeti-instance-debootstrap |
650 | 28e15341 | Iustin Pop | |
651 | a425810f | Guido Trotter | .. admonition:: KVM |
652 | a425810f | Guido Trotter | |
653 | a425810f | Guido Trotter | In order for debootstrap instances to be able to shutdown cleanly |
654 | f6d62af4 | Iustin Pop | they must install have basic ACPI support inside the instance. Which |
655 | f6d62af4 | Iustin Pop | packages are needed depend on the exact flavor of Debian or Ubuntu |
656 | a425810f | Guido Trotter | which you're installing, but the example defaults file has a |
657 | f6d62af4 | Iustin Pop | commented out configuration line that works for Debian Lenny and |
658 | f6d62af4 | Iustin Pop | Squeeze:: |
659 | a425810f | Guido Trotter | |
660 | a425810f | Guido Trotter | EXTRA_PKGS="acpi-support-base,console-tools,udev" |
661 | a425810f | Guido Trotter | |
662 | f6d62af4 | Iustin Pop | ``kbd`` can be used instead of ``console-tools``, and more packages |
663 | f6d62af4 | Iustin Pop | can be added, of course, if needed. |
664 | a425810f | Guido Trotter | |
665 | 28e15341 | Iustin Pop | Alternatively, you can create your own OS definitions. See the manpage |
666 | 22ac4136 | Michael Hanselmann | :manpage:`ganeti-os-interface(7)`. |
667 | 28e15341 | Iustin Pop | |
668 | 28e15341 | Iustin Pop | Initializing the cluster |
669 | 28e15341 | Iustin Pop | ++++++++++++++++++++++++ |
670 | 28e15341 | Iustin Pop | |
671 | c71a1a3d | Iustin Pop | **Mandatory** once per cluster, on the first node. |
672 | 28e15341 | Iustin Pop | |
673 | c71a1a3d | Iustin Pop | The last step is to initialize the cluster. After you have repeated the |
674 | 28e15341 | Iustin Pop | above process on all of your nodes, choose one as the master, and |
675 | 28e15341 | Iustin Pop | execute:: |
676 | 28e15341 | Iustin Pop | |
677 | f6d62af4 | Iustin Pop | $ gnt-cluster init %CLUSTERNAME% |
678 | 28e15341 | Iustin Pop | |
679 | c71a1a3d | Iustin Pop | The *CLUSTERNAME* is a hostname, which must be resolvable (e.g. it must |
680 | c71a1a3d | Iustin Pop | exist in DNS or in ``/etc/hosts``) by all the nodes in the cluster. You |
681 | c71a1a3d | Iustin Pop | must choose a name different from any of the nodes names for a |
682 | c71a1a3d | Iustin Pop | multi-node cluster. In general the best choice is to have a unique name |
683 | c71a1a3d | Iustin Pop | for a cluster, even if it consists of only one machine, as you will be |
684 | c71a1a3d | Iustin Pop | able to expand it later without any problems. Please note that the |
685 | c71a1a3d | Iustin Pop | hostname used for this must resolve to an IP address reserved |
686 | 28e15341 | Iustin Pop | **exclusively** for this purpose, and cannot be the name of the first |
687 | 28e15341 | Iustin Pop | (master) node. |
688 | 28e15341 | Iustin Pop | |
689 | 7faf5110 | Michael Hanselmann | If you want to use a bridge which is not ``xen-br0``, or no bridge at |
690 | e721c742 | Guido Trotter | all, change it with the ``--nic-parameters`` option. For example to |
691 | f6d62af4 | Iustin Pop | bridge on br0 you can add:: |
692 | e721c742 | Guido Trotter | |
693 | e721c742 | Guido Trotter | --nic-parameters link=br0 |
694 | e721c742 | Guido Trotter | |
695 | e721c742 | Guido Trotter | Or to not bridge at all, and use a separate routing table:: |
696 | e721c742 | Guido Trotter | |
697 | e721c742 | Guido Trotter | --nic-parameters mode=routed,link=100 |
698 | 9f83899a | Guido Trotter | |
699 | f6d62af4 | Iustin Pop | If you don't have a ``xen-br0`` interface you also have to specify a |
700 | f6d62af4 | Iustin Pop | different network interface which will get the cluster IP, on the master |
701 | e721c742 | Guido Trotter | node, by using the ``--master-netdev <device>`` option. |
702 | 28e15341 | Iustin Pop | |
703 | 28e15341 | Iustin Pop | You can use a different name than ``xenvg`` for the volume group (but |
704 | 28e15341 | Iustin Pop | note that the name must be identical on all nodes). In this case you |
705 | 240c769f | Andrea Spadaccini | need to specify it by passing the *--vg-name <VGNAME>* option to |
706 | 240c769f | Andrea Spadaccini | ``gnt-cluster init``. |
707 | 28e15341 | Iustin Pop | |
708 | c71a1a3d | Iustin Pop | To set up the cluster as an Xen HVM cluster, use the |
709 | 28e15341 | Iustin Pop | ``--enabled-hypervisors=xen-hvm`` option to enable the HVM hypervisor |
710 | c71a1a3d | Iustin Pop | (you can also add ``,xen-pvm`` to enable the PVM one too). You will also |
711 | c71a1a3d | Iustin Pop | need to create the VNC cluster password file |
712 | 28e15341 | Iustin Pop | ``/etc/ganeti/vnc-cluster-password`` which contains one line with the |
713 | 28e15341 | Iustin Pop | default VNC password for the cluster. |
714 | 28e15341 | Iustin Pop | |
715 | 28e15341 | Iustin Pop | To setup the cluster for KVM-only usage (KVM and Xen cannot be mixed), |
716 | 28e15341 | Iustin Pop | pass ``--enabled-hypervisors=kvm`` to the init command. |
717 | 28e15341 | Iustin Pop | |
718 | 28e15341 | Iustin Pop | You can also invoke the command with the ``--help`` option in order to |
719 | 28e15341 | Iustin Pop | see all the possibilities. |
720 | 28e15341 | Iustin Pop | |
721 | b8313b29 | Guido Trotter | Hypervisor/Network/Cluster parameters |
722 | b8313b29 | Guido Trotter | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
723 | b8313b29 | Guido Trotter | |
724 | b8313b29 | Guido Trotter | Please note that the default hypervisor/network/cluster parameters may |
725 | b8313b29 | Guido Trotter | not be the correct one for your environment. Carefully check them, and |
726 | f6d62af4 | Iustin Pop | change them either at cluster init time, or later with ``gnt-cluster |
727 | f6d62af4 | Iustin Pop | modify``. |
728 | b8313b29 | Guido Trotter | |
729 | b8313b29 | Guido Trotter | Your instance types, networking environment, hypervisor type and version |
730 | b8313b29 | Guido Trotter | may all affect what kind of parameters should be used on your cluster. |
731 | b8313b29 | Guido Trotter | |
732 | 18c3417b | Guido Trotter | .. admonition:: KVM |
733 | 18c3417b | Guido Trotter | |
734 | 18c3417b | Guido Trotter | Instances are by default configured to use a host kernel, and to be |
735 | 18c3417b | Guido Trotter | reached via serial console, which works nice for Linux paravirtualized |
736 | 18c3417b | Guido Trotter | instances. If you want fully virtualized instances you may want to |
737 | 18c3417b | Guido Trotter | handle their kernel inside the instance, and to use VNC. |
738 | 18c3417b | Guido Trotter | |
739 | 18c3417b | Guido Trotter | Some versions of KVM have a bug that will make an instance hang when |
740 | 18c3417b | Guido Trotter | configured to use the serial console (which is the default) unless a |
741 | 18c3417b | Guido Trotter | connection is made to it within about 2 seconds of the instance's |
742 | 18c3417b | Guido Trotter | startup. For such case it's recommended to disable the |
743 | 18c3417b | Guido Trotter | ``serial_console`` option. |
744 | 18c3417b | Guido Trotter | |
745 | b8313b29 | Guido Trotter | |
746 | 28e15341 | Iustin Pop | Joining the nodes to the cluster |
747 | 28e15341 | Iustin Pop | ++++++++++++++++++++++++++++++++ |
748 | 28e15341 | Iustin Pop | |
749 | 28e15341 | Iustin Pop | **Mandatory** for all the other nodes. |
750 | 28e15341 | Iustin Pop | |
751 | c71a1a3d | Iustin Pop | After you have initialized your cluster you need to join the other nodes |
752 | c71a1a3d | Iustin Pop | to it. You can do so by executing the following command on the master |
753 | c71a1a3d | Iustin Pop | node:: |
754 | 28e15341 | Iustin Pop | |
755 | f6d62af4 | Iustin Pop | $ gnt-node add %NODENAME% |
756 | 28e15341 | Iustin Pop | |
757 | 28e15341 | Iustin Pop | Separate replication network |
758 | 28e15341 | Iustin Pop | ++++++++++++++++++++++++++++ |
759 | 28e15341 | Iustin Pop | |
760 | 28e15341 | Iustin Pop | **Optional** |
761 | 28e15341 | Iustin Pop | |
762 | 28e15341 | Iustin Pop | Ganeti uses DRBD to mirror the disk of the virtual instances between |
763 | 28e15341 | Iustin Pop | nodes. To use a dedicated network interface for this (in order to |
764 | 28e15341 | Iustin Pop | improve performance or to enhance security) you need to configure an |
765 | 28e15341 | Iustin Pop | additional interface for each node. Use the *-s* option with |
766 | 28e15341 | Iustin Pop | ``gnt-cluster init`` and ``gnt-node add`` to specify the IP address of |
767 | 28e15341 | Iustin Pop | this secondary interface to use for each node. Note that if you |
768 | c71a1a3d | Iustin Pop | specified this option at cluster setup time, you must afterwards use it |
769 | c71a1a3d | Iustin Pop | for every node add operation. |
770 | 28e15341 | Iustin Pop | |
771 | 28e15341 | Iustin Pop | Testing the setup |
772 | 28e15341 | Iustin Pop | +++++++++++++++++ |
773 | 28e15341 | Iustin Pop | |
774 | c71a1a3d | Iustin Pop | Execute the ``gnt-node list`` command to see all nodes in the cluster:: |
775 | 28e15341 | Iustin Pop | |
776 | f6d62af4 | Iustin Pop | $ gnt-node list |
777 | 28e15341 | Iustin Pop | Node DTotal DFree MTotal MNode MFree Pinst Sinst |
778 | 28e15341 | Iustin Pop | node1.example.com 197404 197404 2047 1896 125 0 0 |
779 | 28e15341 | Iustin Pop | |
780 | c71a1a3d | Iustin Pop | The above shows a couple of things: |
781 | 28e15341 | Iustin Pop | |
782 | c71a1a3d | Iustin Pop | - The various Ganeti daemons can talk to each other |
783 | c71a1a3d | Iustin Pop | - Ganeti can examine the storage of the node (DTotal/DFree) |
784 | c71a1a3d | Iustin Pop | - Ganeti can talk to the selected hypervisor (MTotal/MNode/MFree) |
785 | 28e15341 | Iustin Pop | |
786 | c71a1a3d | Iustin Pop | Cluster burnin |
787 | c71a1a3d | Iustin Pop | ~~~~~~~~~~~~~~ |
788 | 28e15341 | Iustin Pop | |
789 | c71a1a3d | Iustin Pop | With Ganeti a tool called :command:`burnin` is provided that can test |
790 | c71a1a3d | Iustin Pop | most of the Ganeti functionality. The tool is installed under the |
791 | c71a1a3d | Iustin Pop | ``lib/ganeti/tools`` directory (either under ``/usr`` or ``/usr/local`` |
792 | c71a1a3d | Iustin Pop | based on the installation method). See more details under |
793 | c71a1a3d | Iustin Pop | :ref:`burnin-label`. |
794 | 28e15341 | Iustin Pop | |
795 | c71a1a3d | Iustin Pop | Further steps |
796 | c71a1a3d | Iustin Pop | ------------- |
797 | 28e15341 | Iustin Pop | |
798 | c71a1a3d | Iustin Pop | You can now proceed either to the :doc:`admin`, or read the manpages of |
799 | c71a1a3d | Iustin Pop | the various commands (:manpage:`ganeti(7)`, :manpage:`gnt-cluster(8)`, |
800 | c71a1a3d | Iustin Pop | :manpage:`gnt-node(8)`, :manpage:`gnt-instance(8)`, |
801 | c71a1a3d | Iustin Pop | :manpage:`gnt-job(8)`). |
802 | 28e15341 | Iustin Pop | |
803 | c71a1a3d | Iustin Pop | .. rubric:: Footnotes |
804 | 28e15341 | Iustin Pop | |
805 | c71a1a3d | Iustin Pop | .. [#defkernel] The kernel and initrd paths can be changed at either |
806 | c71a1a3d | Iustin Pop | cluster level (which changes the default for all instances) or at |
807 | c71a1a3d | Iustin Pop | instance level. |
808 | 558fd122 | Michael Hanselmann | |
809 | 558fd122 | Michael Hanselmann | .. vim: set textwidth=72 : |
810 | c71a1a3d | Iustin Pop | .. Local Variables: |
811 | c71a1a3d | Iustin Pop | .. mode: rst |
812 | c71a1a3d | Iustin Pop | .. fill-column: 72 |
813 | c71a1a3d | Iustin Pop | .. End: |