root / README.deploy @ 3829bd80
History | View | Annotate | Download (15.8 kB)
1 |
README.deploy -- Instructions for a basic deployment of Synnefo v0.4 |
---|---|
2 |
|
3 |
This document describes the basic steps to obtain a basic, working Synnefo |
4 |
deployment. It begins by examining the different node roles, then moves to the |
5 |
installation and setup of distinct software components. |
6 |
|
7 |
It applies to Synnefo v0.4. |
8 |
|
9 |
|
10 |
Node types |
11 |
=========== |
12 |
|
13 |
Nodes in a Synnefo deployment belong in one of the following types: |
14 |
|
15 |
* DB: |
16 |
A node [or more than one nodes, if using an HA configuration], |
17 |
running a DB engine supported by the Django ORM layer. The DB |
18 |
is the single source of truth for the servicing of API requests by |
19 |
Synnefo. |
20 |
Services: PostgreSQL / MySQL |
21 |
|
22 |
* APISERVER: |
23 |
A node running the implementation of the OpenStack API, in Django. |
24 |
Any number of APISERVERs can be used, in a load-balancing configuration, |
25 |
without any special consideration. Access to a common DB ensures |
26 |
consistency. |
27 |
Services: Web server, vncauthproxy |
28 |
|
29 |
* QUEUE: |
30 |
A node running the RabbitMQ software, which provides AMQP functionality. |
31 |
More than one QUEUE nodes may be deployed, in an HA configuration. Such |
32 |
deployments require shared storage, provided e.g., by DRBD. |
33 |
Services: RabbitMQ [rabbitmq-server] |
34 |
|
35 |
* LOGIC: |
36 |
A node running the business logic of Synnefo, in Django. It dequeues |
37 |
messages from QUEUE nodes, and provides the context in which business |
38 |
logic functions run. It uses Django ORM to connect to the common DB and |
39 |
update the state of the system, based on notifications received from the |
40 |
rest of the infrastructure, over AMQP. |
41 |
Services: the Synnefo logic dispatcher [/logic/dispatcher.py] |
42 |
|
43 |
* GANETI-MASTER and GANETI-NODE: |
44 |
A single GANETI-MASTER and a large number of GANETI-NODEs constitute the |
45 |
Ganeti backend for Synnefo, which undertakes all VM management functions. |
46 |
Any APISERVER can issue commands to the GANETI-MASTER, over RAPI, to effect |
47 |
changes in the state of the VMs. The GANETI-MASTER runs the Ganeti request |
48 |
queue. |
49 |
Services: |
50 |
only on GANETI-MASTER: |
51 |
the Synnefo Ganeti monitoring daemon [/ganeti/snf-ganeti-eventd], |
52 |
and the Synnefo Ganeti hook [/ganeti/snf-ganeti-hook.py]. |
53 |
on each GANETI_NODE: |
54 |
a deployment-specific KVM ifup script |
55 |
properly configured NFDHCPD |
56 |
|
57 |
|
58 |
Installation Process |
59 |
===================== |
60 |
|
61 |
This section describes the installation process of the various node roles |
62 |
in a Synnefo deployment. |
63 |
|
64 |
|
65 |
0. Allocation of physical nodes: |
66 |
Determine the role of every physical node in your deployment. |
67 |
|
68 |
|
69 |
1. Ganeti installation: |
70 |
Synnefo requires a working Ganeti installation at the backend. Installation |
71 |
of Ganeti is not covered by this document, please refer to |
72 |
http://docs.ganeti.org/ganeti/current/html for all the gory details. A |
73 |
successful Ganeti installation concludes with a working GANETI-MASTER and a |
74 |
number of GANETI-NODEs. |
75 |
|
76 |
|
77 |
2. RabbitMQ installation: |
78 |
RabbitMQ is used as a generic message broker for the system. It should |
79 |
be installed on two seperate QUEUE nodes (VMs should be enough for the |
80 |
moment) in a high availability configuration as described here: |
81 |
|
82 |
http://www.rabbitmq.com/pacemaker.html |
83 |
|
84 |
After installation, create a user and set its permissions |
85 |
rabbitmqctl add_user okeanos 0k3@n0s |
86 |
rabbitmqctl set_permissions -p / okeanos "^.*" ".*" ".*" |
87 |
|
88 |
The values set for the user and password must be mirrored in the |
89 |
RABBIT_* variables in settings.py (see step 6) |
90 |
|
91 |
|
92 |
3. Web server installation: |
93 |
A Web Server (e.g., Apache) needs to be installed on the APISERVERs, |
94 |
and be configured to run the Synnefo Django project appropriately. |
95 |
Selection and configuration of a Web server is outside the scope of this |
96 |
document. |
97 |
|
98 |
For testing or development purposes, Django's own development server, |
99 |
./manage.py runserver can be used. |
100 |
|
101 |
|
102 |
4. Installation of the Synnefo Django project: |
103 |
As of v0.4, the Synnefo Django project needs to be installed on nodes |
104 |
of type APISERVER, LOGIC and on the GANETI-MASTER, with a properly |
105 |
configured settings.py. In later revisions, the specific parts of the Django |
106 |
project which need to run on each node type will be identified. |
107 |
|
108 |
Synnefo is written in Python 2.6 and depends on the following Python |
109 |
modules: [package versions confirmed to be compatible are in braces] |
110 |
|
111 |
* django 1.2 [Django==1.2.4] |
112 |
* simplejson [simplejson==2.1.3] |
113 |
* pycurl [pycurl==7.19.0] |
114 |
* python-dateutil [python-dateutil==1.4.1] |
115 |
WARNING: version python-dateutil==2.0 downloaded by pip known *not* to |
116 |
work with Python 2.6 |
117 |
* south [south==0.7.1] |
118 |
WARNING: not working with Debian squeeze's default south-0.7-1 package. |
119 |
* amqplib [amqplib==0.6.1] |
120 |
|
121 |
also, depending on the database engine of choice, on one of the following: |
122 |
* MySQL-python [MySQL-python==1.2.3] |
123 |
* psycopg2 [psycopg2==2.4] |
124 |
|
125 |
if the invitations application is deployed, the following |
126 |
dependencies should be installed: |
127 |
|
128 |
* pycrypto==2.1.0 |
129 |
|
130 |
if you want to test the ganeti (FIXME) |
131 |
|
132 |
to run the user interface tests, selenium must be installed |
133 |
* selenium [?] |
134 |
|
135 |
The easiest method for installation of the Django project is to setup a |
136 |
working environment through virtualenv. Alternatively, you can use your |
137 |
system's package manager to install the dependencies (e.g. Macports has them |
138 |
all). |
139 |
|
140 |
* On Snow Leopard and linux (64-bit), you have to set the following |
141 |
environment variable for pip to compile the dependencies correctly. |
142 |
|
143 |
$ export ARCHFLAGS="-arch x86_64" |
144 |
|
145 |
* On Ubuntu, a few more packages must be installed before installing the |
146 |
prerequisite Python libraries |
147 |
|
148 |
$ sudo aptitude install libcurl3-gnutls libcurl3-gnutls-dev uuid-dev |
149 |
|
150 |
Checkout the code and install the Python prerequisites. This assumes |
151 |
that python is already installed on the host. |
152 |
|
153 |
$ sudo easy_install virtualenv |
154 |
$ git clone https://user@code.grnet.gr/git/synnefo synnefo |
155 |
$ virtualenv --python=python2.6 synnefo --no-site-packages |
156 |
... |
157 |
$ cd synnefo |
158 |
$ ./bin/pip install <list_of_dependencies> |
159 |
|
160 |
|
161 |
5. Database installation: |
162 |
A database supported by the Django ORM layer must be installed on nodes |
163 |
of type DB. The choices are: SQLIte, MySQL, PostgreSQL. |
164 |
|
165 |
* SQLite: |
166 |
The python sqlite driver is available by default with Python so no |
167 |
additional configuration is required. Also, most self-respecting systems |
168 |
have the sqlite library installed by default. |
169 |
|
170 |
* MySQL: |
171 |
MySQL must be installed first: |
172 |
|
173 |
* Ubuntu - Debian |
174 |
$ sudo apt-get install libmysqlclient-dev |
175 |
|
176 |
* MacPorts |
177 |
$ sudo port install mysql5 |
178 |
|
179 |
Install the MySQL python library on servers running the Django project: |
180 |
|
181 |
$ bin/pip install MySQL-python |
182 |
|
183 |
Note: On MacOSX with Mysql install from MacPorts the above command will |
184 |
fail complaining that it cannot find the mysql_config command. Do the |
185 |
following and restart the installation |
186 |
$ echo "mysql_config = /opt/local/bin/mysql_config5" >> ./build/MySQL-python/site.cfg |
187 |
|
188 |
Configure a MySQL db/account for synnefo |
189 |
$ mysql -u root -p |
190 |
|
191 |
mysql> create database synnefo; |
192 |
mysql> show databases; |
193 |
mysql> GRANT ALL on synnefo.* TO username IDENTIFIED BY 'password'; |
194 |
|
195 |
* PostgreSQL |
196 |
You need to install the PostgreSQL binaries: |
197 |
* Ubuntu - Debian |
198 |
$ sudo apt-get install postgresql-8.4 libpq-dev |
199 |
|
200 |
* MacPorts |
201 |
$ sudo port install postgresql84 |
202 |
|
203 |
Install the postgres Python library |
204 |
$ bin/pip install psycopg2 |
205 |
|
206 |
Configure a postgres db/account for synnefo: |
207 |
|
208 |
Become the postgres user, connect to PostgreSQL: |
209 |
$ sudo su - postgres |
210 |
$ psql |
211 |
|
212 |
Run the following commands: |
213 |
DROP DATABASE synnefo; |
214 |
DROP USER username; |
215 |
CREATE USER username WITH PASSWORD 'password'; |
216 |
CREATE DATABASE synnefo; |
217 |
GRANT ALL PRIVILEGES ON DATABASE synnefo TO username; |
218 |
ALTER DATABASE synnefo OWNER TO username; |
219 |
ALTER USER username CREATEDB; |
220 |
|
221 |
The last line enables the newly created user to create own databases. This |
222 |
is needed for Django to create and drop the test_synnefo database for unit |
223 |
testing. |
224 |
|
225 |
|
226 |
6. Setting up the Django project: |
227 |
The settings.py file for Django may be derived by concatenating the |
228 |
settings.py.dist file contained in the Synnefo distribution with a file |
229 |
containing custom modifications, which shall override all settings deviating |
230 |
from the supplied settings.py.dist. This is recommended to minimize the load |
231 |
of reconstructing settings.py from scratch, since each release currently |
232 |
brings heavy changes to settings.py.dist. |
233 |
|
234 |
Add the following to your custom settings.py, depending on your choice |
235 |
of DB: |
236 |
* SQLite |
237 |
|
238 |
PROJECT_PATH = os.path.dirname(os.path.abspath(__file__)) + '/' |
239 |
|
240 |
DATABASES = { |
241 |
'default': { |
242 |
'ENGINE': 'django.db.backends.sqlite3', |
243 |
'NAME': PROJECT_PATH + 'synnefo.db' # WARN: This must be an absolute path |
244 |
} |
245 |
} |
246 |
|
247 |
* MySQL |
248 |
|
249 |
DATABASES = { |
250 |
'default': { |
251 |
'ENGINE': 'django.db.backends.mysql', |
252 |
'NAME': 'synnefo', |
253 |
'USER': 'USERNAME', |
254 |
'PASSWORD': 'PASSWORD', |
255 |
'HOST': 'HOST', |
256 |
'PORT': 'PORT', |
257 |
'OPTIONS': { |
258 |
'init_command': 'SET storage_engine=INNODB', |
259 |
} |
260 |
} |
261 |
} |
262 |
|
263 |
* PostgreSQL |
264 |
|
265 |
DATABASES = { |
266 |
'default': { |
267 |
'ENGINE': 'django.db.backends.postgresql_psycopg2', |
268 |
'NAME': 'DATABASE', |
269 |
'USER': 'USERNAME', |
270 |
'PASSWORD': 'PASSWORD', |
271 |
'HOST': 'HOST', |
272 |
'PORT': 'PORT', |
273 |
} |
274 |
} |
275 |
|
276 |
Try it out. The following command will attempt to connect to the DB and |
277 |
print out DDL statements. It should not fail. |
278 |
|
279 |
$ ./bin/python manage.py sql db |
280 |
|
281 |
|
282 |
7. Initialization of Synnefo DB: |
283 |
You need to initialize the Synnefo DB and load fixtures |
284 |
db/fixtures/{flavors,images}.json, which make the API usable by end users by |
285 |
defining a sample set of hardware configurations (flavors) and OS images. |
286 |
|
287 |
$ ./bin/python manage.py syncdb |
288 |
$ ./bin/python manage.py migrate db |
289 |
$ ./bin/python manage.py loaddata db/fixtures/flavors.json |
290 |
$ ./bin/python manage.py loaddata db/fixtures/images.json |
291 |
|
292 |
|
293 |
8. Finalization of settings.py: |
294 |
Set the BACKEND_PREFIX_ID variable to some unique prefix, e.g. your commit |
295 |
username in settings.py. Several functional conventions within the system |
296 |
require this variable to include a dash at its end (e.g. snf-) |
297 |
|
298 |
Fix the AMQP-specific settings based on the selected BACKEND_PREFIX_ID. |
299 |
The fix_amqp_settings() function is called once at the end of |
300 |
settings.py.dist, you must call it again if you change BACKEND_PREFIX_ID |
301 |
at some later point. |
302 |
|
303 |
|
304 |
9. Installation of the Ganeti monitoring daemon, /ganeti/snf-ganeti-eventd: |
305 |
The Ganeti monitoring daemon must run on GANETI-MASTER. |
306 |
The Ganeti monitoring daemon has no dependency on Django. |
307 |
|
308 |
Override all relevant settings in settings.d/99-snf-ganeti-eventd.conf, |
309 |
GANETI_* variables. |
310 |
Then, make sure PYTHONPATH contains the parent of the Django project, |
311 |
and start the server on the Ganeti master as root. |
312 |
|
313 |
root:~# export PYTHONPATH=$PYTHONPATH:/opt |
314 |
root:~# /opt/synnefo/ganeti/snf-ganeti-eventd.py |
315 |
|
316 |
TBD: how to handle master migration. |
317 |
|
318 |
|
319 |
10. Installation of the Synnefo dispatcher, /logic/dispatcher.py: |
320 |
The logic dispatcher is part of the Synnefo Django project and must run |
321 |
on LOGIC nodes. |
322 |
|
323 |
The dispatcher retrieves messages from the queue and calls the |
324 |
appropriate handler function as defined in the queue configuration in |
325 |
setttings.py. The default configuration should work directly without |
326 |
any modifications. |
327 |
|
328 |
For the time being The dispatcher must be run by hand: |
329 |
$ ./bin/python ./logic/dispatcher.py |
330 |
|
331 |
The dispatcher should run in at least 2 instances to ensure high |
332 |
(actually, increased) availability. |
333 |
|
334 |
|
335 |
11. Installation of the Synnefo Ganeti hook: |
336 |
The Python script ganeti/snf-ganeti-hook.py is the generic launcher for |
337 |
Synnefo hooks in Ganeti. It resides in the ganeti/ directory under the |
338 |
root of the Synnefo Django project. |
339 |
|
340 |
The hook needs to be enabled for phases |
341 |
post-{add,modify,reboot,start,stop} by *symlinking* |
342 |
in /etc/ganeti/hooks/instance-{add,modify,reboot,start,stop}-post.d |
343 |
on GANETI-MASTER, e.g.: |
344 |
|
345 |
root@ganeti-master:/etc/ganeti/hooks/instance-start-post.d# ls -l |
346 |
lrwxrwxrwx 1 root root 45 May 3 13:45 00-snf-ganeti-hook -> /home/devel/synnefo/ganeti/snf-ganeti-hook.py* |
347 |
|
348 |
IMPORTANT: The link name may only contain "upper and lower case, digits, |
349 |
underscores and hyphens. In other words, the regexp ^[a-zA-Z0-9_-]+$." |
350 |
See http://docs.ganeti.org/ganeti/master/html/hooks.html?highlight=hooks#naming. |
351 |
|
352 |
The script uses the location of the link target to determine the Synnefo |
353 |
Project root, before passing control to the relevant Python code. |
354 |
|
355 |
FIXME: Perhaps require a SYNNEFO_PROJECT_ROOT environment variable? |
356 |
|
357 |
|
358 |
12. Installation of the VNC authentication proxy, vncauthproxy: |
359 |
To support OOB console access to the VMs over VNC, the vncauthproxy |
360 |
daemon must be running on every node of type APISERVER. |
361 |
|
362 |
Download and install vncauthproxy from its own repository, |
363 |
at https://code.grnet.gr/git/vncauthproxy (known good commit: 8799ab6d6e). |
364 |
|
365 |
Edit default settings on top of vncauthproxy.py. |
366 |
Set CTRL_SOCKET in util/vapclient.py to point to its control socket. |
367 |
|
368 |
FIXME: The CTRL_SOCKET setting will be moved to settings.py as |
369 |
VNCAUTHPROXY_CTRL_SOCKET. |
370 |
|
371 |
Create /var/log/vncauthproxy and set its permissions appropriately. |
372 |
|
373 |
|
374 |
13. Installation of the customized Ganeti Instance Image for image deployment: |
375 |
For Synnefo to be able to launch VMs from specified Images, you need |
376 |
the gnt-instance-image OS Provider installed on the Ganeti backend. |
377 |
|
378 |
Download and install gnt-instance-image in all Ganeti nodes from its own |
379 |
repository, at https://code.grnet.gr/git/gnt-instance-image. |
380 |
|
381 |
After installing gnt-instance-image do the following: |
382 |
1. root@ganeti-master:/path-to-repo# cp ./defaults /etc/default/ganeti-instance-image |
383 |
2. Uncomment the following in /etc/default/ganeti-instance-image: |
384 |
SWAP=no |
385 |
ARCH="x86_64" |
386 |
3. Add to /etc/default/ganeti-instance-image, so that make-dump makes |
387 |
no /boot partition: |
388 |
KERNEL_PATH="True" |
389 |
4. Change the path in make-dump line 22 according to your installation |
390 |
(/usr/share/ganeti/os/image/common.sh --> /srv/ganeti/os/image/common.sh) |
391 |
5. In common.sh, comment out the KERNEL_PATH variable initialization. |
392 |
(#KERNEL_PATH="$INSTANCE_HV_kernel_path") |
393 |
6. In /etc/ganeti/instance-image/hooks, make sure the hooks you want to |
394 |
run during instance creation process have execute permission. At least |
395 |
`grub' and `root_passwd' should be triggered to make the instance |
396 |
usable: |
397 |
chmod +x /etc/ganeti/instance-image/hooks/{grub,root_passwd} |
398 |
|
399 |
Your custom mmages should be stored in a dump format under |
400 |
/var/cache/ganeti-instance-image |
401 |
and their filenames should have the following format: |
402 |
{backend_id}-x86_64-root.dump |
403 |
e.g., debian-6.0.1a-x86_64-root.dump (backend_id = "debian-6.0.1a") |
404 |
|
405 |
|
406 |
14. Setup Synnefo-specific networking on the Ganeti backend: |
407 |
This part is deployment-specific and must be customized based on the |
408 |
specific needs of the system administrators. |
409 |
|
410 |
A reference installation will use a Synnefo-specific KVM ifup script, |
411 |
NFDHCPD and pre-provisioned Linux bridges to support public and private |
412 |
network functionality. For this: |
413 |
|
414 |
Grab NFDHCPD from its own repository (https://code.grnet.gr/git/nfdhcpd), |
415 |
install it, modify /etc/nfdhcpd/nfdhcpd.conf to reflect your network |
416 |
configuration. |
417 |
|
418 |
Install a custom KVM ifup script for use by Ganeti, as |
419 |
/etc/ganeti/kvm-vif-bridge, on GANETI-NODEs. A sample implementation is |
420 |
provided under /contrib/ganeti-hooks. Set NFDHCPD_STATE_DIR to point |
421 |
to NFDHCPD's state directory, usually /var/lib/nfdhcpd. |
422 |
|
423 |
|
424 |
15. Create log file directories for Synnefo components, set appropriate |
425 |
permissions. By default logic/dispatcher.py and ganeti/snf-ganeti-eventd.py |
426 |
use /var/log/synnefo. |
427 |
|
428 |
|
429 |
16. (Hopefully) Done |
430 |
|