root / docs / quick-install-intgrt-guide.rst @ d29f0371
History | View | Annotate | Download (17.4 kB)
1 |
.. _quick-install-intgrt-guide: |
---|---|
2 |
|
3 |
Integrator's Quick Installation Guide |
4 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
5 |
|
6 |
This is the Integrator's quick installation guide. |
7 |
|
8 |
It is intended for developers, wishing to implement new functionality |
9 |
inside Synnefo. It assumes thorough familiarity with the |
10 |
:ref:`Synnefo Administrator's Guide <admin-guide>`. |
11 |
|
12 |
It describes how to install the whole synnefo stack on two (2) physical nodes, |
13 |
with minimum configuration. It installs synnefo in a ``virtualenv`` using ``pip |
14 |
install``, and assumes the nodes run Debian Squeeze. After successful |
15 |
installation, you will have the following services running: |
16 |
|
17 |
* Identity Management (Astakos) |
18 |
* Object Storage Service (Pithos) |
19 |
* Compute Service (Cyclades) |
20 |
* Image Service (part of Cyclades) |
21 |
* Network Service (part of Cyclades) |
22 |
|
23 |
and a single unified Web UI to manage them all. |
24 |
|
25 |
The Volume Storage Service (Archipelago) and the Billing Service (Aquarium) are |
26 |
not released yet. |
27 |
|
28 |
If you just want to install the Object Storage Service (Pithos), follow the guide |
29 |
and just stop after the "Testing of Pithos" section. |
30 |
|
31 |
Building a dev environment |
32 |
-------------------------- |
33 |
|
34 |
virtualenv |
35 |
********** |
36 |
|
37 |
The easiest method to deploy a development environment is using |
38 |
:command:`virtualenv`. Alternatively, you can use your system's package manager |
39 |
to install any dependencies of synnefo components (e.g. Macports has them all). |
40 |
|
41 |
.. code-block:: console |
42 |
|
43 |
$ virtualenv ~/synnefo-env |
44 |
$ source ~/synnefo-env/bin/activate |
45 |
(synnefo-env)$ |
46 |
|
47 |
Virtualenv creates an isolated python environment to the path you pass as the |
48 |
first argument of the command. That means that all packages you install using |
49 |
:command:`pip` or :command:`easy_install` will be placed in |
50 |
``ENV/lib/pythonX.X/site-packages`` and their console scripts in ``ENV/bin/``. |
51 |
|
52 |
This allows you to develop against multiple versions of packages that your |
53 |
software depends on without messing with system python packages that may be |
54 |
needed in specific versions for other software you have installed on your |
55 |
system. |
56 |
|
57 |
* It is also recommended to install development helpers: |
58 |
|
59 |
.. code-block:: console |
60 |
|
61 |
(synnefo-env)$ pip install django_extensions fabric>=1.3 |
62 |
|
63 |
* Create a custom settings directory for :ref:`snf-common <snf-common>` and set |
64 |
the ``SYNNEFO_SETTINGS_DIR`` environment variable to use development-specific |
65 |
file:`*.conf` files inside this directory. |
66 |
|
67 |
(synnefo-env)$ mkdir ~/synnefo-settings-dir |
68 |
(synnefo-env)$ export SYNNEFO_SETTINGS_DIR=~/synnefo-settings-dir |
69 |
|
70 |
Insert your custom settings in a file such as :file:`$SYNNEFO_SETTINGS_DIR/99-local.conf`: |
71 |
|
72 |
.. code-block:: python |
73 |
|
74 |
# uncomment this if have django-extensions installed (pip install django_extensions) |
75 |
#INSTALLED_APPS = list(INSTALLED_APPS) + ['django_extensions'] |
76 |
|
77 |
DEV_PATH = os.path.abspath(os.path.dirname(__file__)) |
78 |
DATABASES['default']['NAME'] = os.path.join(DEV_PATH, "synnefo.sqlite") |
79 |
|
80 |
# development rabitmq configuration |
81 |
RABBIT_HOST = "<RabbitMQ_host>" |
82 |
RABBIT_USERNAME = "<RabbitMQ_username>" |
83 |
RABBIT_PASSWORD = "<RabbitMQ_password>" |
84 |
RABBIT_VHOST = "/" |
85 |
|
86 |
# development ganeti settings |
87 |
GANETI_MASTER_IP = "<Ganeti_master_IP>" |
88 |
GANETI_CLUSTER_INFO = (GANETI_MASTER_IP, 5080, "<username>", "<password>") |
89 |
GANETI_CREATEINSTANCE_KWARGS['disk_template'] = 'plain' |
90 |
|
91 |
# This prefix gets used when determining the instance names |
92 |
# of Synnefo VMs at the Ganeti backend. |
93 |
# The dash must always appear in the name! |
94 |
BACKEND_PREFIX_ID = "<your_commit_name>-" |
95 |
|
96 |
IGNORE_FLAVOR_DISK_SIZES = True |
97 |
|
98 |
# do not actually send emails |
99 |
# save them as files in /tmp/synnefo-mails |
100 |
EMAIL_BACKEND = 'django.core.mail.backends.filebased.EmailBackend' |
101 |
EMAIL_FILE_PATH = '/tmp/synnefo-mails' |
102 |
|
103 |
# for UI developers |
104 |
UI_HANDLE_WINDOW_EXCEPTIONS = False |
105 |
|
106 |
# allow login using /?test url |
107 |
BYPASS_AUTHENTICATION = True |
108 |
|
109 |
synnefo source |
110 |
************** |
111 |
|
112 |
* Clone the repository of the synnefo software components you wish |
113 |
to work on, e.g.: |
114 |
|
115 |
.. code-block:: console |
116 |
|
117 |
(synnefo-env)$ git clone https://code.grnet.gr/git/synnefo synnefo |
118 |
(synnefo-env)$ git clone https://code.grnet.gr/git/pithos pithos |
119 |
|
120 |
* Install the software components you wish to work on inside the |
121 |
virtualenv, in development mode: |
122 |
|
123 |
.. code-block:: console |
124 |
|
125 |
(synnefo-env)$ cd snf-cyclades-app |
126 |
(synnefo-env)$ python setup.py develop -N |
127 |
|
128 |
* Initialize database: |
129 |
|
130 |
.. code-block:: console |
131 |
|
132 |
(synnefo-env)$ snf-manage syndb |
133 |
(synnefo-env)$ snf-manage migrate |
134 |
(synnefo-env)$ snf-manage loaddata users flavors images |
135 |
|
136 |
Development tips |
137 |
**************** |
138 |
|
139 |
* Running a development web server: |
140 |
|
141 |
.. code-block:: console |
142 |
|
143 |
(synnefo-env)$ snf-manage runserver |
144 |
|
145 |
or, if you have the ``django_extensions`` and ``werkzeug`` packages installed: |
146 |
|
147 |
.. code-block:: console |
148 |
|
149 |
(synnefo-env)$ snf-manage runserver_plus |
150 |
|
151 |
* Opening a python console with the synnefo environment initialized: |
152 |
|
153 |
.. code-block:: console |
154 |
|
155 |
(synnefo-env)$ snf-manage shell |
156 |
|
157 |
or, with the django_extensions package installed: |
158 |
|
159 |
.. code-block:: console |
160 |
|
161 |
(synnefo-env)$ snf-manage shell_plus |
162 |
|
163 |
|
164 |
South Database Migrations |
165 |
------------------------- |
166 |
|
167 |
.. _cyclades-dev-initialmigration: |
168 |
|
169 |
Initial Migration |
170 |
***************** |
171 |
|
172 |
To initialize south migrations in your database the following commands must be |
173 |
executed: |
174 |
|
175 |
.. code-block:: console |
176 |
|
177 |
$ snf-manage syncdb --all # Create / update the database with the south tables |
178 |
$ snf-manage migrate --fake # Perform migration in the database |
179 |
|
180 |
|
181 |
Note that ``--all`` and ``--fake`` arguments are only needed when you are |
182 |
initializing your database. If you want to migrate a previously create databse |
183 |
to the latest db scheme just run the same commands without those arguments. |
184 |
|
185 |
If you are trying to migrate a database that already contains the changes that |
186 |
applied from a specific migration script, ``south`` will probably notify you for |
187 |
inconsistent db scheme, a workaround for that issue is to use ``--fake`` option |
188 |
for a specific migration. |
189 |
|
190 |
For example: |
191 |
|
192 |
|
193 |
.. code-block:: console |
194 |
|
195 |
$ snf-manage migrate db 0001 --fake |
196 |
|
197 |
To be sure that all migrations are applied use: |
198 |
|
199 |
.. code-block:: console |
200 |
|
201 |
$ snf-manage migrate db --list |
202 |
|
203 |
All starred migrations are applied. |
204 |
|
205 |
Schema migrations |
206 |
***************** |
207 |
|
208 |
Do not use the syncdb management command. It can only be used the first time |
209 |
and/or if you drop the database and must recreate it from scratch. See |
210 |
:ref:`cyclades-dev-initialmigration`. |
211 |
|
212 |
|
213 |
Every time you make changes to the database and data migration is not required |
214 |
(WARNING: always perform this with extreme care): |
215 |
|
216 |
.. code-block:: console |
217 |
|
218 |
$ snf-manage schemamigration db --auto |
219 |
|
220 |
The above will create the migration script. Now this must be applied to the live |
221 |
database: |
222 |
|
223 |
.. code-block:: console |
224 |
|
225 |
$ snf-manage migrate db |
226 |
|
227 |
Consider this example (adding a field to the ``SynnefoUser`` model): |
228 |
|
229 |
.. code-block:: console |
230 |
|
231 |
$ ./bin/python manage.py schemamigration db --auto |
232 |
+ Added field new_south_test_field on db.SynnefoUser |
233 |
|
234 |
Created 0002_auto__add_field_synnefouser_new_south_test_field.py. |
235 |
|
236 |
You can now apply this migration with: |
237 |
|
238 |
.. code-block:: console |
239 |
|
240 |
$ ./manage.py migrate db |
241 |
Running migrations for db: |
242 |
- Migrating forwards to 0002_auto__add_field_synnefouser_new_south_test_field. |
243 |
> db:0002_auto__add_field_synnefouser_new_south_test_field |
244 |
- Loading initial data for db. |
245 |
|
246 |
Installing json fixture 'initial_data' from '/home/bkarak/devel/synnefo/../synnefo/db/fixtures'. |
247 |
Installed 1 object(s) from 1 fixture(s) |
248 |
|
249 |
South needs some extra definitions to the model to preserve and migrate the |
250 |
existing data, for example, if we add a field in a model, we should declare its |
251 |
default value. If not, South will propably fail, after indicating the error: |
252 |
|
253 |
.. code-block:: console |
254 |
|
255 |
$ ./bin/python manage.py schemamigration db --auto |
256 |
? The field 'SynnefoUser.new_south_field_2' does not have a default specified, yet is NOT NULL. |
257 |
? Since you are adding or removing this field, you MUST specify a default |
258 |
? value to use for existing rows. Would you like to: |
259 |
? 1. Quit now, and add a default to the field in models.py |
260 |
? 2. Specify a one-off value to use for existing columns now |
261 |
? Please select a choice: 1 |
262 |
|
263 |
Data migrations |
264 |
*************** |
265 |
|
266 |
To do data migration as well, for example rename a field, use the |
267 |
``datamigration`` management command. |
268 |
|
269 |
In contrast with ``schemamigration``, to perform complex data migration, we |
270 |
must write the script manually. The process is the following: |
271 |
|
272 |
1. Introduce the changes in the code and fixtures (initial data). |
273 |
2. Execute: |
274 |
|
275 |
.. code-block:: console |
276 |
|
277 |
$ snf-manage datamigration <migration_name_here> |
278 |
|
279 |
For example: |
280 |
|
281 |
.. code-block:: console |
282 |
|
283 |
$ ./bin/python manage.py datamigration db rename_credit_wallet |
284 |
Created 0003_rename_credit_wallet.py. |
285 |
|
286 |
3. Edit the generated script. It contains two methods, ``forwards`` and |
287 |
``backwards``. |
288 |
|
289 |
For database operations (column additions, alter tables etc), use the |
290 |
South database API (http://south.aeracode.org/docs/databaseapi.html). |
291 |
|
292 |
To access the data, use the database reference (``orm``) provided as |
293 |
parameter in ``forwards``, ``backwards`` method declarations in the |
294 |
migration script. For example: |
295 |
|
296 |
.. code-block:: python |
297 |
|
298 |
class Migration(DataMigration): |
299 |
|
300 |
def forwards(self, orm): |
301 |
orm.SynnefoUser.objects.all() |
302 |
|
303 |
4. To migrate the database to the latest version, run: |
304 |
|
305 |
.. code-block:: console |
306 |
|
307 |
$ snf-manage migrate db |
308 |
|
309 |
To see which migrations are applied: |
310 |
|
311 |
.. code-block:: console |
312 |
|
313 |
$ snf-manage migrate db --list |
314 |
|
315 |
db |
316 |
(*) 0001_initial |
317 |
(*) 0002_auto__add_field_synnefouser_new_south_test_field |
318 |
(*) 0003_rename_credit_wallet |
319 |
|
320 |
.. seealso:: |
321 |
More information and more thorough examples can be found in the South web site, |
322 |
http://south.aeracode.org/ |
323 |
|
324 |
Test coverage |
325 |
------------- |
326 |
|
327 |
.. warning:: This section may be out of date. |
328 |
|
329 |
In order to get code coverage reports you need to install django-test-coverage |
330 |
|
331 |
.. code-block:: console |
332 |
|
333 |
$ pip install django-test-coverage |
334 |
|
335 |
Then configure the test runner inside Django settings: |
336 |
|
337 |
.. code-block:: python |
338 |
|
339 |
TEST_RUNNER = 'django-test-coverage.runner.run_tests' |
340 |
|
341 |
|
342 |
Internationalization |
343 |
-------------------- |
344 |
|
345 |
This section describes how to translate static strings in Django projects: |
346 |
|
347 |
0. From our project's base, we add directory locale |
348 |
|
349 |
.. code-block:: console |
350 |
|
351 |
$ mkdir locale |
352 |
|
353 |
then we add on the settings.py the language code e.g., |
354 |
|
355 |
.. code-block:: python |
356 |
|
357 |
LANGUAGES = ( |
358 |
('el', u'Greek'), |
359 |
('en', u'English'),) |
360 |
|
361 |
1. For each language we want to add, we run ``makemessages`` from the project's |
362 |
base: |
363 |
|
364 |
.. code-block:: python |
365 |
|
366 |
$ ./bin/django-admin.py makemessages -l el -e html,txt,py |
367 |
(./bin/django-admin.py makemessages -l el -e html,txt,py --ignore=lib/\*) |
368 |
|
369 |
This will add the Greek language, and we specify that :file:`*.html`, |
370 |
:file:`*.txt` and :file:`*.py` files contain translatable strings |
371 |
|
372 |
2. We translate our strings: |
373 |
|
374 |
On :file:`.py` files, e.g., :file:`views.py`, first import ``gettext``: |
375 |
|
376 |
.. code-block:: python |
377 |
|
378 |
from django.utils.translation import gettext_lazy as _ |
379 |
|
380 |
Then every ``string`` to be translated becomes: ``_('string')`` |
381 |
e.g.: |
382 |
|
383 |
.. code-block:: python |
384 |
|
385 |
help_text=_("letters and numbers only")) |
386 |
'title': _('Ubuntu 10.10 server 64bit'), |
387 |
|
388 |
On django templates (``html`` files), on the beggining of the file we add |
389 |
``{% load i18n %}`` then rewrite every string that needs to be translated, |
390 |
as ``{% trans "string" %}``. For example: ``{% trans "Home" %}`` |
391 |
|
392 |
3. When all strings have been translated, run: |
393 |
|
394 |
.. code-block:: console |
395 |
|
396 |
$ django-admin.py makemessages -l el -e html,txt,py |
397 |
|
398 |
processing language ``el``. This creates (or updates) the :file:`po` file |
399 |
for the Greek language. We run this command each time we add new strings to |
400 |
be translated. After that, we can translate our strings in the :file:`po` |
401 |
file (:file:`locale/el/LC_MESSAGES/django.po`) |
402 |
|
403 |
4. When the :file:`po` file is ready, run |
404 |
|
405 |
.. code-block:: console |
406 |
|
407 |
$ ./bin/django-admin.py compilemessages |
408 |
|
409 |
This compiles the ``po`` files to ``mo``. Our strings will appear translated |
410 |
once we change the language (e.g., from a dropdown menu in the page) |
411 |
|
412 |
.. seealso:: |
413 |
http://docs.djangoproject.com/en/dev/topics/i18n/internationalization/ |
414 |
|
415 |
|
416 |
Building source packages |
417 |
------------------------ |
418 |
|
419 |
.. warning:: This section may be out of date. |
420 |
|
421 |
To create a python package from the Synnefo source code run |
422 |
|
423 |
.. code-block:: bash |
424 |
|
425 |
$ cd snf-cyclades-app |
426 |
$ python setup.py sdist |
427 |
|
428 |
this command will create a ``tar.gz`` python source package inside ``dist`` directory. |
429 |
|
430 |
|
431 |
Building documentation |
432 |
---------------------- |
433 |
|
434 |
Make sure you have ``sphinx`` installed. |
435 |
|
436 |
.. code-block:: bash |
437 |
|
438 |
$ cd snf-cyclades-app/docs |
439 |
$ make html |
440 |
|
441 |
.. note:: |
442 |
|
443 |
The theme define in the Sphinx configuration file ``conf.py`` is ``nature``, |
444 |
not available in the version of Sphinx shipped with Debian Squeeze. Replace |
445 |
it with ``default`` to build with a Squeeze-provided Sphinx. |
446 |
|
447 |
html files are generated in the ``snf-cyclades-app/docs/_build/html`` directory. |
448 |
|
449 |
|
450 |
Continuous integration with Jenkins |
451 |
----------------------------------- |
452 |
.. warning:: This section may be out of date. |
453 |
|
454 |
Preparing a GIT mirror |
455 |
********************** |
456 |
|
457 |
Jenkins cannot currently work with Git over encrypted HTTP. To solve this |
458 |
problem we currently mirror the central Git repository locally on the jenkins |
459 |
installation machine. To setup such a mirror do the following: |
460 |
|
461 |
edit .netrc:: |
462 |
|
463 |
machine code.grnet.gr |
464 |
login accountname |
465 |
password accountpasswd |
466 |
|
467 |
Create the mirror:: |
468 |
|
469 |
git clone --mirror https://code.grnet.gr/git/synnefo synnefo |
470 |
|
471 |
Setup cron to pull from the mirror periodically. Ideally, Git mirror updates |
472 |
should run just before Jenkins jobs check the mirror for changes:: |
473 |
|
474 |
4,14,24,34,44,54 * * * * cd /path/to/mirror && git fetch && git remote prune origin |
475 |
|
476 |
Jenkins setup |
477 |
************* |
478 |
|
479 |
The following instructions will setup Jenkins to run synnefo tests with the |
480 |
SQLite database. To run the tests on MySQL and/or Postgres, step 5 must be |
481 |
replicated. Also, the correct configuration file must be copied (line 6 of the |
482 |
build script). |
483 |
|
484 |
1. Install and start Jenkins. On Debian Squeeze: |
485 |
|
486 |
wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | apt-key add - |
487 |
echo "deb http://pkg.jenkins-ci.org/debian binary/" >>/etc/apt/sources.list |
488 |
echo "deb http://ppa.launchpad.net/chris-lea/zeromq/ubuntu lucid main" >> /etc/apt/sources.list |
489 |
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C7917B12 |
490 |
sudo apt-get update |
491 |
sudo apt-get install jenkins |
492 |
|
493 |
Also install the following packages: |
494 |
|
495 |
apt-get install python-virtualenv libcurl3-gnutls libcurl3-gnutls-dev |
496 |
uuid-dev libmysqlclient-dev libpq-dev libsqlite-dev |
497 |
python-dev libzmq-dev |
498 |
|
499 |
2. After Jenkins starts, go to |
500 |
|
501 |
http://$HOST:8080/pluginManager/ |
502 |
|
503 |
and install the following plug-ins at |
504 |
|
505 |
-Jenkins Cobertura Plugin |
506 |
-Jenkins Email Extension Plugin |
507 |
-Jenkins GIT plugin |
508 |
-Jenkins SLOCCount Plug-in |
509 |
-Hudson/Jenkins Violations plugin |
510 |
|
511 |
3. Configure the Jenkins user's Git details: |
512 |
su jenkins |
513 |
git config --global user.email "buildbot@lists.grnet.gr" |
514 |
git config --global user.name "Buildbot" |
515 |
|
516 |
4. Make sure that all system-level dependencies specified in README.develop |
517 |
are correctly installed |
518 |
|
519 |
5. Create a new "free-style software" job and set the following values:: |
520 |
|
521 |
Project name: synnefo |
522 |
Source Code Management: Git |
523 |
URL of repository: Jenkins Git does not support HTTPS for checking out directly |
524 |
from the repository. The temporary solution is to checkout |
525 |
with a cron script in a directory and set the checkout path |
526 |
in this field |
527 |
Branches to build: master and perhaps others |
528 |
Git->Advanced->Local subdirectory for repo (optional): synnefo |
529 |
Git->Advanced->Prune remote branches before build: check |
530 |
Repository browser: redmineweb, |
531 |
URL: https://code.grnet.gr/projects/synnefo/repository/ |
532 |
Build Triggers->Poll SCM: check |
533 |
Schedule: # every five minutes |
534 |
0,5,10,15,20,25,30,35,40,45,50,55 * * * * |
535 |
|
536 |
Build -> Add build step-> Execute shell |
537 |
|
538 |
Command:: |
539 |
|
540 |
#!/bin/bash -ex |
541 |
cd synnefo |
542 |
mkdir -p reports |
543 |
/usr/bin/sloccount --duplicates --wide --details api util ui logic auth > reports/sloccount.sc |
544 |
cp conf/ci/manage.py . |
545 |
if [ ! -e requirements.pip ]; then cp conf/ci/pip-1.2.conf requirements.pip; fi |
546 |
cat settings.py.dist conf/ci/settings.py.sqlite > settings.py |
547 |
python manage.py update_ve |
548 |
python manage.py hudson api db logic |
549 |
|
550 |
Post-build Actions->Publish JUnit test result report: check |
551 |
Test report XMLs: synnefo/reports/TEST-*.xml |
552 |
|
553 |
Post-build Actions->Publish Cobertura Coverage Report: check |
554 |
Cobertura xml report pattern: synnefo/reports/coverage.xml |
555 |
|
556 |
Post-build Actions->Report Violations: check |
557 |
pylint[XML filename pattern]: synnefo/reports/pylint.report |
558 |
|
559 |
Post-build Actions->Publish SLOCCount analysis results |
560 |
SLOCCount reports: synnefo/reports/sloccount.sc |
561 |
(also, remember to install sloccount at /usr/bin) |
562 |
|
563 |
.. seealso:: |
564 |
http://sites.google.com/site/kmmbvnr/home/django-hudson-tutorial |