root / docs / quick-install-intgrt-guide.rst @ d2a9f85f
History | View | Annotate | Download (17.4 kB)
1 |
.. _quick-install-intgrt-guide: |
---|---|
2 |
|
3 |
Integrator's Quick Installation Guide |
4 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
5 |
|
6 |
This is the Integrator's quick installation guide. |
7 |
|
8 |
It is intended for developers, wishing to implement new functionality |
9 |
inside Synnefo. It assumes thorough familiarity with the |
10 |
:ref:`Synnefo Administrator's Guide <admin-guide>`. |
11 |
|
12 |
It describes how to install the whole synnefo stack on two (2) physical nodes, |
13 |
with minimum configuration. It installs synnefo in a ``virtualenv`` using ``pip |
14 |
install``, and assumes the nodes run Debian Squeeze. After successful |
15 |
installation, you will have the following services running: |
16 |
|
17 |
* Identity Management (Astakos) |
18 |
* Object Storage Service (Pithos+) |
19 |
* Compute Service (Cyclades) |
20 |
* Image Registry Service (Plankton) |
21 |
|
22 |
and a single unified Web UI to manage them all. |
23 |
|
24 |
The Volume Storage Service (Archipelago) and the Billing Service (Aquarium) are |
25 |
not released yet. |
26 |
|
27 |
If you just want to install the Object Storage Service (Pithos+), follow the guide |
28 |
and just stop after the "Testing of Pithos+" section. |
29 |
|
30 |
Building a dev environment |
31 |
-------------------------- |
32 |
|
33 |
virtualenv |
34 |
********** |
35 |
|
36 |
The easiest method to deploy a development environment is using |
37 |
:command:`virtualenv`. Alternatively, you can use your system's package manager |
38 |
to install any dependencies of synnefo components (e.g. Macports has them all). |
39 |
|
40 |
.. code-block:: console |
41 |
|
42 |
$ virtualenv ~/synnefo-env |
43 |
$ source ~/synnefo-env/bin/activate |
44 |
(synnefo-env)$ |
45 |
|
46 |
Virtualenv creates an isolated python environment to the path you pass as the |
47 |
first argument of the command. That means that all packages you install using |
48 |
:command:`pip` or :command:`easy_install` will be placed in |
49 |
``ENV/lib/pythonX.X/site-packages`` and their console scripts in ``ENV/bin/``. |
50 |
|
51 |
This allows you to develop against multiple versions of packages that your |
52 |
software depends on without messing with system python packages that may be |
53 |
needed in specific versions for other software you have installed on your |
54 |
system. |
55 |
|
56 |
* It is also recommended to install development helpers: |
57 |
|
58 |
.. code-block:: console |
59 |
|
60 |
(synnefo-env)$ pip install django_extensions fabric>=1.3 |
61 |
|
62 |
* Create a custom settings directory for :ref:`snf-common <snf-common>` and set |
63 |
the ``SYNNEFO_SETTINGS_DIR`` environment variable to use development-specific |
64 |
file:`*.conf` files inside this directory. |
65 |
|
66 |
(synnefo-env)$ mkdir ~/synnefo-settings-dir |
67 |
(synnefo-env)$ export SYNNEFO_SETTINGS_DIR=~/synnefo-settings-dir |
68 |
|
69 |
Insert your custom settings in a file such as :file:`$SYNNEFO_SETTINGS_DIR/99-local.conf`: |
70 |
|
71 |
.. code-block:: python |
72 |
|
73 |
# uncomment this if have django-extensions installed (pip install django_extensions) |
74 |
#INSTALLED_APPS = list(INSTALLED_APPS) + ['django_extensions'] |
75 |
|
76 |
DEV_PATH = os.path.abspath(os.path.dirname(__file__)) |
77 |
DATABASES['default']['NAME'] = os.path.join(DEV_PATH, "synnefo.sqlite") |
78 |
|
79 |
# development rabitmq configuration |
80 |
RABBIT_HOST = "<RabbitMQ_host>" |
81 |
RABBIT_USERNAME = "<RabbitMQ_username>" |
82 |
RABBIT_PASSWORD = "<RabbitMQ_password>" |
83 |
RABBIT_VHOST = "/" |
84 |
|
85 |
# development ganeti settings |
86 |
GANETI_MASTER_IP = "<Ganeti_master_IP>" |
87 |
GANETI_CLUSTER_INFO = (GANETI_MASTER_IP, 5080, "<username>", "<password>") |
88 |
GANETI_CREATEINSTANCE_KWARGS['disk_template'] = 'plain' |
89 |
|
90 |
# This prefix gets used when determining the instance names |
91 |
# of Synnefo VMs at the Ganeti backend. |
92 |
# The dash must always appear in the name! |
93 |
BACKEND_PREFIX_ID = "<your_commit_name>-" |
94 |
|
95 |
IGNORE_FLAVOR_DISK_SIZES = True |
96 |
|
97 |
# do not actually send emails |
98 |
# save them as files in /tmp/synnefo-mails |
99 |
EMAIL_BACKEND = 'django.core.mail.backends.filebased.EmailBackend' |
100 |
EMAIL_FILE_PATH = '/tmp/synnefo-mails' |
101 |
|
102 |
# for UI developers |
103 |
UI_HANDLE_WINDOW_EXCEPTIONS = False |
104 |
|
105 |
# allow login using /?test url |
106 |
BYPASS_AUTHENTICATION = True |
107 |
|
108 |
synnefo source |
109 |
************** |
110 |
|
111 |
* Clone the repository of the synnefo software components you wish |
112 |
to work on, e.g.: |
113 |
|
114 |
.. code-block:: console |
115 |
|
116 |
(synnefo-env)$ git clone https://code.grnet.gr/git/synnefo synnefo |
117 |
(synnefo-env)$ git clone https://code.grnet.gr/git/pithos pithos |
118 |
|
119 |
* Install the software components you wish to work on inside the |
120 |
virtualenv, in development mode: |
121 |
|
122 |
.. code-block:: console |
123 |
|
124 |
(synnefo-env)$ cd snf-cyclades-app |
125 |
(synnefo-env)$ python setup.py develop -N |
126 |
|
127 |
* Initialize database: |
128 |
|
129 |
.. code-block:: console |
130 |
|
131 |
(synnefo-env)$ snf-manage syndb |
132 |
(synnefo-env)$ snf-manage migrate |
133 |
(synnefo-env)$ snf-manage loaddata users flavors images |
134 |
|
135 |
Development tips |
136 |
**************** |
137 |
|
138 |
* Running a development web server: |
139 |
|
140 |
.. code-block:: console |
141 |
|
142 |
(synnefo-env)$ snf-manage runserver |
143 |
|
144 |
or, if you have the ``django_extensions`` and ``werkzeug`` packages installed: |
145 |
|
146 |
.. code-block:: console |
147 |
|
148 |
(synnefo-env)$ snf-manage runserver_plus |
149 |
|
150 |
* Opening a python console with the synnefo environment initialized: |
151 |
|
152 |
.. code-block:: console |
153 |
|
154 |
(synnefo-env)$ snf-manage shell |
155 |
|
156 |
or, with the django_extensions package installed: |
157 |
|
158 |
.. code-block:: console |
159 |
|
160 |
(synnefo-env)$ snf-manage shell_plus |
161 |
|
162 |
|
163 |
South Database Migrations |
164 |
------------------------- |
165 |
|
166 |
.. _cyclades-dev-initialmigration: |
167 |
|
168 |
Initial Migration |
169 |
***************** |
170 |
|
171 |
To initialize south migrations in your database the following commands must be |
172 |
executed: |
173 |
|
174 |
.. code-block:: console |
175 |
|
176 |
$ snf-manage syncdb --all # Create / update the database with the south tables |
177 |
$ snf-manage migrate --fake # Perform migration in the database |
178 |
|
179 |
|
180 |
Note that ``--all`` and ``--fake`` arguments are only needed when you are |
181 |
initializing your database. If you want to migrate a previously create databse |
182 |
to the latest db scheme just run the same commands without those arguments. |
183 |
|
184 |
If you are trying to migrate a database that already contains the changes that |
185 |
applied from a specific migration script, ``south`` will probably notify you for |
186 |
inconsistent db scheme, a workaround for that issue is to use ``--fake`` option |
187 |
for a specific migration. |
188 |
|
189 |
For example: |
190 |
|
191 |
|
192 |
.. code-block:: console |
193 |
|
194 |
$ snf-manage migrate db 0001 --fake |
195 |
|
196 |
To be sure that all migrations are applied use: |
197 |
|
198 |
.. code-block:: console |
199 |
|
200 |
$ snf-manage migrate db --list |
201 |
|
202 |
All starred migrations are applied. |
203 |
|
204 |
Schema migrations |
205 |
***************** |
206 |
|
207 |
Do not use the syncdb management command. It can only be used the first time |
208 |
and/or if you drop the database and must recreate it from scratch. See |
209 |
:ref:`cyclades-dev-initialmigration`. |
210 |
|
211 |
|
212 |
Every time you make changes to the database and data migration is not required |
213 |
(WARNING: always perform this with extreme care): |
214 |
|
215 |
.. code-block:: console |
216 |
|
217 |
$ snf-manage schemamigration db --auto |
218 |
|
219 |
The above will create the migration script. Now this must be applied to the live |
220 |
database: |
221 |
|
222 |
.. code-block:: console |
223 |
|
224 |
$ snf-manage migrate db |
225 |
|
226 |
Consider this example (adding a field to the ``SynnefoUser`` model): |
227 |
|
228 |
.. code-block:: console |
229 |
|
230 |
$ ./bin/python manage.py schemamigration db --auto |
231 |
+ Added field new_south_test_field on db.SynnefoUser |
232 |
|
233 |
Created 0002_auto__add_field_synnefouser_new_south_test_field.py. |
234 |
|
235 |
You can now apply this migration with: |
236 |
|
237 |
.. code-block:: console |
238 |
|
239 |
$ ./manage.py migrate db |
240 |
Running migrations for db: |
241 |
- Migrating forwards to 0002_auto__add_field_synnefouser_new_south_test_field. |
242 |
> db:0002_auto__add_field_synnefouser_new_south_test_field |
243 |
- Loading initial data for db. |
244 |
|
245 |
Installing json fixture 'initial_data' from '/home/bkarak/devel/synnefo/../synnefo/db/fixtures'. |
246 |
Installed 1 object(s) from 1 fixture(s) |
247 |
|
248 |
South needs some extra definitions to the model to preserve and migrate the |
249 |
existing data, for example, if we add a field in a model, we should declare its |
250 |
default value. If not, South will propably fail, after indicating the error: |
251 |
|
252 |
.. code-block:: console |
253 |
|
254 |
$ ./bin/python manage.py schemamigration db --auto |
255 |
? The field 'SynnefoUser.new_south_field_2' does not have a default specified, yet is NOT NULL. |
256 |
? Since you are adding or removing this field, you MUST specify a default |
257 |
? value to use for existing rows. Would you like to: |
258 |
? 1. Quit now, and add a default to the field in models.py |
259 |
? 2. Specify a one-off value to use for existing columns now |
260 |
? Please select a choice: 1 |
261 |
|
262 |
Data migrations |
263 |
*************** |
264 |
|
265 |
To do data migration as well, for example rename a field, use the |
266 |
``datamigration`` management command. |
267 |
|
268 |
In contrast with ``schemamigration``, to perform complex data migration, we |
269 |
must write the script manually. The process is the following: |
270 |
|
271 |
1. Introduce the changes in the code and fixtures (initial data). |
272 |
2. Execute: |
273 |
|
274 |
.. code-block:: console |
275 |
|
276 |
$ snf-manage datamigration <migration_name_here> |
277 |
|
278 |
For example: |
279 |
|
280 |
.. code-block:: console |
281 |
|
282 |
$ ./bin/python manage.py datamigration db rename_credit_wallet |
283 |
Created 0003_rename_credit_wallet.py. |
284 |
|
285 |
3. Edit the generated script. It contains two methods, ``forwards`` and |
286 |
``backwards``. |
287 |
|
288 |
For database operations (column additions, alter tables etc), use the |
289 |
South database API (http://south.aeracode.org/docs/databaseapi.html). |
290 |
|
291 |
To access the data, use the database reference (``orm``) provided as |
292 |
parameter in ``forwards``, ``backwards`` method declarations in the |
293 |
migration script. For example: |
294 |
|
295 |
.. code-block:: python |
296 |
|
297 |
class Migration(DataMigration): |
298 |
|
299 |
def forwards(self, orm): |
300 |
orm.SynnefoUser.objects.all() |
301 |
|
302 |
4. To migrate the database to the latest version, run: |
303 |
|
304 |
.. code-block:: console |
305 |
|
306 |
$ snf-manage migrate db |
307 |
|
308 |
To see which migrations are applied: |
309 |
|
310 |
.. code-block:: console |
311 |
|
312 |
$ snf-manage migrate db --list |
313 |
|
314 |
db |
315 |
(*) 0001_initial |
316 |
(*) 0002_auto__add_field_synnefouser_new_south_test_field |
317 |
(*) 0003_rename_credit_wallet |
318 |
|
319 |
.. seealso:: |
320 |
More information and more thorough examples can be found in the South web site, |
321 |
http://south.aeracode.org/ |
322 |
|
323 |
Test coverage |
324 |
------------- |
325 |
|
326 |
.. warning:: This section may be out of date. |
327 |
|
328 |
In order to get code coverage reports you need to install django-test-coverage |
329 |
|
330 |
.. code-block:: console |
331 |
|
332 |
$ pip install django-test-coverage |
333 |
|
334 |
Then configure the test runner inside Django settings: |
335 |
|
336 |
.. code-block:: python |
337 |
|
338 |
TEST_RUNNER = 'django-test-coverage.runner.run_tests' |
339 |
|
340 |
|
341 |
Internationalization |
342 |
-------------------- |
343 |
|
344 |
This section describes how to translate static strings in Django projects: |
345 |
|
346 |
0. From our project's base, we add directory locale |
347 |
|
348 |
.. code-block:: console |
349 |
|
350 |
$ mkdir locale |
351 |
|
352 |
then we add on the settings.py the language code e.g., |
353 |
|
354 |
.. code-block:: python |
355 |
|
356 |
LANGUAGES = ( |
357 |
('el', u'Greek'), |
358 |
('en', u'English'),) |
359 |
|
360 |
1. For each language we want to add, we run ``makemessages`` from the project's |
361 |
base: |
362 |
|
363 |
.. code-block:: python |
364 |
|
365 |
$ ./bin/django-admin.py makemessages -l el -e html,txt,py |
366 |
(./bin/django-admin.py makemessages -l el -e html,txt,py --ignore=lib/\*) |
367 |
|
368 |
This will add the Greek language, and we specify that :file:`*.html`, |
369 |
:file:`*.txt` and :file:`*.py` files contain translatable strings |
370 |
|
371 |
2. We translate our strings: |
372 |
|
373 |
On :file:`.py` files, e.g., :file:`views.py`, first import ``gettext``: |
374 |
|
375 |
.. code-block:: python |
376 |
|
377 |
from django.utils.translation import gettext_lazy as _ |
378 |
|
379 |
Then every ``string`` to be translated becomes: ``_('string')`` |
380 |
e.g.: |
381 |
|
382 |
.. code-block:: python |
383 |
|
384 |
help_text=_("letters and numbers only")) |
385 |
'title': _('Ubuntu 10.10 server 64bit'), |
386 |
|
387 |
On django templates (``html`` files), on the beggining of the file we add |
388 |
``{% load i18n %}`` then rewrite every string that needs to be translated, |
389 |
as ``{% trans "string" %}``. For example: ``{% trans "Home" %}`` |
390 |
|
391 |
3. When all strings have been translated, run: |
392 |
|
393 |
.. code-block:: console |
394 |
|
395 |
$ django-admin.py makemessages -l el -e html,txt,py |
396 |
|
397 |
processing language ``el``. This creates (or updates) the :file:`po` file |
398 |
for the Greek language. We run this command each time we add new strings to |
399 |
be translated. After that, we can translate our strings in the :file:`po` |
400 |
file (:file:`locale/el/LC_MESSAGES/django.po`) |
401 |
|
402 |
4. When the :file:`po` file is ready, run |
403 |
|
404 |
.. code-block:: console |
405 |
|
406 |
$ ./bin/django-admin.py compilemessages |
407 |
|
408 |
This compiles the ``po`` files to ``mo``. Our strings will appear translated |
409 |
once we change the language (e.g., from a dropdown menu in the page) |
410 |
|
411 |
.. seealso:: |
412 |
http://docs.djangoproject.com/en/dev/topics/i18n/internationalization/ |
413 |
|
414 |
|
415 |
Building source packages |
416 |
------------------------ |
417 |
|
418 |
.. warning:: This section may be out of date. |
419 |
|
420 |
To create a python package from the Synnefo source code run |
421 |
|
422 |
.. code-block:: bash |
423 |
|
424 |
$ cd snf-cyclades-app |
425 |
$ python setup.py sdist |
426 |
|
427 |
this command will create a ``tar.gz`` python source package inside ``dist`` directory. |
428 |
|
429 |
|
430 |
Building documentation |
431 |
---------------------- |
432 |
|
433 |
Make sure you have ``sphinx`` installed. |
434 |
|
435 |
.. code-block:: bash |
436 |
|
437 |
$ cd snf-cyclades-app/docs |
438 |
$ make html |
439 |
|
440 |
.. note:: |
441 |
|
442 |
The theme define in the Sphinx configuration file ``conf.py`` is ``nature``, |
443 |
not available in the version of Sphinx shipped with Debian Squeeze. Replace |
444 |
it with ``default`` to build with a Squeeze-provided Sphinx. |
445 |
|
446 |
html files are generated in the ``snf-cyclades-app/docs/_build/html`` directory. |
447 |
|
448 |
|
449 |
Continuous integration with Jenkins |
450 |
----------------------------------- |
451 |
.. warning:: This section may be out of date. |
452 |
|
453 |
Preparing a GIT mirror |
454 |
********************** |
455 |
|
456 |
Jenkins cannot currently work with Git over encrypted HTTP. To solve this |
457 |
problem we currently mirror the central Git repository locally on the jenkins |
458 |
installation machine. To setup such a mirror do the following: |
459 |
|
460 |
edit .netrc:: |
461 |
|
462 |
machine code.grnet.gr |
463 |
login accountname |
464 |
password accountpasswd |
465 |
|
466 |
Create the mirror:: |
467 |
|
468 |
git clone --mirror https://code.grnet.gr/git/synnefo synnefo |
469 |
|
470 |
Setup cron to pull from the mirror periodically. Ideally, Git mirror updates |
471 |
should run just before Jenkins jobs check the mirror for changes:: |
472 |
|
473 |
4,14,24,34,44,54 * * * * cd /path/to/mirror && git fetch && git remote prune origin |
474 |
|
475 |
Jenkins setup |
476 |
************* |
477 |
|
478 |
The following instructions will setup Jenkins to run synnefo tests with the |
479 |
SQLite database. To run the tests on MySQL and/or Postgres, step 5 must be |
480 |
replicated. Also, the correct configuration file must be copied (line 6 of the |
481 |
build script). |
482 |
|
483 |
1. Install and start Jenkins. On Debian Squeeze: |
484 |
|
485 |
wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | apt-key add - |
486 |
echo "deb http://pkg.jenkins-ci.org/debian binary/" >>/etc/apt/sources.list |
487 |
echo "deb http://ppa.launchpad.net/chris-lea/zeromq/ubuntu lucid main" >> /etc/apt/sources.list |
488 |
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C7917B12 |
489 |
sudo apt-get update |
490 |
sudo apt-get install jenkins |
491 |
|
492 |
Also install the following packages: |
493 |
|
494 |
apt-get install python-virtualenv libcurl3-gnutls libcurl3-gnutls-dev |
495 |
uuid-dev libmysqlclient-dev libpq-dev libsqlite-dev |
496 |
python-dev libzmq-dev |
497 |
|
498 |
2. After Jenkins starts, go to |
499 |
|
500 |
http://$HOST:8080/pluginManager/ |
501 |
|
502 |
and install the following plug-ins at |
503 |
|
504 |
-Jenkins Cobertura Plugin |
505 |
-Jenkins Email Extension Plugin |
506 |
-Jenkins GIT plugin |
507 |
-Jenkins SLOCCount Plug-in |
508 |
-Hudson/Jenkins Violations plugin |
509 |
|
510 |
3. Configure the Jenkins user's Git details: |
511 |
su jenkins |
512 |
git config --global user.email "buildbot@lists.grnet.gr" |
513 |
git config --global user.name "Buildbot" |
514 |
|
515 |
4. Make sure that all system-level dependencies specified in README.develop |
516 |
are correctly installed |
517 |
|
518 |
5. Create a new "free-style software" job and set the following values:: |
519 |
|
520 |
Project name: synnefo |
521 |
Source Code Management: Git |
522 |
URL of repository: Jenkins Git does not support HTTPS for checking out directly |
523 |
from the repository. The temporary solution is to checkout |
524 |
with a cron script in a directory and set the checkout path |
525 |
in this field |
526 |
Branches to build: master and perhaps others |
527 |
Git->Advanced->Local subdirectory for repo (optional): synnefo |
528 |
Git->Advanced->Prune remote branches before build: check |
529 |
Repository browser: redmineweb, |
530 |
URL: https://code.grnet.gr/projects/synnefo/repository/ |
531 |
Build Triggers->Poll SCM: check |
532 |
Schedule: # every five minutes |
533 |
0,5,10,15,20,25,30,35,40,45,50,55 * * * * |
534 |
|
535 |
Build -> Add build step-> Execute shell |
536 |
|
537 |
Command:: |
538 |
|
539 |
#!/bin/bash -ex |
540 |
cd synnefo |
541 |
mkdir -p reports |
542 |
/usr/bin/sloccount --duplicates --wide --details api util ui logic auth > reports/sloccount.sc |
543 |
cp conf/ci/manage.py . |
544 |
if [ ! -e requirements.pip ]; then cp conf/ci/pip-1.2.conf requirements.pip; fi |
545 |
cat settings.py.dist conf/ci/settings.py.sqlite > settings.py |
546 |
python manage.py update_ve |
547 |
python manage.py hudson api db logic |
548 |
|
549 |
Post-build Actions->Publish JUnit test result report: check |
550 |
Test report XMLs: synnefo/reports/TEST-*.xml |
551 |
|
552 |
Post-build Actions->Publish Cobertura Coverage Report: check |
553 |
Cobertura xml report pattern: synnefo/reports/coverage.xml |
554 |
|
555 |
Post-build Actions->Report Violations: check |
556 |
pylint[XML filename pattern]: synnefo/reports/pylint.report |
557 |
|
558 |
Post-build Actions->Publish SLOCCount analysis results |
559 |
SLOCCount reports: synnefo/reports/sloccount.sc |
560 |
(also, remember to install sloccount at /usr/bin) |
561 |
|
562 |
.. seealso:: |
563 |
http://sites.google.com/site/kmmbvnr/home/django-hudson-tutorial |