root / snf-docs / asterias-dev-guide.rst @ 4c099520
History | View | Annotate | Download (16.6 kB)
1 |
.. _asterias-developer-guide: |
---|---|
2 |
|
3 |
=============== |
4 |
Developer Guide |
5 |
=============== |
6 |
|
7 |
This is the asterias developer guide. |
8 |
|
9 |
It is intended for developers, wishing to implement new functionality |
10 |
inside :ref:`asterias <asterias>`. |
11 |
|
12 |
It assumes thorough familiarity with the :ref:`asterias-admin-guide`. |
13 |
|
14 |
Building a dev environment |
15 |
-------------------------- |
16 |
|
17 |
virtualenv |
18 |
********** |
19 |
|
20 |
The easiest method to deploy a development environment is using |
21 |
:command:`virtualenv`. Alternatively, you can use your system's package manager |
22 |
to install any dependencies of synnefo components (e.g. Macports has them all). |
23 |
|
24 |
.. code-block:: console |
25 |
|
26 |
$ virtualenv ~/synnefo-env |
27 |
$ source ~/synnefo-env/bin/activate |
28 |
(synnefo-env)$ |
29 |
|
30 |
Virtualenv creates an isolated python environment to the path you pass as the |
31 |
first argument of the command. That means that all packages you install using |
32 |
:command:`pip` or :command:`easy_install` will be placed in |
33 |
``ENV/lib/pythonX.X/site-packages`` and their console scripts in ``ENV/bin/``. |
34 |
|
35 |
This allows you to develop against multiple versions of packages that your |
36 |
software depends on without messing with system python packages that may be |
37 |
needed in specific versions for other software you have installed on your |
38 |
system. |
39 |
|
40 |
* It is also recommended to install development helpers: |
41 |
|
42 |
.. code-block:: console |
43 |
|
44 |
(synnefo-env)$ pip install django_extensions fabric>=1.3 |
45 |
|
46 |
* Create a custom settings directory for :ref:`snf-common <snf-common>` and set |
47 |
the ``SYNNEFO_SETTINGS_DIR`` environment variable to use development-specific |
48 |
file:`*.conf` files inside this directory. |
49 |
|
50 |
(synnefo-env)$ mkdir ~/synnefo-settings-dir |
51 |
(synnefo-env)$ export SYNNEFO_SETTINGS_DIR=~/synnefo-settings-dir |
52 |
|
53 |
Insert your custom settings in a file such as :file:`$SYNNEFO_SETTINGS_DIR/99-local.conf`: |
54 |
|
55 |
.. code-block:: python |
56 |
|
57 |
# uncomment this if have django-extensions installed (pip install django_extensions) |
58 |
#INSTALLED_APPS = list(INSTALLED_APPS) + ['django_extensions'] |
59 |
|
60 |
DEV_PATH = os.path.abspath(os.path.dirname(__file__)) |
61 |
DATABASES['default']['NAME'] = os.path.join(DEV_PATH, "synnefo.sqlite") |
62 |
|
63 |
# development rabitmq configuration |
64 |
RABBIT_HOST = "<RabbitMQ_host>" |
65 |
RABBIT_USERNAME = "<RabbitMQ_username>" |
66 |
RABBIT_PASSWORD = "<RabbitMQ_password>" |
67 |
RABBIT_VHOST = "/" |
68 |
|
69 |
# development ganeti settings |
70 |
GANETI_MASTER_IP = "<Ganeti_master_IP>" |
71 |
GANETI_CLUSTER_INFO = (GANETI_MASTER_IP, 5080, "<username>", "<password>") |
72 |
GANETI_CREATEINSTANCE_KWARGS['disk_template'] = 'plain' |
73 |
|
74 |
# This prefix gets used when determining the instance names |
75 |
# of Synnefo VMs at the Ganeti backend. |
76 |
# The dash must always appear in the name! |
77 |
BACKEND_PREFIX_ID = "<your_commit_name>-" |
78 |
|
79 |
IGNORE_FLAVOR_DISK_SIZES = True |
80 |
|
81 |
# do not actually send emails |
82 |
# save them as files in /tmp/synnefo-mails |
83 |
EMAIL_BACKEND = 'django.core.mail.backends.filebased.EmailBackend' |
84 |
EMAIL_FILE_PATH = '/tmp/synnefo-mails' |
85 |
|
86 |
# for UI developers |
87 |
UI_HANDLE_WINDOW_EXCEPTIONS = False |
88 |
|
89 |
# allow login using /?test url |
90 |
BYPASS_AUTHENTICATION = True |
91 |
|
92 |
synnefo source |
93 |
************** |
94 |
|
95 |
* Clone the repository of the synnefo software components you wish |
96 |
to work on, e.g.: |
97 |
|
98 |
.. code-block:: console |
99 |
|
100 |
(synnefo-env)$ git clone https://code.grnet.gr/git/synnefo synnefo |
101 |
(synnefo-env)$ git clone https://code.grnet.gr/git/pithos pithos |
102 |
|
103 |
* Install the software components you wish to work on inside the |
104 |
virtualenv, in development mode: |
105 |
|
106 |
.. code-block:: console |
107 |
|
108 |
(synnefo-env)$ cd snf-asterias-app |
109 |
(synnefo-env)$ python setup.py develop -N |
110 |
|
111 |
* Initialize database: |
112 |
|
113 |
.. code-block:: console |
114 |
|
115 |
(synnefo-env)$ snf-manage syndb |
116 |
(synnefo-env)$ snf-manage migrate |
117 |
(synnefo-env)$ snf-manage loaddata users flavors images |
118 |
|
119 |
Development tips |
120 |
**************** |
121 |
|
122 |
* Running a development web server: |
123 |
|
124 |
.. code-block:: console |
125 |
|
126 |
(synnefo-env)$ snf-manage runserver |
127 |
|
128 |
or, if you have the ``django_extensions`` and ``werkzeug`` packages installed: |
129 |
|
130 |
.. code-block:: console |
131 |
|
132 |
(synnefo-env)$ snf-manage runserver_plus |
133 |
|
134 |
* Opening a python console with the synnefo environment initialized: |
135 |
|
136 |
.. code-block:: console |
137 |
|
138 |
(synnefo-env)$ snf-manage shell |
139 |
|
140 |
or, with the django_extensions package installed: |
141 |
|
142 |
.. code-block:: console |
143 |
|
144 |
(synnefo-env)$ snf-manage shell_plus |
145 |
|
146 |
|
147 |
South Database Migrations |
148 |
------------------------- |
149 |
|
150 |
.. _asterias-dev-initialmigration: |
151 |
|
152 |
Initial Migration |
153 |
***************** |
154 |
|
155 |
To initialize south migrations in your database the following commands must be |
156 |
executed: |
157 |
|
158 |
.. code-block:: console |
159 |
|
160 |
$ snf-manage syncdb --all # Create / update the database with the south tables |
161 |
$ snf-manage migrate --fake # Perform migration in the database |
162 |
|
163 |
|
164 |
Note that ``--all`` and ``--fake`` arguments are only needed when you are |
165 |
initializing your database. If you want to migrate a previously create databse |
166 |
to the latest db scheme just run the same commands without those arguments. |
167 |
|
168 |
If you are trying to migrate a database that already contains the changes that |
169 |
applied from a specific migration script, ``south`` will probably notify you for |
170 |
inconsistent db scheme, a workaround for that issue is to use ``--fake`` option |
171 |
for a specific migration. |
172 |
|
173 |
For example: |
174 |
|
175 |
|
176 |
.. code-block:: console |
177 |
|
178 |
$ snf-manage migrate db 0001 --fake |
179 |
|
180 |
To be sure that all migrations are applied use: |
181 |
|
182 |
.. code-block:: console |
183 |
|
184 |
$ snf-manage migrate db --list |
185 |
|
186 |
All starred migrations are applied. |
187 |
|
188 |
Schema migrations |
189 |
***************** |
190 |
|
191 |
Do not use the syncdb management command. It can only be used the first time |
192 |
and/or if you drop the database and must recreate it from scratch. See |
193 |
:ref:`asterias-dev-initialmigration`. |
194 |
|
195 |
|
196 |
Every time you make changes to the database and data migration is not required |
197 |
(WARNING: always perform this with extreme care): |
198 |
|
199 |
.. code-block:: console |
200 |
|
201 |
$ snf-manage schemamigration db --auto |
202 |
|
203 |
The above will create the migration script. Now this must be applied to the live |
204 |
database: |
205 |
|
206 |
.. code-block:: console |
207 |
|
208 |
$ snf-manage migrate db |
209 |
|
210 |
Consider this example (adding a field to the ``SynnefoUser`` model): |
211 |
|
212 |
.. code-block:: console |
213 |
|
214 |
$ ./bin/python manage.py schemamigration db --auto |
215 |
+ Added field new_south_test_field on db.SynnefoUser |
216 |
|
217 |
Created 0002_auto__add_field_synnefouser_new_south_test_field.py. |
218 |
|
219 |
You can now apply this migration with: |
220 |
|
221 |
.. code-block:: console |
222 |
|
223 |
$ ./manage.py migrate db |
224 |
Running migrations for db: |
225 |
- Migrating forwards to 0002_auto__add_field_synnefouser_new_south_test_field. |
226 |
> db:0002_auto__add_field_synnefouser_new_south_test_field |
227 |
- Loading initial data for db. |
228 |
|
229 |
Installing json fixture 'initial_data' from '/home/bkarak/devel/synnefo/../synnefo/db/fixtures'. |
230 |
Installed 1 object(s) from 1 fixture(s) |
231 |
|
232 |
South needs some extra definitions to the model to preserve and migrate the |
233 |
existing data, for example, if we add a field in a model, we should declare its |
234 |
default value. If not, South will propably fail, after indicating the error: |
235 |
|
236 |
.. code-block:: console |
237 |
|
238 |
$ ./bin/python manage.py schemamigration db --auto |
239 |
? The field 'SynnefoUser.new_south_field_2' does not have a default specified, yet is NOT NULL. |
240 |
? Since you are adding or removing this field, you MUST specify a default |
241 |
? value to use for existing rows. Would you like to: |
242 |
? 1. Quit now, and add a default to the field in models.py |
243 |
? 2. Specify a one-off value to use for existing columns now |
244 |
? Please select a choice: 1 |
245 |
|
246 |
Data migrations |
247 |
*************** |
248 |
|
249 |
To do data migration as well, for example rename a field, use the |
250 |
``datamigration`` management command. |
251 |
|
252 |
In contrast with ``schemamigration``, to perform complex data migration, we |
253 |
must write the script manually. The process is the following: |
254 |
|
255 |
1. Introduce the changes in the code and fixtures (initial data). |
256 |
2. Execute: |
257 |
|
258 |
.. code-block:: console |
259 |
|
260 |
$ snf-manage datamigration <migration_name_here> |
261 |
|
262 |
For example: |
263 |
|
264 |
.. code-block:: console |
265 |
|
266 |
$ ./bin/python manage.py datamigration db rename_credit_wallet |
267 |
Created 0003_rename_credit_wallet.py. |
268 |
|
269 |
3. Edit the generated script. It contains two methods, ``forwards`` and |
270 |
``backwards``. |
271 |
|
272 |
For database operations (column additions, alter tables etc), use the |
273 |
South database API (http://south.aeracode.org/docs/databaseapi.html). |
274 |
|
275 |
To access the data, use the database reference (``orm``) provided as |
276 |
parameter in ``forwards``, ``backwards`` method declarations in the |
277 |
migration script. For example: |
278 |
|
279 |
.. code-block:: python |
280 |
|
281 |
class Migration(DataMigration): |
282 |
|
283 |
def forwards(self, orm): |
284 |
orm.SynnefoUser.objects.all() |
285 |
|
286 |
4. To migrate the database to the latest version, run: |
287 |
|
288 |
.. code-block:: console |
289 |
|
290 |
$ snf-manage migrate db |
291 |
|
292 |
To see which migrations are applied: |
293 |
|
294 |
.. code-block:: console |
295 |
|
296 |
$ snf-manage migrate db --list |
297 |
|
298 |
db |
299 |
(*) 0001_initial |
300 |
(*) 0002_auto__add_field_synnefouser_new_south_test_field |
301 |
(*) 0003_rename_credit_wallet |
302 |
|
303 |
.. seealso:: |
304 |
More information and more thorough examples can be found in the South web site, |
305 |
http://south.aeracode.org/ |
306 |
|
307 |
Test coverage |
308 |
------------- |
309 |
|
310 |
.. warning:: This section may be out of date. |
311 |
|
312 |
In order to get code coverage reports you need to install django-test-coverage |
313 |
|
314 |
.. code-block:: console |
315 |
|
316 |
$ pip install django-test-coverage |
317 |
|
318 |
Then configure the test runner inside Django settings: |
319 |
|
320 |
.. code-block:: python |
321 |
|
322 |
TEST_RUNNER = 'django-test-coverage.runner.run_tests' |
323 |
|
324 |
|
325 |
Internationalization |
326 |
-------------------- |
327 |
|
328 |
This section describes how to translate static strings in Django projects: |
329 |
|
330 |
0. From our project's base, we add directory locale |
331 |
|
332 |
.. code-block:: console |
333 |
|
334 |
$ mkdir locale |
335 |
|
336 |
then we add on the settings.py the language code e.g., |
337 |
|
338 |
.. code-block:: python |
339 |
|
340 |
LANGUAGES = ( |
341 |
('el', u'Greek'), |
342 |
('en', u'English'),) |
343 |
|
344 |
1. For each language we want to add, we run ``makemessages`` from the project's |
345 |
base: |
346 |
|
347 |
.. code-block:: python |
348 |
|
349 |
$ ./bin/django-admin.py makemessages -l el -e html,txt,py |
350 |
(./bin/django-admin.py makemessages -l el -e html,txt,py --ignore=lib/\*) |
351 |
|
352 |
This will add the Greek language, and we specify that :file:`*.html`, |
353 |
:file:`*.txt` and :file:`*.py` files contain translatable strings |
354 |
|
355 |
2. We translate our strings: |
356 |
|
357 |
On :file:`.py` files, e.g., :file:`views.py`, first import ``gettext``: |
358 |
|
359 |
.. code-block:: python |
360 |
|
361 |
from django.utils.translation import gettext_lazy as _ |
362 |
|
363 |
Then every ``string`` to be translated becomes: ``_('string')`` |
364 |
e.g.: |
365 |
|
366 |
.. code-block:: python |
367 |
|
368 |
help_text=_("letters and numbers only")) |
369 |
'title': _('Ubuntu 10.10 server 64bit'), |
370 |
|
371 |
On django templates (``html`` files), on the beggining of the file we add |
372 |
``{% load i18n %}`` then rewrite every string that needs to be translated, |
373 |
as ``{% trans "string" %}``. For example: ``{% trans "Home" %}`` |
374 |
|
375 |
3. When all strings have been translated, run: |
376 |
|
377 |
.. code-block:: console |
378 |
|
379 |
$ django-admin.py makemessages -l el -e html,txt,py |
380 |
|
381 |
processing language ``el``. This creates (or updates) the :file:`po` file |
382 |
for the Greek language. We run this command each time we add new strings to |
383 |
be translated. After that, we can translate our strings in the :file:`po` |
384 |
file (:file:`locale/el/LC_MESSAGES/django.po`) |
385 |
|
386 |
4. When the :file:`po` file is ready, run |
387 |
|
388 |
.. code-block:: console |
389 |
|
390 |
$ ./bin/django-admin.py compilemessages |
391 |
|
392 |
This compiles the ``po`` files to ``mo``. Our strings will appear translated |
393 |
once we change the language (e.g., from a dropdown menu in the page) |
394 |
|
395 |
.. seealso:: |
396 |
http://docs.djangoproject.com/en/dev/topics/i18n/internationalization/ |
397 |
|
398 |
|
399 |
Building source packages |
400 |
------------------------ |
401 |
|
402 |
.. warning:: This section may be out of date. |
403 |
|
404 |
To create a python package from the Synnefo source code run |
405 |
|
406 |
.. code-block:: bash |
407 |
|
408 |
$ cd snf-app |
409 |
$ python setup.py sdist |
410 |
|
411 |
this command will create a ``tar.gz`` python source package inside ``dist`` directory. |
412 |
|
413 |
|
414 |
Building documentation |
415 |
---------------------- |
416 |
|
417 |
Make sure you have ``sphinx`` installed. |
418 |
|
419 |
.. code-block:: bash |
420 |
|
421 |
$ cd snf-app/docs |
422 |
$ make html |
423 |
|
424 |
.. note:: |
425 |
|
426 |
The theme define in the Sphinx configuration file ``conf.py`` is ``nature``, |
427 |
not available in the version of Sphinx shipped with Debian Squeeze. Replace |
428 |
it with ``default`` to build with a Squeeze-provided Sphinx. |
429 |
|
430 |
html files are generated in the ``snf-app/docs/_build/html`` directory. |
431 |
|
432 |
|
433 |
Continuous integration with Jenkins |
434 |
----------------------------------- |
435 |
.. warning:: This section may be out of date. |
436 |
|
437 |
Preparing a GIT mirror |
438 |
********************** |
439 |
|
440 |
Jenkins cannot currently work with Git over encrypted HTTP. To solve this |
441 |
problem we currently mirror the central Git repository locally on the jenkins |
442 |
installation machine. To setup such a mirror do the following: |
443 |
|
444 |
edit .netrc:: |
445 |
|
446 |
machine code.grnet.gr |
447 |
login accountname |
448 |
password accountpasswd |
449 |
|
450 |
Create the mirror:: |
451 |
|
452 |
git clone --mirror https://code.grnet.gr/git/synnefo synnefo |
453 |
|
454 |
Setup cron to pull from the mirror periodically. Ideally, Git mirror updates |
455 |
should run just before Jenkins jobs check the mirror for changes:: |
456 |
|
457 |
4,14,24,34,44,54 * * * * cd /path/to/mirror && git fetch && git remote prune origin |
458 |
|
459 |
Jenkins setup |
460 |
************* |
461 |
|
462 |
The following instructions will setup Jenkins to run synnefo tests with the |
463 |
SQLite database. To run the tests on MySQL and/or Postgres, step 5 must be |
464 |
replicated. Also, the correct configuration file must be copied (line 6 of the |
465 |
build script). |
466 |
|
467 |
1. Install and start Jenkins. On Debian Squeeze: |
468 |
|
469 |
wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | apt-key add - |
470 |
echo "deb http://pkg.jenkins-ci.org/debian binary/" >>/etc/apt/sources.list |
471 |
echo "deb http://ppa.launchpad.net/chris-lea/zeromq/ubuntu lucid main" >> /etc/apt/sources.list |
472 |
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C7917B12 |
473 |
sudo apt-get update |
474 |
sudo apt-get install jenkins |
475 |
|
476 |
Also install the following packages: |
477 |
|
478 |
apt-get install python-virtualenv libcurl3-gnutls libcurl3-gnutls-dev |
479 |
uuid-dev libmysqlclient-dev libpq-dev libsqlite-dev |
480 |
python-dev libzmq-dev |
481 |
|
482 |
2. After Jenkins starts, go to |
483 |
|
484 |
http://$HOST:8080/pluginManager/ |
485 |
|
486 |
and install the following plug-ins at |
487 |
|
488 |
-Jenkins Cobertura Plugin |
489 |
-Jenkins Email Extension Plugin |
490 |
-Jenkins GIT plugin |
491 |
-Jenkins SLOCCount Plug-in |
492 |
-Hudson/Jenkins Violations plugin |
493 |
|
494 |
3. Configure the Jenkins user's Git details: |
495 |
su jenkins |
496 |
git config --global user.email "buildbot@lists.grnet.gr" |
497 |
git config --global user.name "Buildbot" |
498 |
|
499 |
4. Make sure that all system-level dependencies specified in README.develop |
500 |
are correctly installed |
501 |
|
502 |
5. Create a new "free-style software" job and set the following values:: |
503 |
|
504 |
Project name: synnefo |
505 |
Source Code Management: Git |
506 |
URL of repository: Jenkins Git does not support HTTPS for checking out directly |
507 |
from the repository. The temporary solution is to checkout |
508 |
with a cron script in a directory and set the checkout path |
509 |
in this field |
510 |
Branches to build: master and perhaps others |
511 |
Git->Advanced->Local subdirectory for repo (optional): synnefo |
512 |
Git->Advanced->Prune remote branches before build: check |
513 |
Repository browser: redmineweb, |
514 |
URL: https://code.grnet.gr/projects/synnefo/repository/ |
515 |
Build Triggers->Poll SCM: check |
516 |
Schedule: # every five minutes |
517 |
0,5,10,15,20,25,30,35,40,45,50,55 * * * * |
518 |
|
519 |
Build -> Add build step-> Execute shell |
520 |
|
521 |
Command:: |
522 |
|
523 |
#!/bin/bash -ex |
524 |
cd synnefo |
525 |
mkdir -p reports |
526 |
/usr/bin/sloccount --duplicates --wide --details api util ui logic auth > reports/sloccount.sc |
527 |
cp conf/ci/manage.py . |
528 |
if [ ! -e requirements.pip ]; then cp conf/ci/pip-1.2.conf requirements.pip; fi |
529 |
cat settings.py.dist conf/ci/settings.py.sqlite > settings.py |
530 |
python manage.py update_ve |
531 |
python manage.py hudson api db logic |
532 |
|
533 |
Post-build Actions->Publish JUnit test result report: check |
534 |
Test report XMLs: synnefo/reports/TEST-*.xml |
535 |
|
536 |
Post-build Actions->Publish Cobertura Coverage Report: check |
537 |
Cobertura xml report pattern: synnefo/reports/coverage.xml |
538 |
|
539 |
Post-build Actions->Report Violations: check |
540 |
pylint[XML filename pattern]: synnefo/reports/pylint.report |
541 |
|
542 |
Post-build Actions->Publish SLOCCount analysis results |
543 |
SLOCCount reports: synnefo/reports/sloccount.sc |
544 |
(also, remember to install sloccount at /usr/bin) |
545 |
|
546 |
.. seealso:: |
547 |
http://sites.google.com/site/kmmbvnr/home/django-hudson-tutorial |