root / snf-docs / asterias-dev-guide.rst @ 9940eacf
History | View | Annotate | Download (15.8 kB)
1 |
.. _asterias-developer-guide: |
---|---|
2 |
|
3 |
=============== |
4 |
Developer Guide |
5 |
=============== |
6 |
|
7 |
This is the asterias developer guide. |
8 |
|
9 |
It is intended for developers, wishing to implement new functionality |
10 |
inside :ref:`asterias <asterias>`. |
11 |
|
12 |
It assumes thorough familiarity with the :ref:`asterias-admin-guide`. |
13 |
|
14 |
Building a dev environment |
15 |
-------------------------- |
16 |
|
17 |
virtualenv |
18 |
********** |
19 |
|
20 |
The easiest method to deploy a development environment is using |
21 |
:command:`virtualenv`. Alternatively, you can use your system's package manager |
22 |
to install any dependencies of synnefo components (e.g. Macports has them all). |
23 |
|
24 |
.. code-block:: console |
25 |
|
26 |
$ virtualenv ~/synnefo-env |
27 |
$ . ~/synnefo-env/bin/activate |
28 |
(synnefo-env)$ |
29 |
|
30 |
Any :command:`pip` commands executed from now on, affect the ``synnefo-env`` |
31 |
virtualenv. |
32 |
|
33 |
* It is also recommended to install development helpers: |
34 |
|
35 |
.. code-block:: console |
36 |
|
37 |
(synnefo-env)$ pip install django_extensions |
38 |
|
39 |
* Create a custom settings directory for :ref:`snf-common <snf-common>` and set |
40 |
the ``SYNNEFO_SETTINGS_DIR`` environment variable to use development-specific |
41 |
file:`*.conf` files inside this directory. |
42 |
|
43 |
(synnefo-env)$ mkdir ~/synnefo-settings-dir |
44 |
(synnefo-env)$ export SYNNEFO_SETTINGS_DIR=~/synnefo-settings-dir |
45 |
|
46 |
Insert your custom settings in a file such as :file:`$SYNNEFO_SETTINGS_DIR/99-local.conf`: |
47 |
|
48 |
.. code-block:: python |
49 |
|
50 |
# uncomment this if have django-extensions installed (pip install django_extensions) |
51 |
#INSTALLED_APPS = list(INSTALLED_APPS) + ['django_extensions'] |
52 |
|
53 |
DEV_PATH = os.path.abspath(os.path.dirname(__file__)) |
54 |
DATABASES['default']['NAME'] = os.path.join(DEV_PATH, "synnefo.sqlite") |
55 |
|
56 |
# development rabitmq configuration |
57 |
RABBIT_HOST = "<RabbitMQ_host>" |
58 |
RABBIT_USERNAME = "<RabbitMQ_username>" |
59 |
RABBIT_PASSWORD = "<RabbitMQ_password>" |
60 |
RABBIT_VHOST = "/" |
61 |
|
62 |
# development ganeti settings |
63 |
GANETI_MASTER_IP = "<Ganeti_master_IP>" |
64 |
GANETI_CLUSTER_INFO = (GANETI_MASTER_IP, 5080, "<username>", "<password>") |
65 |
GANETI_CREATEINSTANCE_KWARGS['disk_template'] = 'plain' |
66 |
|
67 |
# This prefix gets used when determining the instance names |
68 |
# of Synnefo VMs at the Ganeti backend. |
69 |
# The dash must always appear in the name! |
70 |
BACKEND_PREFIX_ID = "<your_commit_name>-" |
71 |
|
72 |
IGNORE_FLAVOR_DISK_SIZES = True |
73 |
|
74 |
# do not actually send emails |
75 |
# save them as files in /tmp/synnefo-mails |
76 |
EMAIL_BACKEND = 'django.core.mail.backends.filebased.EmailBackend' |
77 |
EMAIL_FILE_PATH = '/tmp/synnefo-mails' |
78 |
|
79 |
# for UI developers |
80 |
UI_HANDLE_WINDOW_EXCEPTIONS = False |
81 |
|
82 |
# allow login using /?test url |
83 |
BYPASS_AUTHENTICATION = True |
84 |
|
85 |
synnefo source |
86 |
************** |
87 |
|
88 |
* Clone the repository of the synnefo software components you wish |
89 |
to work on, e.g.: |
90 |
|
91 |
.. code-block:: console |
92 |
|
93 |
(synnefo-env)$ git clone https://code.grnet.gr/git/synnefo synnefo |
94 |
|
95 |
* Install the software components you wish to work on inside the |
96 |
virtualenv, in development mode: |
97 |
|
98 |
.. code-block:: console |
99 |
|
100 |
(synnefo-env)$ cd snf-asterias-app |
101 |
(synnefo-env)$ python setup.py develop -N |
102 |
|
103 |
* Initialize database: |
104 |
|
105 |
.. code-block:: console |
106 |
|
107 |
(synnefo-env)$ snf-manage syndb |
108 |
(synnefo-env)$ snf-manage migrate |
109 |
(synnefo-env)$ snf-manage loaddata users flavors images |
110 |
|
111 |
Development tips |
112 |
**************** |
113 |
|
114 |
* Running a development web server: |
115 |
|
116 |
.. code-block:: console |
117 |
|
118 |
(synnefo-env)$ snf-manage runserver |
119 |
|
120 |
or, if you have the django_extensions and werkzeug packages installed: |
121 |
|
122 |
.. code-block:: console |
123 |
|
124 |
(synnefo-env)$ snf-manage runserver_plus |
125 |
|
126 |
* Opening a python console with the synnefo environment initialized: |
127 |
|
128 |
.. code-block:: console |
129 |
|
130 |
(synnefo-env)$ snf-manage shell |
131 |
|
132 |
or, with the django_extensions package installed: |
133 |
|
134 |
.. code-block:: console |
135 |
|
136 |
(synnefo-env)$ snf-manage shell_plus |
137 |
|
138 |
|
139 |
South Database Migrations |
140 |
------------------------- |
141 |
|
142 |
.. _asterias-dev-initialmigration: |
143 |
|
144 |
Initial Migration |
145 |
***************** |
146 |
|
147 |
To initialize south migrations in your database the following commands must be |
148 |
executed: |
149 |
|
150 |
.. code-block:: console |
151 |
|
152 |
$ snf-manage syncdb # Create / update the database with the south tables |
153 |
$ snf-manage migrate # Perform migration in the database |
154 |
|
155 |
Note that syncdb will create the latest models that exist in the db app, so some |
156 |
migrations may fail. If you are sure a migration has already taken place you |
157 |
must use the ``--fake`` option, to apply it. |
158 |
|
159 |
For example: |
160 |
|
161 |
|
162 |
.. code-block:: console |
163 |
|
164 |
$ snf-manage migrate db 0001 --fake |
165 |
|
166 |
To be sure that all migrations are applied use: |
167 |
|
168 |
.. code-block:: console |
169 |
|
170 |
$ snf-manage migrate db --list |
171 |
|
172 |
All starred migrations are applied. |
173 |
|
174 |
Schema migrations |
175 |
***************** |
176 |
|
177 |
Do not use the syncdb management command. It can only be used the first time |
178 |
and/or if you drop the database and must recreate it from scratch. See |
179 |
:ref:`asterias-dev-initialmigration`. |
180 |
|
181 |
|
182 |
Every time you make changes to the database and data migration is not required |
183 |
(WARNING: always perform this with extreme care): |
184 |
|
185 |
.. code-block:: console |
186 |
|
187 |
$ snf-manage schemamigration db --auto |
188 |
|
189 |
The above will create the migration script. Now this must be applied to the live |
190 |
database: |
191 |
|
192 |
.. code-block:: console |
193 |
|
194 |
$ snf-manage migrate db |
195 |
|
196 |
Consider this example (adding a field to the ``SynnefoUser`` model): |
197 |
|
198 |
.. code-block:: console |
199 |
|
200 |
$ ./bin/python manage.py schemamigration db --auto |
201 |
+ Added field new_south_test_field on db.SynnefoUser |
202 |
|
203 |
Created 0002_auto__add_field_synnefouser_new_south_test_field.py. |
204 |
|
205 |
You can now apply this migration with: |
206 |
|
207 |
.. code-block:: console |
208 |
|
209 |
$ ./manage.py migrate db |
210 |
Running migrations for db: |
211 |
- Migrating forwards to 0002_auto__add_field_synnefouser_new_south_test_field. |
212 |
> db:0002_auto__add_field_synnefouser_new_south_test_field |
213 |
- Loading initial data for db. |
214 |
|
215 |
Installing json fixture 'initial_data' from '/home/bkarak/devel/synnefo/../synnefo/db/fixtures'. |
216 |
Installed 1 object(s) from 1 fixture(s) |
217 |
|
218 |
South needs some extra definitions to the model to preserve and migrate the |
219 |
existing data, for example, if we add a field in a model, we should declare its |
220 |
default value. If not, South will propably fail, after indicating the error: |
221 |
|
222 |
.. code-block:: console |
223 |
|
224 |
$ ./bin/python manage.py schemamigration db --auto |
225 |
? The field 'SynnefoUser.new_south_field_2' does not have a default specified, yet is NOT NULL. |
226 |
? Since you are adding or removing this field, you MUST specify a default |
227 |
? value to use for existing rows. Would you like to: |
228 |
? 1. Quit now, and add a default to the field in models.py |
229 |
? 2. Specify a one-off value to use for existing columns now |
230 |
? Please select a choice: 1 |
231 |
|
232 |
Data migrations |
233 |
*************** |
234 |
|
235 |
To do data migration as well, for example rename a field, use the |
236 |
``datamigration`` management command. |
237 |
|
238 |
In contrast with ``schemamigration``, to perform complex data migration, we |
239 |
must write the script manually. The process is the following: |
240 |
|
241 |
1. Introduce the changes in the code and fixtures (initial data). |
242 |
2. Execute: |
243 |
|
244 |
.. code-block:: console |
245 |
|
246 |
$ snf-manage datamigration <migration_name_here> |
247 |
|
248 |
For example: |
249 |
|
250 |
.. code-block:: console |
251 |
|
252 |
$ ./bin/python manage.py datamigration db rename_credit_wallet |
253 |
Created 0003_rename_credit_wallet.py. |
254 |
|
255 |
3. Edit the generated script. It contains two methods, ``forwards`` and |
256 |
``backwards``. |
257 |
|
258 |
For database operations (column additions, alter tables etc), use the |
259 |
South database API (http://south.aeracode.org/docs/databaseapi.html). |
260 |
|
261 |
To access the data, use the database reference (``orm``) provided as |
262 |
parameter in ``forwards``, ``backwards`` method declarations in the |
263 |
migration script. For example: |
264 |
|
265 |
.. code-block:: python |
266 |
|
267 |
class Migration(DataMigration): |
268 |
|
269 |
def forwards(self, orm): |
270 |
orm.SynnefoUser.objects.all() |
271 |
|
272 |
4. To migrate the database to the latest version, run: |
273 |
|
274 |
.. code-block:: console |
275 |
|
276 |
$ snf-manage migrate db |
277 |
|
278 |
To see which migrations are applied: |
279 |
|
280 |
.. code-block:: console |
281 |
|
282 |
$ snf-manage migrate db --list |
283 |
|
284 |
db |
285 |
(*) 0001_initial |
286 |
(*) 0002_auto__add_field_synnefouser_new_south_test_field |
287 |
(*) 0003_rename_credit_wallet |
288 |
|
289 |
.. seealso:: |
290 |
More information and more thorough examples can be found in the South web site, |
291 |
http://south.aeracode.org/ |
292 |
|
293 |
Test coverage |
294 |
------------- |
295 |
|
296 |
.. warning:: This section may be out of date. |
297 |
|
298 |
In order to get code coverage reports you need to install django-test-coverage |
299 |
|
300 |
.. code-block:: console |
301 |
|
302 |
$ pip install django-test-coverage |
303 |
|
304 |
Then configure the test runner inside Django settings: |
305 |
|
306 |
.. code-block:: python |
307 |
|
308 |
TEST_RUNNER = 'django-test-coverage.runner.run_tests' |
309 |
|
310 |
|
311 |
Internationalization |
312 |
-------------------- |
313 |
|
314 |
This section describes how to translate static strings in Django projects: |
315 |
|
316 |
0. From our project's base, we add directory locale |
317 |
|
318 |
.. code-block:: console |
319 |
|
320 |
$ mkdir locale |
321 |
|
322 |
then we add on the settings.py the language code e.g., |
323 |
|
324 |
.. code-block:: python |
325 |
|
326 |
LANGUAGES = ( |
327 |
('el', u'Greek'), |
328 |
('en', u'English'),) |
329 |
|
330 |
1. For each language we want to add, we run ``makemessages`` from the project's |
331 |
base: |
332 |
|
333 |
.. code-block:: python |
334 |
|
335 |
$ ./bin/django-admin.py makemessages -l el -e html,txt,py |
336 |
(./bin/django-admin.py makemessages -l el -e html,txt,py --ignore=lib/\*) |
337 |
|
338 |
This will add the Greek language, and we specify that :file:`*.html`, |
339 |
:file:`*.txt` and :file:`*.py` files contain translatable strings |
340 |
|
341 |
2. We translate our strings: |
342 |
|
343 |
On :file:`.py` files, e.g., :file:`views.py`, first import ``gettext``: |
344 |
|
345 |
.. code-block:: python |
346 |
|
347 |
from django.utils.translation import gettext_lazy as _ |
348 |
|
349 |
Then every ``string`` to be translated becomes: ``_('string')`` |
350 |
e.g.: |
351 |
|
352 |
.. code-block:: python |
353 |
|
354 |
help_text=_("letters and numbers only")) |
355 |
'title': _('Ubuntu 10.10 server 64bit'), |
356 |
|
357 |
On django templates (``html`` files), on the beggining of the file we add |
358 |
``{% load i18n %}`` then rewrite every string that needs to be translated, |
359 |
as ``{% trans "string" %}``. For example: ``{% trans "Home" %}`` |
360 |
|
361 |
3. When all strings have been translated, run: |
362 |
|
363 |
.. code-block:: console |
364 |
|
365 |
$ django-admin.py makemessages -l el -e html,txt,py |
366 |
|
367 |
processing language ``el``. This creates (or updates) the :file:`po` file |
368 |
for the Greek language. We run this command each time we add new strings to |
369 |
be translated. After that, we can translate our strings in the :file:`po` |
370 |
file (:file:`locale/el/LC_MESSAGES/django.po`) |
371 |
|
372 |
4. When the :file:`po` file is ready, run |
373 |
|
374 |
.. code-block:: console |
375 |
|
376 |
$ ./bin/django-admin.py compilemessages |
377 |
|
378 |
This compiles the ``po`` files to ``mo``. Our strings will appear translated |
379 |
once we change the language (e.g., from a dropdown menu in the page) |
380 |
|
381 |
.. seealso:: |
382 |
http://docs.djangoproject.com/en/dev/topics/i18n/internationalization/ |
383 |
|
384 |
|
385 |
Building source packages |
386 |
------------------------ |
387 |
|
388 |
.. warning:: This section may be out of date. |
389 |
|
390 |
To create a python package from the Synnefo source code run |
391 |
|
392 |
.. code-block:: bash |
393 |
|
394 |
$ cd snf-app |
395 |
$ python setup.py sdist |
396 |
|
397 |
this command will create a ``tar.gz`` python source package inside ``dist`` directory. |
398 |
|
399 |
|
400 |
Building documentation |
401 |
---------------------- |
402 |
|
403 |
Make sure you have ``sphinx`` installed. |
404 |
|
405 |
.. code-block:: bash |
406 |
|
407 |
$ cd snf-app/docs |
408 |
$ make html |
409 |
|
410 |
.. note:: |
411 |
|
412 |
The theme define in the Sphinx configuration file ``conf.py`` is ``nature``, |
413 |
not available in the version of Sphinx shipped with Debian Squeeze. Replace |
414 |
it with ``default`` to build with a Squeeze-provided Sphinx. |
415 |
|
416 |
html files are generated in the ``snf-app/docs/_build/html`` directory. |
417 |
|
418 |
|
419 |
Continuous integration with Jenkins |
420 |
----------------------------------- |
421 |
.. warning:: This section may be out of date. |
422 |
|
423 |
Preparing a GIT mirror |
424 |
********************** |
425 |
|
426 |
Jenkins cannot currently work with Git over encrypted HTTP. To solve this |
427 |
problem we currently mirror the central Git repository locally on the jenkins |
428 |
installation machine. To setup such a mirror do the following: |
429 |
|
430 |
edit .netrc:: |
431 |
|
432 |
machine code.grnet.gr |
433 |
login accountname |
434 |
password accountpasswd |
435 |
|
436 |
Create the mirror:: |
437 |
|
438 |
git clone --mirror https://code.grnet.gr/git/synnefo synnefo |
439 |
|
440 |
Setup cron to pull from the mirror periodically. Ideally, Git mirror updates |
441 |
should run just before Jenkins jobs check the mirror for changes:: |
442 |
|
443 |
4,14,24,34,44,54 * * * * cd /path/to/mirror && git fetch && git remote prune origin |
444 |
|
445 |
Jenkins setup |
446 |
************* |
447 |
|
448 |
The following instructions will setup Jenkins to run synnefo tests with the |
449 |
SQLite database. To run the tests on MySQL and/or Postgres, step 5 must be |
450 |
replicated. Also, the correct configuration file must be copied (line 6 of the |
451 |
build script). |
452 |
|
453 |
1. Install and start Jenkins. On Debian Squeeze: |
454 |
|
455 |
wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | apt-key add - |
456 |
echo "deb http://pkg.jenkins-ci.org/debian binary/" >>/etc/apt/sources.list |
457 |
echo "deb http://ppa.launchpad.net/chris-lea/zeromq/ubuntu lucid main" >> /etc/apt/sources.list |
458 |
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C7917B12 |
459 |
sudo apt-get update |
460 |
sudo apt-get install jenkins |
461 |
|
462 |
Also install the following packages: |
463 |
|
464 |
apt-get install python-virtualenv libcurl3-gnutls libcurl3-gnutls-dev |
465 |
uuid-dev libmysqlclient-dev libpq-dev libsqlite-dev |
466 |
python-dev libzmq-dev |
467 |
|
468 |
2. After Jenkins starts, go to |
469 |
|
470 |
http://$HOST:8080/pluginManager/ |
471 |
|
472 |
and install the following plug-ins at |
473 |
|
474 |
-Jenkins Cobertura Plugin |
475 |
-Jenkins Email Extension Plugin |
476 |
-Jenkins GIT plugin |
477 |
-Jenkins SLOCCount Plug-in |
478 |
-Hudson/Jenkins Violations plugin |
479 |
|
480 |
3. Configure the Jenkins user's Git details: |
481 |
su jenkins |
482 |
git config --global user.email "buildbot@lists.grnet.gr" |
483 |
git config --global user.name "Buildbot" |
484 |
|
485 |
4. Make sure that all system-level dependencies specified in README.develop |
486 |
are correctly installed |
487 |
|
488 |
5. Create a new "free-style software" job and set the following values:: |
489 |
|
490 |
Project name: synnefo |
491 |
Source Code Management: Git |
492 |
URL of repository: Jenkins Git does not support HTTPS for checking out directly |
493 |
from the repository. The temporary solution is to checkout |
494 |
with a cron script in a directory and set the checkout path |
495 |
in this field |
496 |
Branches to build: master and perhaps others |
497 |
Git->Advanced->Local subdirectory for repo (optional): synnefo |
498 |
Git->Advanced->Prune remote branches before build: check |
499 |
Repository browser: redmineweb, |
500 |
URL: https://code.grnet.gr/projects/synnefo/repository/ |
501 |
Build Triggers->Poll SCM: check |
502 |
Schedule: # every five minutes |
503 |
0,5,10,15,20,25,30,35,40,45,50,55 * * * * |
504 |
|
505 |
Build -> Add build step-> Execute shell |
506 |
|
507 |
Command:: |
508 |
|
509 |
#!/bin/bash -ex |
510 |
cd synnefo |
511 |
mkdir -p reports |
512 |
/usr/bin/sloccount --duplicates --wide --details api util ui logic auth > reports/sloccount.sc |
513 |
cp conf/ci/manage.py . |
514 |
if [ ! -e requirements.pip ]; then cp conf/ci/pip-1.2.conf requirements.pip; fi |
515 |
cat settings.py.dist conf/ci/settings.py.sqlite > settings.py |
516 |
python manage.py update_ve |
517 |
python manage.py hudson api db logic |
518 |
|
519 |
Post-build Actions->Publish JUnit test result report: check |
520 |
Test report XMLs: synnefo/reports/TEST-*.xml |
521 |
|
522 |
Post-build Actions->Publish Cobertura Coverage Report: check |
523 |
Cobertura xml report pattern: synnefo/reports/coverage.xml |
524 |
|
525 |
Post-build Actions->Report Violations: check |
526 |
pylint[XML filename pattern]: synnefo/reports/pylint.report |
527 |
|
528 |
Post-build Actions->Publish SLOCCount analysis results |
529 |
SLOCCount reports: synnefo/reports/sloccount.sc |
530 |
(also, remember to install sloccount at /usr/bin) |
531 |
|
532 |
.. seealso:: |
533 |
http://sites.google.com/site/kmmbvnr/home/django-hudson-tutorial |