Statistics
| Branch: | Tag: | Revision:

root / docs / quick-install-intgrt-guide.rst @ dad708b4

History | View | Annotate | Download (17.4 kB)

1
.. _quick-install-intgrt-guide:
2

    
3
Integrator's Quick Installation Guide
4
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
5

    
6
This is the Integrator's quick installation guide.
7

    
8
It is intended for developers, wishing to implement new functionality
9
inside Synnefo. It assumes thorough familiarity with the
10
:ref:`Synnefo Administrator's Guide <admin-guide>`.
11

    
12
It describes how to install the whole synnefo stack on two (2) physical nodes,
13
with minimum configuration. It installs synnefo in a ``virtualenv`` using ``pip
14
install``, and assumes the nodes run Debian Squeeze. After successful
15
installation, you will have the following services running:
16

    
17
 * Identity Management (Astakos)
18
 * Object Storage Service (Pithos+)
19
 * Compute Service (Cyclades)
20
 * Image Registry Service (Plankton)
21

    
22
and a single unified Web UI to manage them all.
23

    
24
The Volume Storage Service (Archipelago) and the Billing Service (Aquarium) are
25
not released yet.
26

    
27
If you just want to install the Object Storage Service (Pithos+), follow the guide
28
and just stop after the "Testing of Pithos+" section.
29

    
30
Building a dev environment
31
--------------------------
32

    
33
virtualenv
34
**********
35

    
36
The easiest method to deploy a development environment is using
37
:command:`virtualenv`. Alternatively, you can use your system's package manager
38
to install any dependencies of synnefo components (e.g. Macports has them all).
39

    
40
   .. code-block:: console
41
   
42
      $ virtualenv ~/synnefo-env
43
      $ source ~/synnefo-env/bin/activate
44
      (synnefo-env)$ 
45

    
46
Virtualenv creates an isolated python environment to the path you pass as the
47
first argument of the command. That means that all packages you install using
48
:command:`pip` or :command:`easy_install` will be placed in
49
``ENV/lib/pythonX.X/site-packages`` and their console scripts in ``ENV/bin/``.
50

    
51
This allows you to develop against multiple versions of packages that your
52
software depends on without messing with system python packages that may be
53
needed in specific versions for other software you have installed on your
54
system.
55

    
56
* It is also recommended to install development helpers:
57

    
58
  .. code-block:: console
59
 
60
     (synnefo-env)$ pip install django_extensions fabric>=1.3
61

    
62
* Create a custom settings directory for :ref:`snf-common <snf-common>` and set
63
  the ``SYNNEFO_SETTINGS_DIR`` environment variable to use development-specific
64
  file:`*.conf` files inside this directory.
65

    
66
  (synnefo-env)$ mkdir ~/synnefo-settings-dir
67
  (synnefo-env)$ export SYNNEFO_SETTINGS_DIR=~/synnefo-settings-dir
68
    
69
  Insert your custom settings in a file such as :file:`$SYNNEFO_SETTINGS_DIR/99-local.conf`:
70

    
71
  .. code-block:: python
72
    
73
        # uncomment this if have django-extensions installed (pip install django_extensions)
74
        #INSTALLED_APPS = list(INSTALLED_APPS) + ['django_extensions']
75

    
76
        DEV_PATH = os.path.abspath(os.path.dirname(__file__))
77
        DATABASES['default']['NAME'] = os.path.join(DEV_PATH, "synnefo.sqlite")
78

    
79
        # development rabitmq configuration
80
        RABBIT_HOST = "<RabbitMQ_host>"
81
        RABBIT_USERNAME = "<RabbitMQ_username>"
82
        RABBIT_PASSWORD = "<RabbitMQ_password>"
83
        RABBIT_VHOST = "/"
84

    
85
        # development ganeti settings
86
        GANETI_MASTER_IP = "<Ganeti_master_IP>"
87
        GANETI_CLUSTER_INFO = (GANETI_MASTER_IP, 5080, "<username>", "<password>")
88
        GANETI_CREATEINSTANCE_KWARGS['disk_template'] = 'plain'
89

    
90
        # This prefix gets used when determining the instance names
91
        # of Synnefo VMs at the Ganeti backend.
92
        # The dash must always appear in the name!
93
        BACKEND_PREFIX_ID = "<your_commit_name>-"
94

    
95
        IGNORE_FLAVOR_DISK_SIZES = True
96

    
97
        # do not actually send emails
98
        # save them as files in /tmp/synnefo-mails
99
        EMAIL_BACKEND = 'django.core.mail.backends.filebased.EmailBackend'
100
        EMAIL_FILE_PATH = '/tmp/synnefo-mails'
101

    
102
        # for UI developers
103
        UI_HANDLE_WINDOW_EXCEPTIONS = False
104

    
105
        # allow login using /?test url
106
        BYPASS_AUTHENTICATION = True 
107

    
108
synnefo source
109
**************
110

    
111
* Clone the repository of the synnefo software components you wish
112
  to work on, e.g.:
113

    
114
   .. code-block:: console
115
   
116
     (synnefo-env)$ git clone https://code.grnet.gr/git/synnefo synnefo
117
     (synnefo-env)$ git clone https://code.grnet.gr/git/pithos pithos
118
   
119
* Install the software components you wish to work on inside the
120
  virtualenv, in development mode:
121

    
122
   .. code-block:: console
123
   
124
      (synnefo-env)$ cd snf-cyclades-app
125
      (synnefo-env)$ python setup.py develop -N
126
   
127
* Initialize database:
128

    
129
   .. code-block:: console
130
     
131
      (synnefo-env)$ snf-manage syndb
132
      (synnefo-env)$ snf-manage migrate
133
      (synnefo-env)$ snf-manage loaddata users flavors images
134
  
135
Development tips
136
****************
137

    
138
* Running a development web server:
139

    
140
  .. code-block:: console
141

    
142
     (synnefo-env)$ snf-manage runserver
143

    
144
  or, if you have the ``django_extensions`` and ``werkzeug`` packages installed:
145

    
146
  .. code-block:: console
147

    
148
     (synnefo-env)$ snf-manage runserver_plus
149

    
150
* Opening a python console with the synnefo environment initialized:
151

    
152
  .. code-block:: console
153

    
154
     (synnefo-env)$ snf-manage shell
155

    
156
  or, with the django_extensions package installed:
157

    
158
  .. code-block:: console
159
     
160
     (synnefo-env)$ snf-manage shell_plus
161

    
162

    
163
South Database Migrations
164
-------------------------
165

    
166
.. _cyclades-dev-initialmigration:
167

    
168
Initial Migration
169
*****************
170

    
171
To initialize south migrations in your database the following commands must be
172
executed:
173

    
174
.. code-block:: console
175

    
176
   $ snf-manage syncdb --all      # Create / update the database with the south tables
177
   $ snf-manage migrate --fake    # Perform migration in the database
178

    
179

    
180
Note that ``--all`` and ``--fake`` arguments are only needed when you are
181
initializing your database. If you want to migrate a previously create databse
182
to the latest db scheme just run the same commands without those arguments.
183

    
184
If you are trying to migrate a database that already contains the changes that
185
applied from a specific migration script, ``south`` will probably notify you for
186
inconsistent db scheme, a workaround for that issue is to use ``--fake`` option
187
for a specific migration.
188

    
189
For example:
190

    
191

    
192
.. code-block:: console
193

    
194
   $ snf-manage migrate db 0001 --fake
195

    
196
To be sure that all migrations are applied use:
197

    
198
.. code-block:: console
199

    
200
   $ snf-manage migrate db --list
201

    
202
All starred migrations are applied.
203

    
204
Schema migrations
205
*****************
206

    
207
Do not use the syncdb management command. It can only be used the first time
208
and/or if you drop the database and must recreate it from scratch. See
209
:ref:`cyclades-dev-initialmigration`.
210

    
211

    
212
Every time you make changes to the database and data migration is not required
213
(WARNING: always perform this with extreme care):
214

    
215
.. code-block:: console
216
   
217
   $ snf-manage schemamigration db --auto
218

    
219
The above will create the migration script. Now this must be applied to the live
220
database:
221

    
222
.. code-block:: console
223

    
224
   $ snf-manage migrate db
225

    
226
Consider this example (adding a field to the ``SynnefoUser`` model):
227

    
228
.. code-block:: console
229

    
230
   $ ./bin/python manage.py schemamigration db --auto
231
   + Added field new_south_test_field on db.SynnefoUser
232

    
233
   Created 0002_auto__add_field_synnefouser_new_south_test_field.py.
234

    
235
You can now apply this migration with:
236

    
237
.. code-block:: console
238

    
239
   $ ./manage.py migrate db
240
   Running migrations for db:
241
   - Migrating forwards to 0002_auto__add_field_synnefouser_new_south_test_field.
242
   > db:0002_auto__add_field_synnefouser_new_south_test_field
243
   - Loading initial data for db.
244

    
245
   Installing json fixture 'initial_data' from '/home/bkarak/devel/synnefo/../synnefo/db/fixtures'.
246
   Installed 1 object(s) from 1 fixture(s)
247

    
248
South needs some extra definitions to the model to preserve and migrate the
249
existing data, for example, if we add a field in a model, we should declare its
250
default value. If not, South will propably fail, after indicating the error:
251

    
252
.. code-block:: console
253

    
254
   $ ./bin/python manage.py schemamigration db --auto
255
   ? The field 'SynnefoUser.new_south_field_2' does not have a default specified, yet is NOT NULL.
256
   ? Since you are adding or removing this field, you MUST specify a default
257
   ? value to use for existing rows. Would you like to:
258
   ?  1. Quit now, and add a default to the field in models.py
259
   ?  2. Specify a one-off value to use for existing columns now
260
   ? Please select a choice: 1
261

    
262
Data migrations
263
***************
264

    
265
To do data migration as well, for example rename a field, use the
266
``datamigration`` management command.
267

    
268
In contrast with ``schemamigration``, to perform complex data migration, we
269
must write the script manually. The process is the following:
270

    
271
1. Introduce the changes in the code and fixtures (initial data).
272
2. Execute:
273

    
274
   .. code-block:: console
275

    
276
      $ snf-manage datamigration <migration_name_here>
277

    
278
   For example:
279

    
280
   .. code-block:: console
281

    
282
      $ ./bin/python manage.py datamigration db rename_credit_wallet
283
      Created 0003_rename_credit_wallet.py.
284

    
285
3. Edit the generated script. It contains two methods, ``forwards`` and
286
   ``backwards``.
287

    
288
   For database operations (column additions, alter tables etc), use the
289
   South database API (http://south.aeracode.org/docs/databaseapi.html).
290

    
291
   To access the data, use the database reference (``orm``) provided as
292
   parameter in ``forwards``, ``backwards`` method declarations in the
293
   migration script. For example:
294

    
295
   .. code-block:: python
296

    
297
      class Migration(DataMigration):
298

    
299
      def forwards(self, orm):
300
          orm.SynnefoUser.objects.all()
301

    
302
4. To migrate the database to the latest version, run:
303

    
304
   .. code-block:: console     
305
     
306
      $ snf-manage migrate db
307

    
308
   To see which migrations are applied:
309

    
310
   .. code-block:: console
311

    
312
      $ snf-manage migrate db --list
313

    
314
      db
315
        (*) 0001_initial
316
        (*) 0002_auto__add_field_synnefouser_new_south_test_field
317
        (*) 0003_rename_credit_wallet
318

    
319
.. seealso::
320
    More information and more thorough examples can be found in the South web site,
321
    http://south.aeracode.org/
322

    
323
Test coverage
324
-------------
325

    
326
.. warning:: This section may be out of date.
327

    
328
In order to get code coverage reports you need to install django-test-coverage
329

    
330
.. code-block:: console
331

    
332
   $ pip install django-test-coverage
333

    
334
Then configure the test runner inside Django settings:
335

    
336
.. code-block:: python
337

    
338
   TEST_RUNNER = 'django-test-coverage.runner.run_tests'
339

    
340

    
341
Internationalization
342
--------------------
343

    
344
This section describes how to translate static strings in Django projects:
345

    
346
0. From our project's base, we add directory locale
347

    
348
   .. code-block:: console
349
   
350
      $ mkdir locale
351
   
352
then we add on the settings.py the language code e.g.,
353

    
354
   .. code-block:: python
355
   
356
      LANGUAGES = (
357
          ('el', u'Greek'),
358
          ('en', u'English'),)
359
   
360
1. For each language we want to add, we run ``makemessages`` from the project's
361
   base:
362

    
363
   .. code-block:: python
364

    
365
      $ ./bin/django-admin.py makemessages -l el -e html,txt,py
366
      (./bin/django-admin.py makemessages -l el -e html,txt,py --ignore=lib/\*)
367

    
368
   This will add the Greek language, and we specify that :file:`*.html`,
369
   :file:`*.txt` and :file:`*.py` files contain translatable strings
370

    
371
2. We translate our strings:
372

    
373
   On :file:`.py` files, e.g., :file:`views.py`, first import ``gettext``:
374
   
375
   .. code-block:: python
376

    
377
      from django.utils.translation import gettext_lazy as _
378

    
379
   Then every ``string`` to be translated becomes:  ``_('string')``
380
   e.g.:
381

    
382
   .. code-block:: python
383

    
384
      help_text=_("letters and numbers only"))
385
      'title': _('Ubuntu 10.10 server 64bit'),
386

    
387
   On django templates (``html`` files), on the beggining of the file we add
388
   ``{% load i18n %}`` then rewrite every string that needs to be translated,
389
   as ``{% trans "string" %}``. For example: ``{% trans "Home" %}``
390

    
391
3. When all strings have been translated, run:
392

    
393
   .. code-block:: console
394

    
395
      $ django-admin.py makemessages -l el -e html,txt,py
396

    
397
   processing language ``el``. This creates (or updates) the :file:`po` file
398
   for the Greek language. We run this command each time we add new strings to
399
   be translated.  After that, we can translate our strings in the :file:`po`
400
   file (:file:`locale/el/LC_MESSAGES/django.po`)
401

    
402
4. When the :file:`po` file is ready, run
403
    
404
   .. code-block:: console
405

    
406
      $ ./bin/django-admin.py compilemessages
407

    
408
   This compiles the ``po`` files to ``mo``. Our strings will appear translated
409
   once we change the language (e.g., from a dropdown menu in the page)
410

    
411
.. seealso::
412
    http://docs.djangoproject.com/en/dev/topics/i18n/internationalization/
413

    
414

    
415
Building source packages
416
------------------------
417

    
418
.. warning:: This section may be out of date.
419

    
420
To create a python package from the Synnefo source code run
421

    
422
.. code-block:: bash
423

    
424
    $ cd snf-cyclades-app
425
    $ python setup.py sdist
426

    
427
this command will create a ``tar.gz`` python source package inside ``dist`` directory.
428

    
429

    
430
Building documentation
431
----------------------
432

    
433
Make sure you have ``sphinx`` installed.
434

    
435
.. code-block:: bash
436
    
437
    $ cd snf-cyclades-app/docs
438
    $ make html
439

    
440
.. note::
441

    
442
   The theme define in the Sphinx configuration file ``conf.py`` is ``nature``,
443
   not available in the version of Sphinx shipped with Debian Squeeze. Replace
444
   it with ``default`` to build with a Squeeze-provided Sphinx.
445

    
446
html files are generated in the ``snf-cyclades-app/docs/_build/html`` directory.
447

    
448

    
449
Continuous integration with Jenkins
450
-----------------------------------
451
.. warning:: This section may be out of date.
452

    
453
Preparing a GIT mirror
454
**********************
455

    
456
Jenkins cannot currently work with Git over encrypted HTTP. To solve this
457
problem we currently mirror the central Git repository locally on the jenkins
458
installation machine. To setup such a mirror do the following:
459

    
460
edit .netrc::
461

    
462
    machine code.grnet.gr
463
    login accountname
464
    password accountpasswd
465

    
466
Create the mirror::
467

    
468
    git clone --mirror https://code.grnet.gr/git/synnefo synnefo
469

    
470
Setup cron to pull from the mirror periodically. Ideally, Git mirror updates
471
should run just before Jenkins jobs check the mirror for changes::
472

    
473
    4,14,24,34,44,54 * * * * cd /path/to/mirror && git fetch && git remote prune origin
474

    
475
Jenkins setup
476
*************
477

    
478
The following instructions will setup Jenkins to run synnefo tests with the
479
SQLite database. To run the tests on MySQL and/or Postgres, step 5 must be
480
replicated. Also, the correct configuration file must be copied (line 6 of the
481
build script).
482

    
483
1. Install and start Jenkins. On Debian Squeeze:
484

    
485
   wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | apt-key add -
486
   echo "deb http://pkg.jenkins-ci.org/debian binary/" >>/etc/apt/sources.list
487
   echo "deb http://ppa.launchpad.net/chris-lea/zeromq/ubuntu lucid main" >> /etc/apt/sources.list
488
   sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C7917B12  
489
   sudo apt-get update
490
   sudo apt-get install jenkins
491

    
492
   Also install the following packages:
493

    
494
   apt-get install python-virtualenv libcurl3-gnutls libcurl3-gnutls-dev
495
                   uuid-dev libmysqlclient-dev libpq-dev libsqlite-dev
496
                   python-dev libzmq-dev
497

    
498
2. After Jenkins starts, go to
499

    
500
   http://$HOST:8080/pluginManager/
501

    
502
   and install the following plug-ins at
503

    
504
   -Jenkins Cobertura Plugin
505
   -Jenkins Email Extension Plugin
506
   -Jenkins GIT plugin
507
   -Jenkins SLOCCount Plug-in
508
   -Hudson/Jenkins Violations plugin
509

    
510
3. Configure the Jenkins user's Git details:
511
   su jenkins
512
   git config --global user.email "buildbot@lists.grnet.gr"
513
   git config --global user.name "Buildbot"
514

    
515
4. Make sure that all system-level dependencies specified in README.develop
516
   are correctly installed
517

    
518
5. Create a new "free-style software" job and set the following values::
519

    
520
    Project name: synnefo
521
    Source Code Management: Git
522
    URL of repository: Jenkins Git does not support HTTPS for checking out directly
523
                        from the repository. The temporary solution is to checkout
524
                        with a cron script in a directory and set the checkout path
525
                        in this field
526
    Branches to build: master and perhaps others
527
    Git->Advanced->Local subdirectory for repo (optional): synnefo
528
    Git->Advanced->Prune remote branches before build: check
529
    Repository browser: redmineweb,
530
                         URL: https://code.grnet.gr/projects/synnefo/repository/
531
    Build Triggers->Poll SCM: check
532
                     Schedule: # every five minutes
533
                   0,5,10,15,20,25,30,35,40,45,50,55 * * * * 
534

    
535
    Build -> Add build step-> Execute shell
536

    
537
    Command::
538

    
539
        #!/bin/bash -ex
540
        cd synnefo
541
        mkdir -p reports
542
        /usr/bin/sloccount --duplicates --wide --details api util ui logic auth > reports/sloccount.sc
543
        cp conf/ci/manage.py .
544
        if [ ! -e requirements.pip ]; then cp conf/ci/pip-1.2.conf requirements.pip; fi
545
        cat settings.py.dist conf/ci/settings.py.sqlite > settings.py
546
        python manage.py update_ve
547
        python manage.py hudson api db logic 
548

    
549
    Post-build Actions->Publish JUnit test result report: check
550
                         Test report XMLs: synnefo/reports/TEST-*.xml
551

    
552
    Post-build Actions->Publish Cobertura Coverage Report: check
553
                         Cobertura xml report pattern: synnefo/reports/coverage.xml
554

    
555
    Post-build Actions->Report Violations: check
556
                         pylint[XML filename pattern]: synnefo/reports/pylint.report
557

    
558
    Post-build Actions->Publish SLOCCount analysis results
559
                         SLOCCount reports: synnefo/reports/sloccount.sc
560
                         (also, remember to install sloccount at /usr/bin)
561

    
562
.. seealso::
563
    http://sites.google.com/site/kmmbvnr/home/django-hudson-tutorial