Synnefo

.. _asterias-developer-guide:

===============

Developer Guide

===============

This is the asterias developer guide.

It is intended for developers, wishing to implement new functionality

inside :ref:`asterias <asterias>`.

It assumes thorough familiarity with the :ref:`asterias-admin-guide`.

Building a dev environment

--------------------------

virtualenv

**********

The easiest method to deploy a development environment is using

:command:`virtualenv`. Alternatively, you can use your system's package manager

to install any dependencies of synnefo components (e.g. Macports has them all).

   .. code-block:: console

      $ virtualenv ~/synnefo-env

      $ . ~/synnefo-env/bin/activate

      (synnefo-env)$ 

Any :command:`pip` commands executed from now on, affect the ``synnefo-env``

virtualenv.

* It is also recommended to install development helpers:

  .. code-block:: console

     (synnefo-env)$ pip install django_extensions

* Create a custom settings directory for :ref:`snf-common <snf-common>` and set

  the ``SYNNEFO_SETTINGS_DIR`` environment variable to use development-specific

  file:`*.conf` files inside this directory.

  (synnefo-env)$ mkdir ~/synnefo-settings-dir

  (synnefo-env)$ export SYNNEFO_SETTINGS_DIR=~/synnefo-settings-dir

  Insert your custom settings in a file such as :file:`$SYNNEFO_SETTINGS_DIR/99-local.conf`:

  .. code-block:: python

        # uncomment this if have django-extensions installed (pip install django_extensions)

        #INSTALLED_APPS = list(INSTALLED_APPS) + ['django_extensions']

        DEV_PATH = os.path.abspath(os.path.dirname(__file__))

        DATABASES['default']['NAME'] = os.path.join(DEV_PATH, "synnefo.sqlite")

        # development rabitmq configuration

        RABBIT_HOST = "<RabbitMQ_host>"

        RABBIT_USERNAME = "<RabbitMQ_username>"

        RABBIT_PASSWORD = "<RabbitMQ_password>"

        RABBIT_VHOST = "/"

        # development ganeti settings

        GANETI_MASTER_IP = "<Ganeti_master_IP>"

        GANETI_CLUSTER_INFO = (GANETI_MASTER_IP, 5080, "<username>", "<password>")

        GANETI_CREATEINSTANCE_KWARGS['disk_template'] = 'plain'

        # This prefix gets used when determining the instance names

        # of Synnefo VMs at the Ganeti backend.

        # The dash must always appear in the name!

        BACKEND_PREFIX_ID = "<your_commit_name>-"

        IGNORE_FLAVOR_DISK_SIZES = True

        # do not actually send emails

        # save them as files in /tmp/synnefo-mails

        EMAIL_BACKEND = 'django.core.mail.backends.filebased.EmailBackend'

        EMAIL_FILE_PATH = '/tmp/synnefo-mails'

        # for UI developers

        UI_HANDLE_WINDOW_EXCEPTIONS = False

        # allow login using /?test url

        BYPASS_AUTHENTICATION = True 

synnefo source

**************

* Clone the repository of the synnefo software components you wish

  to work on, e.g.:

   .. code-block:: console

     (synnefo-env)$ git clone https://code.grnet.gr/git/synnefo synnefo

* Install the software components you wish to work on inside the

  virtualenv, in development mode:

   .. code-block:: console

      (synnefo-env)$ cd snf-asterias-app

      (synnefo-env)$ python setup.py develop -N

* Initialize database:

   .. code-block:: console

      (synnefo-env)$ snf-manage syndb

      (synnefo-env)$ snf-manage migrate

      (synnefo-env)$ snf-manage loaddata users flavors images

Development tips

****************

* Running a development web server:

  .. code-block:: console

     (synnefo-env)$ snf-manage runserver

  or, if you have the django_extensions and werkzeug packages installed:

  .. code-block:: console

     (synnefo-env)$ snf-manage runserver_plus

* Opening a python console with the synnefo environment initialized:

  .. code-block:: console

     (synnefo-env)$ snf-manage shell

  or, with the django_extensions package installed:

  .. code-block:: console

     (synnefo-env)$ snf-manage shell_plus

South Database Migrations

-------------------------

.. _asterias-dev-initialmigration:

Initial Migration

*****************

To initialize south migrations in your database the following commands must be

executed:

.. code-block:: console

   $ snf-manage syncdb       # Create / update the database with the south tables

   $ snf-manage migrate      # Perform migration in the database

Note that syncdb will create the latest models that exist in the db app, so some

migrations may fail.  If you are sure a migration has already taken place you

must use the ``--fake`` option, to apply it.

For example:

.. code-block:: console

   $ snf-manage migrate db 0001 --fake

To be sure that all migrations are applied use:

.. code-block:: console

   $ snf-manage migrate db --list

All starred migrations are applied.

Schema migrations

*****************

Do not use the syncdb management command. It can only be used the first time

and/or if you drop the database and must recreate it from scratch. See

:ref:`asterias-dev-initialmigration`.

Every time you make changes to the database and data migration is not required

(WARNING: always perform this with extreme care):

.. code-block:: console

   $ snf-manage schemamigration db --auto

The above will create the migration script. Now this must be applied to the live

database:

.. code-block:: console

   $ snf-manage migrate db

Consider this example (adding a field to the ``SynnefoUser`` model):

.. code-block:: console

   $ ./bin/python manage.py schemamigration db --auto

   + Added field new_south_test_field on db.SynnefoUser

   Created 0002_auto__add_field_synnefouser_new_south_test_field.py.

You can now apply this migration with:

.. code-block:: console

   $ ./manage.py migrate db

   Running migrations for db:

   - Migrating forwards to 0002_auto__add_field_synnefouser_new_south_test_field.

   > db:0002_auto__add_field_synnefouser_new_south_test_field

   - Loading initial data for db.

   Installing json fixture 'initial_data' from '/home/bkarak/devel/synnefo/../synnefo/db/fixtures'.

   Installed 1 object(s) from 1 fixture(s)

South needs some extra definitions to the model to preserve and migrate the

existing data, for example, if we add a field in a model, we should declare its

default value. If not, South will propably fail, after indicating the error:

.. code-block:: console

   $ ./bin/python manage.py schemamigration db --auto

   ? The field 'SynnefoUser.new_south_field_2' does not have a default specified, yet is NOT NULL.

   ? Since you are adding or removing this field, you MUST specify a default

   ? value to use for existing rows. Would you like to:

   ?  1. Quit now, and add a default to the field in models.py

   ?  2. Specify a one-off value to use for existing columns now

   ? Please select a choice: 1

Data migrations

***************

To do data migration as well, for example rename a field, use the

``datamigration`` management command.

In contrast with ``schemamigration``, to perform complex data migration, we

must write the script manually. The process is the following:

1. Introduce the changes in the code and fixtures (initial data).

2. Execute:

   .. code-block:: console

      $ snf-manage datamigration <migration_name_here>

   For example:

   .. code-block:: console

      $ ./bin/python manage.py datamigration db rename_credit_wallet

      Created 0003_rename_credit_wallet.py.

3. Edit the generated script. It contains two methods, ``forwards`` and

   ``backwards``.

   For database operations (column additions, alter tables etc), use the

   South database API (http://south.aeracode.org/docs/databaseapi.html).

   To access the data, use the database reference (``orm``) provided as

   parameter in ``forwards``, ``backwards`` method declarations in the

   migration script. For example:

   .. code-block:: python

      class Migration(DataMigration):

      def forwards(self, orm):

          orm.SynnefoUser.objects.all()

4. To migrate the database to the latest version, run:

   .. code-block:: console     

      $ snf-manage migrate db

   To see which migrations are applied:

   .. code-block:: console

      $ snf-manage migrate db --list

      db

        (*) 0001_initial

        (*) 0002_auto__add_field_synnefouser_new_south_test_field

        (*) 0003_rename_credit_wallet

.. seealso::

    More information and more thorough examples can be found in the South web site,

    http://south.aeracode.org/

Test coverage

-------------

.. warning:: This section may be out of date.

In order to get code coverage reports you need to install django-test-coverage

.. code-block:: console

   $ pip install django-test-coverage

Then configure the test runner inside Django settings:

.. code-block:: python

   TEST_RUNNER = 'django-test-coverage.runner.run_tests'

Internationalization

--------------------

This section describes how to translate static strings in Django projects:

0. From our project's base, we add directory locale

   .. code-block:: console

      $ mkdir locale

then we add on the settings.py the language code e.g.,

   .. code-block:: python

      LANGUAGES = (

          ('el', u'Greek'),

          ('en', u'English'),)

1. For each language we want to add, we run ``makemessages`` from the project's

   base:

   .. code-block:: python

      $ ./bin/django-admin.py makemessages -l el -e html,txt,py

      (./bin/django-admin.py makemessages -l el -e html,txt,py --ignore=lib/\*)

   This will add the Greek language, and we specify that :file:`*.html`,

   :file:`*.txt` and :file:`*.py` files contain translatable strings

2. We translate our strings:

   On :file:`.py` files, e.g., :file:`views.py`, first import ``gettext``:

   .. code-block:: python

      from django.utils.translation import gettext_lazy as _

   Then every ``string`` to be translated becomes:  ``_('string')``

   e.g.:

   .. code-block:: python

      help_text=_("letters and numbers only"))

      'title': _('Ubuntu 10.10 server 64bit'),

   On django templates (``html`` files), on the beggining of the file we add

   ``{% load i18n %}`` then rewrite every string that needs to be translated,

   as ``{% trans "string" %}``. For example: ``{% trans "Home" %}``

3. When all strings have been translated, run:

   .. code-block:: console

      $ django-admin.py makemessages -l el -e html,txt,py

   processing language ``el``. This creates (or updates) the :file:`po` file

   for the Greek language. We run this command each time we add new strings to

   be translated.  After that, we can translate our strings in the :file:`po`

   file (:file:`locale/el/LC_MESSAGES/django.po`)

4. When the :file:`po` file is ready, run

   .. code-block:: console

      $ ./bin/django-admin.py compilemessages

   This compiles the ``po`` files to ``mo``. Our strings will appear translated

   once we change the language (e.g., from a dropdown menu in the page)

.. seealso::

    http://docs.djangoproject.com/en/dev/topics/i18n/internationalization/

Building source packages

------------------------

.. warning:: This section may be out of date.

To create a python package from the Synnefo source code run

.. code-block:: bash

    $ cd snf-app

    $ python setup.py sdist

this command will create a ``tar.gz`` python source package inside ``dist`` directory.

Building documentation

----------------------

Make sure you have ``sphinx`` installed.

.. code-block:: bash

    $ cd snf-app/docs

    $ make html

.. note::

   The theme define in the Sphinx configuration file ``conf.py`` is ``nature``,

   not available in the version of Sphinx shipped with Debian Squeeze. Replace

   it with ``default`` to build with a Squeeze-provided Sphinx.

html files are generated in the ``snf-app/docs/_build/html`` directory.

Continuous integration with Jenkins

-----------------------------------

.. warning:: This section may be out of date.

Preparing a GIT mirror

**********************

Jenkins cannot currently work with Git over encrypted HTTP. To solve this

problem we currently mirror the central Git repository locally on the jenkins

installation machine. To setup such a mirror do the following:

edit .netrc::

    machine code.grnet.gr

    login accountname

    password accountpasswd

Create the mirror::

    git clone --mirror https://code.grnet.gr/git/synnefo synnefo

Setup cron to pull from the mirror periodically. Ideally, Git mirror updates

should run just before Jenkins jobs check the mirror for changes::

    4,14,24,34,44,54 * * * * cd /path/to/mirror && git fetch && git remote prune origin

Jenkins setup

*************

The following instructions will setup Jenkins to run synnefo tests with the

SQLite database. To run the tests on MySQL and/or Postgres, step 5 must be

replicated. Also, the correct configuration file must be copied (line 6 of the

build script).

1. Install and start Jenkins. On Debian Squeeze:

   wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | apt-key add -

   echo "deb http://pkg.jenkins-ci.org/debian binary/" >>/etc/apt/sources.list

   echo "deb http://ppa.launchpad.net/chris-lea/zeromq/ubuntu lucid main" >> /etc/apt/sources.list

   sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C7917B12  

   sudo apt-get update

   sudo apt-get install jenkins

   Also install the following packages:

   apt-get install python-virtualenv libcurl3-gnutls libcurl3-gnutls-dev

                   uuid-dev libmysqlclient-dev libpq-dev libsqlite-dev

                   python-dev libzmq-dev

2. After Jenkins starts, go to

   http://$HOST:8080/pluginManager/

   and install the following plug-ins at

   -Jenkins Cobertura Plugin

   -Jenkins Email Extension Plugin

   -Jenkins GIT plugin

   -Jenkins SLOCCount Plug-in

   -Hudson/Jenkins Violations plugin

3. Configure the Jenkins user's Git details:

   su jenkins

   git config --global user.email "buildbot@lists.grnet.gr"

   git config --global user.name "Buildbot"

4. Make sure that all system-level dependencies specified in README.develop

   are correctly installed

5. Create a new "free-style software" job and set the following values::

    Project name: synnefo

    Source Code Management: Git

    URL of repository: Jenkins Git does not support HTTPS for checking out directly

                        from the repository. The temporary solution is to checkout

                        with a cron script in a directory and set the checkout path

                        in this field

    Branches to build: master and perhaps others

    Git->Advanced->Local subdirectory for repo (optional): synnefo

    Git->Advanced->Prune remote branches before build: check

    Repository browser: redmineweb,

                         URL: https://code.grnet.gr/projects/synnefo/repository/

    Build Triggers->Poll SCM: check

                     Schedule: # every five minutes

                   0,5,10,15,20,25,30,35,40,45,50,55 * * * * 

    Build -> Add build step-> Execute shell

    Command::

        #!/bin/bash -ex

        cd synnefo

        mkdir -p reports

        /usr/bin/sloccount --duplicates --wide --details api util ui logic auth > reports/sloccount.sc

        cp conf/ci/manage.py .

        if [ ! -e requirements.pip ]; then cp conf/ci/pip-1.2.conf requirements.pip; fi

        cat settings.py.dist conf/ci/settings.py.sqlite > settings.py

        python manage.py update_ve

        python manage.py hudson api db logic 

    Post-build Actions->Publish JUnit test result report: check

                         Test report XMLs: synnefo/reports/TEST-*.xml

    Post-build Actions->Publish Cobertura Coverage Report: check

                         Cobertura xml report pattern: synnefo/reports/coverage.xml

    Post-build Actions->Report Violations: check

                         pylint[XML filename pattern]: synnefo/reports/pylint.report

    Post-build Actions->Publish SLOCCount analysis results

                         SLOCCount reports: synnefo/reports/sloccount.sc

                         (also, remember to install sloccount at /usr/bin)

.. seealso::

    http://sites.google.com/site/kmmbvnr/home/django-hudson-tutorial