Synnefo

Administration

==============

This file contains notes related to administration of a working Synnefo

deployment. This document should be read *after* README.deploy, which contains

step-by-step Synnefo deployment instructions.

Database

--------

MySQL: manage.py dbshell seems to ignore the setting of 'init_command'

       in settings.DATABASES

Reconciliation mechanism

------------------------

On certain occasions, such as a Ganeti or RabbitMQ failure, the VM state in the

system's database may differ from that in the Ganeti installation. The

reconciliation process is designed to bring the system's database in sync with

what Ganeti knows about each VM, and is able to detect the following three

conditions:

 * Stale DB servers without corresponding Ganeti instances

 * Orphan Ganeti instances, without corresponding DB entries

 * Out-of-sync operstate for DB entries wrt to Ganeti instances

The reconciliation mechanism runs as a management command, e.g., as follows:

[PYTHONPATH needs to contain the parent of the synnefo Django project

directory]:

/srv/synnefo$ export PYTHONPATH=/srv:$PYTHONPATH

vkoukis@dev67:~/synnefo [reconc]$ ./manage.py reconcile --detect-all -v 2

Please see ./manage.py reconcile --help for all the details.

The administrator can also trigger reconciliation of operating state manually,

by issuing a Ganeti OP_INSTANCE_QUERY_DATA command on a Synnefo VM, using

gnt-instance info.

Logging

-------

Logging in Synnefo is using Python's logging module. The module is configured

using dictionary configuration, whose format is described here:

http://docs.python.org/release/2.7.1/library/logging.html#logging-config-dictschema

Note that this is a feature of Python 2.7 that we have backported for use in

Python 2.6.

The logging configuration dictionary is defined in settings.d/00-logging.conf

and is broken in 4 separate dictionaries:

  * LOGGING is the logging configuration used by the web app. By default all

    loggers fall back to the main 'synnefo' logger. The subloggers can be

    changed accordingly for finer logging control. e.g. To disable debug

    messages from the API set the level of 'synnefo.api' to 'INFO'.

  * DISPATCHER_LOGGING is the logging configuration of the logic/dispatcher.py

    command line tool.

  * SNFADMIN_LOGGING is the logging configuration of the snf-admin tool.

    Consider using matching configuration for snf-admin and the synnefo.admin

    logger of the web app.

Please note the following:

  * As of Synnefo v0.7, by default the Django webapp logs to syslog, the

    dispatcher logs to /var/log/synnefo/dispatcher.log and the console,

    snf-admin logs to the console.

  * Different handlers can be set to different logging levels:

    for example, everything may appear to the console, but only INFO and higher

    may actually be stored in a longer-term logfile.

Admin Tools

-----------

snf-admin is a tool used to perform various administrative tasks. It needs to

be able to access the django database, so the following should be able to import

the Django settings.

Additionally, administrative tasks can be performed via the admin web interface

located in /admin. Only users of type ADMIN can access the admin pages. To change

the type of a user to ADMIN, snf-admin can be used:

   snf-admin user modify 42 --type ADMIN

.. _cyclades-admin-guide:

===================

Administrator Guide

===================

This is the cyclades administrator guide.

It contains instructions on how to download, install and configure

the synnefo components necessary to deploy the Compute Service. It also covers

maintenance issues, e.g., upgrades of existing deployments.

The guide assumes you are familiar with all aspects of downloading, installing

and configuring packages for the Linux distribution of your choice.

Overview

--------

This guide covers the following:

Architecture

    Node types needed for a complete deployment of cyclades,

    and their roles. Throughout this guide, `node` refers to a physical machine

    in the deployment.

Installation

    The installation of services and synnefo software components for a working

    deployment of cyclades, either from source packages or the provided

    packages for Debian Squeeze.

Configuration

    Configuration of the various software components comprising an cyclades

    deployment.

Upgrades/Changelogs

    Upgrades of existing deployments of cyclades to newer versions, associated

    Changelogs.

.. _cyclades-architecture:

Architecture

------------

Nodes in an cyclades deployment belong in one of the following types.

For every type, we list the services that execute on corresponding nodes.

.. _DB_NODE:

DB

**

A node [or more than one nodes, if using an HA configuration], running a DB

engine supported by the Django ORM layer. The DB is the single source of

truth for the servicing of API requests by cyclades.

*Services:* PostgreSQL / MySQL

.. _APISERVER_NODE:

APISERVER

*********

A node running the ``api`` application contained in

:ref:`snf-cyclades-app <snf-cyclades-app>`. Any number of

:ref:`APISERVER <APISERVER_NODE>` nodes

can be used, in a load-balancing configuration, without any

special consideration. Access to a common DB ensures consistency.

*Services:* Web server, vncauthproxy

.. _QUEUE_NODE:

QUEUE

*****

A node running the RabbitMQ software, which provides AMQP functionality. More

than one :ref:`QUEUE <QUEUE_NODE>` nodes may be deployed, in an HA

configuration. Such deployments require shared storage, provided e.g., by DRBD.

*Services:* RabbitMQ [rabbitmq-server]

.. _LOGIC_NODE:

LOGIC

*****

A node running the business logic of synnefo, in Django. It dequeues

messages from QUEUE nodes, and provides the context in which business logic

functions run. It uses Django ORM to connect to the common DB and update the

state of the system, based on notifications received from the rest of the

infrastructure, over AMQP.

*Services:* the synnefo logic dispatcher, ``snf-dispatcher``

.. _GANETI_NODES:

.. _GANETI_MASTER:

.. _GANETI_NODE:

GANETI-MASTER and GANETI-NODE

*****************************

A single GANETI-MASTER and a large number of GANETI-NODEs constitute the

Ganeti backend for synnefo, which undertakes all VM management functions.

Any APISERVER can issue commands to the GANETI-MASTER, over RAPI, to effect

changes in the state of the VMs. The GANETI-MASTER runs the Ganeti request

queue.

*Services:*

    * only on :ref:`GANETI-MASTER <GANETI_MASTER>`:

        * the synnefo Ganeti monitoring daemon, ``snf-ganeti-eventd``

        * the synnefo Ganeti hook, ``ganeti/snf-ganeti-hook.py``.

    * on every :ref:`GANETI-NODE <GANETI_NODE>`:

        * a deployment-specific KVM ifup script

        * properly configured :ref:`NFDHCPD <cyclades-nfdhcpd-setup>`

.. _WEBAPP_NODE:

Installation

------------

Installation of cyclades is a two step process:

1. install the external services (prerequisites) on which cyclades depends

2. install the synnefo software components associated with cyclades

Prerequisites

*************

.. _cyclades-install-ganeti:

Ganeti installation

```````````````````

Synnefo requires a working Ganeti installation at the backend. Installation

of Ganeti is not covered by this document, please refer to

`ganeti documentation <http://docs.ganeti.org/ganeti/current/html>`_ for all the 

gory details. A successful Ganeti installation concludes with a working 

:ref:`GANETI-MASTER <GANETI_NODES>` and a number of :ref:`GANETI-NODEs <GANETI_NODES>`.

.. _cyclades-install-db:

Database

````````

Database installation is done as part of the

:ref:`snf-webproject <snf-webproject>` component.

.. _cyclades-install-rabbitmq:

RabbitMQ 

````````

RabbitMQ is used as a generic message broker for cyclades. It should be

installed on two seperate :ref:`QUEUE <QUEUE_NODE>` nodes in a high availability

configuration as described here:

    http://www.rabbitmq.com/pacemaker.html

After installation, create a user and set its permissions:

.. code-block:: console

    $ rabbitmqctl add_user <username> <password>

    $ rabbitmqctl set_permissions -p / <username>  "^.*" ".*" ".*"

The values set for the user and password must be mirrored in the

``RABBIT_*`` variables in your settings, as managed by

:ref:`snf-common <snf-common>`.

.. todo:: Document an active-active configuration based on the latest version

   of RabbitMQ.

.. _cyclades-install-vncauthproxy:

vncauthproxy

````````````

To support OOB console access to the VMs over VNC, the vncauthproxy

daemon must be running on every :ref:`APISERVER <APISERVER_NODE>` node.

.. note:: The Debian package for vncauthproxy undertakes all configuration

   automatically.

Download and install the latest vncauthproxy from its own repository,

at `https://code.grnet.gr/git/vncauthproxy`, or a specific commit:

.. code-block:: console

    $ bin/pip install -e git+https://code.grnet.gr/git/vncauthproxy@INSERT_COMMIT_HERE#egg=vncauthproxy

Create ``/var/log/vncauthproxy`` and set its permissions appropriately.

Alternatively, build and install Debian packages.

.. code-block:: console

    $ git checkout debian

    $ dpkg-buildpackage -b -uc -us

    # dpkg -i ../vncauthproxy_1.0-1_all.deb

.. warning::

    **Failure to build the package on the Mac.**

    ``libevent``, a requirement for gevent which in turn is a requirement for

    vncauthproxy is not included in `MacOSX` by default and installing it with

    MacPorts does not lead to a version that can be found by the gevent

    build process. A quick workaround is to execute the following commands::

        $ cd $SYNNEFO

        $ sudo pip install -e git+https://code.grnet.gr/git/vncauthproxy@5a196d8481e171a#egg=vncauthproxy

        <the above fails>

        $ cd build/gevent

        $ sudo python setup.py -I/opt/local/include -L/opt/local/lib build

        $ cd $SYNNEFO

        $ sudo pip install -e git+https://code.grnet.gr/git/vncauthproxy@5a196d8481e171a#egg=vncauthproxy

.. todo:: Mention vncauthproxy bug, snf-vncauthproxy, inability to install using pip

.. todo:: kpap: fix installation commands

.. _cyclades-install-nfdhcpd:

NFDHCPD

```````

Setup Synnefo-specific networking on the Ganeti backend.

This part is deployment-specific and must be customized based on the

specific needs of the system administrators.

A reference installation will use a Synnefo-specific KVM ifup script,

NFDHCPD and pre-provisioned Linux bridges to support public and private

network functionality. For this:

Grab NFDHCPD from its own repository (https://code.grnet.gr/git/nfdhcpd),

install it, modify ``/etc/nfdhcpd/nfdhcpd.conf`` to reflect your network

configuration.

Install a custom KVM ifup script for use by Ganeti, as

``/etc/ganeti/kvm-vif-bridge``, on GANETI-NODEs. A sample implementation is

provided under ``/contrib/ganeti-hooks``. Set ``NFDHCPD_STATE_DIR`` to point

to NFDHCPD's state directory, usually ``/var/lib/nfdhcpd``.

.. todo:: soc: document NFDHCPD installation, settle on KVM ifup script

.. _cyclades-install-snfimage:

snf-image

`````````

Install the :ref:`snf-image <snf-image>` Ganeti OS provider for image

deployment.

For :ref:`cyclades <cyclades>` to be able to launch VMs from specified

Images, you need the snf-image OS Provider installed on *all* Ganeti nodes.

Please see `https://code.grnet.gr/projects/snf-image/wiki`_

for installation instructions and documentation on the design

and implementation of snf-image.

Please see `https://code.grnet.gr/projects/snf-image/files`

for the latest packages.

Images should be stored in ``extdump``, or ``diskdump`` format in a directory

of your choice, configurable as ``IMAGE_DIR`` in 

:file:`/etc/default/snf-image`.

synnefo components

******************

You need to install the appropriate synnefo software components on each node,

depending on its type, see :ref:`Architecture <cyclades-architecture>`.

Most synnefo components have dependencies on additional Python packages.

The dependencies are described inside each package, and are setup

automatically when installing using :command:`pip`, or when installing 

using your system's package manager.

Please see the page of each synnefo software component for specific

installation instructions, where applicable.

Install the following synnefo components:

Nodes of type :ref:`APISERVER <APISERVER_NODE>`

    Components

    :ref:`snf-common <snf-common>`,

    :ref:`snf-webproject <snf-webproject>`,

    :ref:`snf-cyclades-app <snf-cyclades-app>`

Nodes of type :ref:`GANETI-MASTER <GANETI_MASTER>` and :ref:`GANETI-NODE <GANETI_NODE>`

    Components

    :ref:`snf-common <snf-common>`,

    :ref:`snf-cyclades-gtools <snf-cyclades-gtools>`

Nodes of type :ref:`LOGIC <LOGIC_NODE>`

    Components

    :ref:`snf-common <snf-common>`,

    :ref:`snf-webproject <snf-webproject>`,

    :ref:`snf-cyclades-app <snf-cyclades-app>`.

Configuration

-------------

This section targets the configuration of the prerequisites for cyclades,

and the configuration of the associated synnefo software components.

synnefo components

******************

cyclades uses :ref:`snf-common <snf-common>` for settings.

Please refer to the configuration sections of

:ref:`snf-webproject <snf-webproject>`,

:ref:`snf-cyclades-app <snf-cyclades-app>`,

:ref:`snf-cyclades-gtools <snf-cyclades-gtools>` for more

information on their configuration.

Ganeti

``````

Set ``GANETI_NODES``, ``GANETI_MASTER_IP``, ``GANETI_CLUSTER_INFO`` based on

your :ref:`Ganeti installation <cyclades-install-ganeti>` and change the

`BACKEND_PREFIX_ID`` setting, using an custom ``PREFIX_ID``.

Database

````````

Once all components are installed and configured,

initialize the Django DB:

.. code-block:: console

   $ snf-manage syncdb

   $ snf-manage migrate

and load fixtures ``{users, flavors, images}``, 

which make the API usable by end users by defining a sample set of users, 

hardware configurations (flavors) and OS images:

.. code-block:: console

   $ snf-manage loaddata /path/to/users.json

   $ snf-manage loaddata flavors

   $ snf-manage loaddata images

.. warning:: 

    Be sure to load a custom users.json and select a unique token 

    for each of the initial and any other users defined in this file. 

    **DO NOT LEAVE THE SAMPLE AUTHENTICATION TOKENS** enabled in deployed

    configurations.

sample users.json file:

.. literalinclude:: ../../synnefo/db/fixtures/users.json

`download <../_static/users.json>`_

RabbitMQ

````````

Change ``RABBIT_*`` settings to match your :ref:`RabbitMQ setup

<cyclades-install-rabbitmq>`.

.. include:: ../../Changelog

Administration Tools User's Guide

=================================

Configure kamaki

----------------

To upload, register or modify an image you will need the **kamaki** tool.

Before proceeding make sure that it is configured properly. Verify that

*image_url*, *storage_url*, and *token* are set as needed:

.. code-block:: console

  kamaki config list

To chage a setting use ``kamaki config set``:

.. code-block:: console

  kamaki config set image_url https://cyclades.example.com/plankton

  kamaki config set storage_url https://pithos.example.com/v1

  kamaki config set token ...

Upload an image

---------------

As a shortcut, you can configure a default account and container that will be

used by the ``kamaki store`` commands:

.. code-block:: console

  kamaki config set storage_account images@example.com

  kamaki config set storage_container images

If the container does not exist, you will have to create it before uploading

any images:

.. code-block:: console

  kamaki store create images

You are now ready to upload an image. You can upload it with a Pithos client

or use kamaki directly:

.. code-block:: console

  kamaki store upload ubuntu.iso

You can use any Pithos client to verify that the image was uploaded correctly.

The full Pithos URL for the previous example will be

``pithos://images@example.com/images/ubuntu.iso``.

Register the image

------------------

To register an image you will need to use the full Pithos URL. To register as

a public image the one from the previous example use:

.. code-block:: console

  kamaki glance register Ubuntu pithos://images@example.com/images/ubuntu.iso --public

The ``--public`` flag is important, if missing the registered image will not

be listed by ``kamaki glance list``.

Use ``kamaki glance register`` with no arguments to see a list of available

options. A more complete example would be the following:

.. code-block:: console

  kamaki glance register Ubuntu pithos://images@example.com/images/ubuntu.iso \

      --public --disk-format diskdump --property kernel=3.1.2

To verify that the image was registered successfully use:

.. code-block:: console

  kamaki glance list -l

.. _cyclades-developer-guide:

===============

Developer Guide

===============

This is the cyclades developer guide.

It is intended for developers, wishing to implement new functionality

inside :ref:`cyclades <cyclades>`.

It assumes thorough familiarity with the :ref:`cyclades-admin-guide`.

Building a dev environment

--------------------------

virtualenv

**********

The easiest method to deploy a development environment is using

:command:`virtualenv`. Alternatively, you can use your system's package manager

to install any dependencies of synnefo components (e.g. Macports has them all).

   .. code-block:: console

      $ virtualenv ~/synnefo-env

      $ source ~/synnefo-env/bin/activate

      (synnefo-env)$ 

Virtualenv creates an isolated python environment to the path you pass as the

first argument of the command. That means that all packages you install using

:command:`pip` or :command:`easy_install` will be placed in

``ENV/lib/pythonX.X/site-packages`` and their console scripts in ``ENV/bin/``.

This allows you to develop against multiple versions of packages that your

software depends on without messing with system python packages that may be

needed in specific versions for other software you have installed on your

system.

* It is also recommended to install development helpers:

  .. code-block:: console

     (synnefo-env)$ pip install django_extensions fabric>=1.3

* Create a custom settings directory for :ref:`snf-common <snf-common>` and set

  the ``SYNNEFO_SETTINGS_DIR`` environment variable to use development-specific

  file:`*.conf` files inside this directory.

  (synnefo-env)$ mkdir ~/synnefo-settings-dir

  (synnefo-env)$ export SYNNEFO_SETTINGS_DIR=~/synnefo-settings-dir

  Insert your custom settings in a file such as :file:`$SYNNEFO_SETTINGS_DIR/99-local.conf`:

  .. code-block:: python

        # uncomment this if have django-extensions installed (pip install django_extensions)

        #INSTALLED_APPS = list(INSTALLED_APPS) + ['django_extensions']

        DEV_PATH = os.path.abspath(os.path.dirname(__file__))

        DATABASES['default']['NAME'] = os.path.join(DEV_PATH, "synnefo.sqlite")

        # development rabitmq configuration

        RABBIT_HOST = "<RabbitMQ_host>"

        RABBIT_USERNAME = "<RabbitMQ_username>"

        RABBIT_PASSWORD = "<RabbitMQ_password>"

        RABBIT_VHOST = "/"

        # development ganeti settings

        GANETI_MASTER_IP = "<Ganeti_master_IP>"

        GANETI_CLUSTER_INFO = (GANETI_MASTER_IP, 5080, "<username>", "<password>")

        GANETI_CREATEINSTANCE_KWARGS['disk_template'] = 'plain'

        # This prefix gets used when determining the instance names

        # of Synnefo VMs at the Ganeti backend.

        # The dash must always appear in the name!

        BACKEND_PREFIX_ID = "<your_commit_name>-"

        IGNORE_FLAVOR_DISK_SIZES = True

        # do not actually send emails

        # save them as files in /tmp/synnefo-mails

        EMAIL_BACKEND = 'django.core.mail.backends.filebased.EmailBackend'

        EMAIL_FILE_PATH = '/tmp/synnefo-mails'

        # for UI developers

        UI_HANDLE_WINDOW_EXCEPTIONS = False

        # allow login using /?test url

        BYPASS_AUTHENTICATION = True 

synnefo source

**************

* Clone the repository of the synnefo software components you wish

  to work on, e.g.:

   .. code-block:: console

     (synnefo-env)$ git clone https://code.grnet.gr/git/synnefo synnefo

     (synnefo-env)$ git clone https://code.grnet.gr/git/pithos pithos

* Install the software components you wish to work on inside the

  virtualenv, in development mode:

   .. code-block:: console

      (synnefo-env)$ cd snf-cyclades-app

      (synnefo-env)$ python setup.py develop -N

* Initialize database:

   .. code-block:: console

      (synnefo-env)$ snf-manage syndb

      (synnefo-env)$ snf-manage migrate

      (synnefo-env)$ snf-manage loaddata users flavors images

Development tips

****************

* Running a development web server:

  .. code-block:: console

     (synnefo-env)$ snf-manage runserver

  or, if you have the ``django_extensions`` and ``werkzeug`` packages installed:

  .. code-block:: console

     (synnefo-env)$ snf-manage runserver_plus

* Opening a python console with the synnefo environment initialized:

  .. code-block:: console

     (synnefo-env)$ snf-manage shell

  or, with the django_extensions package installed:

  .. code-block:: console

     (synnefo-env)$ snf-manage shell_plus

South Database Migrations

-------------------------

.. _cyclades-dev-initialmigration:

Initial Migration

*****************

To initialize south migrations in your database the following commands must be

executed:

.. code-block:: console

   $ snf-manage syncdb --all      # Create / update the database with the south tables

   $ snf-manage migrate --fake    # Perform migration in the database

Note that ``--all`` and ``--fake`` arguments are only needed when you are

initializing your database. If you want to migrate a previously create databse

to the latest db scheme just run the same commands without those arguments.

If you are trying to migrate a database that already contains the changes that

applied from a specific migration script, ``south`` will probably notify you for

inconsistent db scheme, a workaround for that issue is to use ``--fake`` option

for a specific migration.

For example:

.. code-block:: console

   $ snf-manage migrate db 0001 --fake

To be sure that all migrations are applied use:

.. code-block:: console

   $ snf-manage migrate db --list

All starred migrations are applied.

Schema migrations

*****************

Do not use the syncdb management command. It can only be used the first time

and/or if you drop the database and must recreate it from scratch. See

:ref:`cyclades-dev-initialmigration`.

Every time you make changes to the database and data migration is not required

(WARNING: always perform this with extreme care):

.. code-block:: console

   $ snf-manage schemamigration db --auto

The above will create the migration script. Now this must be applied to the live

database:

.. code-block:: console

   $ snf-manage migrate db

Consider this example (adding a field to the ``SynnefoUser`` model):

.. code-block:: console

   $ ./bin/python manage.py schemamigration db --auto

   + Added field new_south_test_field on db.SynnefoUser

   Created 0002_auto__add_field_synnefouser_new_south_test_field.py.

You can now apply this migration with:

.. code-block:: console

   $ ./manage.py migrate db

   Running migrations for db:

   - Migrating forwards to 0002_auto__add_field_synnefouser_new_south_test_field.

   > db:0002_auto__add_field_synnefouser_new_south_test_field

   - Loading initial data for db.

   Installing json fixture 'initial_data' from '/home/bkarak/devel/synnefo/../synnefo/db/fixtures'.

   Installed 1 object(s) from 1 fixture(s)

South needs some extra definitions to the model to preserve and migrate the

existing data, for example, if we add a field in a model, we should declare its

default value. If not, South will propably fail, after indicating the error:

.. code-block:: console

   $ ./bin/python manage.py schemamigration db --auto

   ? The field 'SynnefoUser.new_south_field_2' does not have a default specified, yet is NOT NULL.

   ? Since you are adding or removing this field, you MUST specify a default

   ? value to use for existing rows. Would you like to:

   ?  1. Quit now, and add a default to the field in models.py

   ?  2. Specify a one-off value to use for existing columns now

   ? Please select a choice: 1

Data migrations

***************

To do data migration as well, for example rename a field, use the

``datamigration`` management command.

In contrast with ``schemamigration``, to perform complex data migration, we

must write the script manually. The process is the following:

1. Introduce the changes in the code and fixtures (initial data).

2. Execute:

   .. code-block:: console

      $ snf-manage datamigration <migration_name_here>

   For example:

   .. code-block:: console

      $ ./bin/python manage.py datamigration db rename_credit_wallet

      Created 0003_rename_credit_wallet.py.

3. Edit the generated script. It contains two methods, ``forwards`` and

   ``backwards``.

   For database operations (column additions, alter tables etc), use the

   South database API (http://south.aeracode.org/docs/databaseapi.html).

   To access the data, use the database reference (``orm``) provided as

   parameter in ``forwards``, ``backwards`` method declarations in the

   migration script. For example:

   .. code-block:: python

      class Migration(DataMigration):

      def forwards(self, orm):

          orm.SynnefoUser.objects.all()

4. To migrate the database to the latest version, run:

   .. code-block:: console     

      $ snf-manage migrate db

   To see which migrations are applied:

   .. code-block:: console

      $ snf-manage migrate db --list

      db

        (*) 0001_initial

        (*) 0002_auto__add_field_synnefouser_new_south_test_field

        (*) 0003_rename_credit_wallet

.. seealso::

    More information and more thorough examples can be found in the South web site,

    http://south.aeracode.org/

Test coverage

-------------

.. warning:: This section may be out of date.

In order to get code coverage reports you need to install django-test-coverage

.. code-block:: console

   $ pip install django-test-coverage

Then configure the test runner inside Django settings:

.. code-block:: python

   TEST_RUNNER = 'django-test-coverage.runner.run_tests'

Internationalization

--------------------

This section describes how to translate static strings in Django projects:

0. From our project's base, we add directory locale

   .. code-block:: console

      $ mkdir locale

then we add on the settings.py the language code e.g.,

   .. code-block:: python

      LANGUAGES = (

          ('el', u'Greek'),

          ('en', u'English'),)

1. For each language we want to add, we run ``makemessages`` from the project's

   base:

   .. code-block:: python

      $ ./bin/django-admin.py makemessages -l el -e html,txt,py

      (./bin/django-admin.py makemessages -l el -e html,txt,py --ignore=lib/\*)

   This will add the Greek language, and we specify that :file:`*.html`,

   :file:`*.txt` and :file:`*.py` files contain translatable strings

2. We translate our strings:

   On :file:`.py` files, e.g., :file:`views.py`, first import ``gettext``:

   .. code-block:: python

      from django.utils.translation import gettext_lazy as _

   Then every ``string`` to be translated becomes:  ``_('string')``

   e.g.:

   .. code-block:: python

      help_text=_("letters and numbers only"))

      'title': _('Ubuntu 10.10 server 64bit'),

   On django templates (``html`` files), on the beggining of the file we add

   ``{% load i18n %}`` then rewrite every string that needs to be translated,

   as ``{% trans "string" %}``. For example: ``{% trans "Home" %}``

3. When all strings have been translated, run:

   .. code-block:: console

      $ django-admin.py makemessages -l el -e html,txt,py

   processing language ``el``. This creates (or updates) the :file:`po` file

   for the Greek language. We run this command each time we add new strings to

   be translated.  After that, we can translate our strings in the :file:`po`

   file (:file:`locale/el/LC_MESSAGES/django.po`)

4. When the :file:`po` file is ready, run

   .. code-block:: console

      $ ./bin/django-admin.py compilemessages

   This compiles the ``po`` files to ``mo``. Our strings will appear translated

   once we change the language (e.g., from a dropdown menu in the page)

.. seealso::

    http://docs.djangoproject.com/en/dev/topics/i18n/internationalization/

Building source packages

------------------------

.. warning:: This section may be out of date.

To create a python package from the Synnefo source code run

.. code-block:: bash

    $ cd snf-cyclades-app

    $ python setup.py sdist

this command will create a ``tar.gz`` python source package inside ``dist`` directory.

Building documentation

----------------------

Make sure you have ``sphinx`` installed.

.. code-block:: bash

    $ cd snf-cyclades-app/docs

    $ make html

.. note::

   The theme define in the Sphinx configuration file ``conf.py`` is ``nature``,

   not available in the version of Sphinx shipped with Debian Squeeze. Replace

   it with ``default`` to build with a Squeeze-provided Sphinx.

html files are generated in the ``snf-cyclades-app/docs/_build/html`` directory.

Continuous integration with Jenkins

-----------------------------------

.. warning:: This section may be out of date.

Preparing a GIT mirror

**********************

Jenkins cannot currently work with Git over encrypted HTTP. To solve this

problem we currently mirror the central Git repository locally on the jenkins

installation machine. To setup such a mirror do the following:

edit .netrc::

    machine code.grnet.gr

    login accountname

    password accountpasswd

Create the mirror::

    git clone --mirror https://code.grnet.gr/git/synnefo synnefo

Setup cron to pull from the mirror periodically. Ideally, Git mirror updates

should run just before Jenkins jobs check the mirror for changes::

    4,14,24,34,44,54 * * * * cd /path/to/mirror && git fetch && git remote prune origin

Jenkins setup

*************

The following instructions will setup Jenkins to run synnefo tests with the

SQLite database. To run the tests on MySQL and/or Postgres, step 5 must be

replicated. Also, the correct configuration file must be copied (line 6 of the

build script).

1. Install and start Jenkins. On Debian Squeeze:

   wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | apt-key add -

   echo "deb http://pkg.jenkins-ci.org/debian binary/" >>/etc/apt/sources.list

   echo "deb http://ppa.launchpad.net/chris-lea/zeromq/ubuntu lucid main" >> /etc/apt/sources.list

   sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C7917B12  

   sudo apt-get update

   sudo apt-get install jenkins

   Also install the following packages:

   apt-get install python-virtualenv libcurl3-gnutls libcurl3-gnutls-dev

                   uuid-dev libmysqlclient-dev libpq-dev libsqlite-dev

                   python-dev libzmq-dev

2. After Jenkins starts, go to

   http://$HOST:8080/pluginManager/

   and install the following plug-ins at

   -Jenkins Cobertura Plugin

   -Jenkins Email Extension Plugin

   -Jenkins GIT plugin

   -Jenkins SLOCCount Plug-in

   -Hudson/Jenkins Violations plugin

3. Configure the Jenkins user's Git details:

   su jenkins

   git config --global user.email "buildbot@lists.grnet.gr"

   git config --global user.name "Buildbot"

4. Make sure that all system-level dependencies specified in README.develop

   are correctly installed

5. Create a new "free-style software" job and set the following values::

    Project name: synnefo

    Source Code Management: Git

    URL of repository: Jenkins Git does not support HTTPS for checking out directly

                        from the repository. The temporary solution is to checkout

                        with a cron script in a directory and set the checkout path

                        in this field

    Branches to build: master and perhaps others

    Git->Advanced->Local subdirectory for repo (optional): synnefo

    Git->Advanced->Prune remote branches before build: check

    Repository browser: redmineweb,

                         URL: https://code.grnet.gr/projects/synnefo/repository/

    Build Triggers->Poll SCM: check

                     Schedule: # every five minutes

                   0,5,10,15,20,25,30,35,40,45,50,55 * * * * 

    Build -> Add build step-> Execute shell

    Command::

        #!/bin/bash -ex

        cd synnefo

        mkdir -p reports

        /usr/bin/sloccount --duplicates --wide --details api util ui logic auth > reports/sloccount.sc

        cp conf/ci/manage.py .

        if [ ! -e requirements.pip ]; then cp conf/ci/pip-1.2.conf requirements.pip; fi

        cat settings.py.dist conf/ci/settings.py.sqlite > settings.py

        python manage.py update_ve

        python manage.py hudson api db logic 

    Post-build Actions->Publish JUnit test result report: check

                         Test report XMLs: synnefo/reports/TEST-*.xml

    Post-build Actions->Publish Cobertura Coverage Report: check

                         Cobertura xml report pattern: synnefo/reports/coverage.xml

    Post-build Actions->Report Violations: check

                         pylint[XML filename pattern]: synnefo/reports/pylint.report

    Post-build Actions->Publish SLOCCount analysis results

                         SLOCCount reports: synnefo/reports/sloccount.sc

                         (also, remember to install sloccount at /usr/bin)

.. seealso::

    http://sites.google.com/site/kmmbvnr/home/django-hudson-tutorial

Compute Service (cyclades)

--------------------------

Cyclades is the the synnefo Compute Service and implements OpenStack Compute API v1.1.

Cyclades need the following synnefo components:

Cyclades as a standalone Service

================================

.. todo:: document what does that mean, what are the limitations, how to install.

Cyclades interconnected with other synnefo Services

===================================================

.. todo:: document with which services it can connect, why and how.

Cyclades Guides

===============

.. todo:: document the Compute Service.

.. toctree::

   :maxdepth: 2

   cyclades-admin-guide

   cyclades-api-guide

   cyclades-admin-tools

   cyclades-dev-guide

   cyclades-upgrade