Synnefo

DEVELOP.txt - Information on how to setup a development environment. 

Dependencies

------------

Synnefo is written in Python 2.6 and depends on the following Python modules

[package versions confirmed to be compatible are in braces]

- django 1.2 [Django==1.2.4]

- simplejson [simplejson==2.1.3]

- pycurl [pycurl==7.19.0]

- python-dateutil  [python-dateutil==1.4.1]

  WARNING: version python-dateutil==2.0 downloaded by pip known *not* to work

           with Python 2.6

- south [south==0.7.1]

  WARNING: not working with Debian squeeze's default south-0.7-1 package.

- amqplib [amqplib==0.6.1]

also, depending on the database engine of choice, on one of the following:

- MySQL-python [MySQL-python==1.2.3]

- psycopg2 [psycopg2==2.4]

if the invitations application is deployed, the following

dependencies should be installed

- pycrypto==2.1.0

if you want to test the ganeti (FIXME)

to run the user interface tests, selenium must be installed

- selenium [?]

Preparing the development environment

-------------------------------------

1. Prepare the system: The easiest method is to setup a working environment

through virtualenv. Alternatively, you can use your system's package manager

to install the dependencies (e.g. Macports has them all).

* On Snow Leopard and linux (64-bit), you have to set the following environment

  variable for pip to compile the dependencies correctly.

	$ export ARCHFLAGS="-arch x86_64"

* On Ubuntu, a few more packages must be installed before installing the

  prerequisite Python libraries

	$ sudo aptitude install libcurl3-gnutls libcurl3-gnutls-dev uuid-dev 

2. Checkout the code and install the Python prerequisites. This assumes

that python is already installed on the host.

    $ sudo easy_install virtualenv

    $ git clone https://user@code.grnet.gr/git/synnefo synnefo

    $ virtualenv --python=python2.6 synnefo --no-site-packages

    ...

    $ cd synnefo

    $ ./bin/pip install <list_of_prerequisites>

3. At this point you should have all required dependencies installed. Now you

have to select a database engine. The choices are: postgres, mysql and sqlite.

- SQLite

The python sqlite driver is available by default with Python so no additional 

configuration is required. Also, most self-respecting systems have the sqlite 

library installed by default.   

- MySQL

MySQL must be installed first

* Ubuntu - Debian

	$ sudo apt-get install libmysqlclient-dev

* MacPorts

	$sudo port install mysql5

Install the MySQL python library

	$ bin/pip install MySQL-python

Note: On MacOSX with Mysql install from MacPorts the above command will fail

complaining that it cannot find the mysql_config command. Do the following and

restart the installation

	$ echo "mysql_config = /opt/local/bin/mysql_config5" >> ./build/MySQL-python/site.cfg

Configure a MySQL db/account for synnefo

	$ mysql -u root -p

	mysql> create database synnefo;

	mysql> show databases;

	mysql> GRANT ALL on synnefo.* TO username IDENTIFIED BY 'password';

- Postgres

# Ubuntu - Debian

	$ sudo apt-get install postgresql-8.4 libpq-dev

# MacPorts

	$ sudo port install postgresql84

Install the postgres Python library

	$ bin/pip install psycopg2

Configure a postgres db/account for synnefo

	Become the postgres user, connect to PostgreSQL:

	$ sudo su - postgres 

	$ psql

	Run the following commands:

	DROP DATABASE synnefo;

	DROP USER username;

	CREATE USER username WITH PASSWORD 'password';

	CREATE DATABASE synnefo;

	GRANT ALL PRIVILEGES ON DATABASE synnefo TO username;

	ALTER DATABASE synnefo OWNER TO username;

	ALTER USER username CREATEDB;

The last line enables the newly created user to create own databases. This is

needed for Django to create and drop the test_synnefo database for unit

testing.

4. At this point you should have a working DB. Now configure Django to access it: 

Copy the default configuration file 

    $ cp settings.py.dist settings.py

and then copy/edit according to the database used:  

- SQLite

	PROJECT_PATH = os.path.dirname(os.path.abspath(__file__)) + '/'

	DATABASES = {

	    'default': {

		'ENGINE': 'django.db.backends.sqlite3',

		'NAME': PROJECT_PATH + 'synnefo.db' #WARN: This must be an absolute path

	    }

	}

- MySQL

	DATABASES = {

	    'default': {

		'ENGINE': 'django.db.backends.mysql', 

		'NAME': 'synnefo',

		'USER': 'USERNAME',

		'PASSWORD': 'PASSWORD',

		'HOST': 'HOST',

		'PORT': 'PORT',

		'OPTIONS': {

		    'init_command': 'SET storage_engine=INNODB',

		}

	    }

	}

- Postgres    

    DATABASES = {

	    'default': {

		'ENGINE': 'django.db.backends.postgresql_psycopg2',

		'NAME': 'DATABASE',

		'USER': 'USERNAME',

		'PASSWORD': 'PASSWORD',

		'HOST': 'HOST',

		'PORT': 'PORT',

	    }

	}

5. Try it out. The following command will attempt to connect to the DB and 

print out DDL statements. It should not fail.

	$ ./bin/python manage.py sql db

6. Create the DB and load initial data:

	$ ./bin/python manage.py syncdb

	$ ./bin/python manage.py migrate db

	$ ./bin/python manage.py loaddata db/fixtures/flavors.json

	$ ./bin/python manage.py loaddata db/fixtures/images.json

The following fixtures can be loaded optionally depending on

testing requirements:

	$ ./bin/python manage.py loaddata db/fixtures/vms.json

	$ ./bin/python manage.py loaddata db/fixtures/disks.json

7. Set the BACKEND_PREFIX_ID variable to some unique prefix, e.g. your commit

   username in settings.py. Several functional conventions within the system

   require this variable to include a dash at its end (e.g. snf-)

8. Fix the AMQP-specific settings based on the selected BACKEND_PREFIX_ID.

   The fix_amqp_settings() function is called once at the end of

   settings.py.dist, you must call it again if you change BACKEND_PREFIX_ID

   at some later point.

   Also make sure that if /var/log/synnefo/dispatcher.log exists, it has proper

   permissions. If not, /var/log/synnefo/ should be writable by the dispatcher.

9. Start the system

    $ ./bin/python logic/dispatcher.py  # DB synch daemon

    $ ./bin/python manage.py runserver  # Django

10. (Hopefully) Done

South Database Migrations

------------------------

* Initial Migration

First, remember to add the south app to settings.py (it is already included in

the settings.py.dist).

To initialise south migrations in your database the following commands must be

executed:

    $ ./bin/python manage.py syncdb       # Create / update the database with the south tables

    $ ./bin/python manage.py migrate db   # Perform migration in the database

Note that syncdb will create the latest models that exist in the db app, so

some migrations may fail.  If you are sure a migration has already taken place

you must use the "--fake" option, to apply it.

For example:

    $ ./bin/python manage.py migrate db 0001 --fake

To be sure that all migrations are applied type:

    $ ./bin/python manage.py migrate db --list

All starred migrations are applied.

Remember, the migration is performed mainly for the data, not for the database

schema. If you do not want to migrate the data, a syncdb and fake migrations

for all the migration versions will suffice.

* Schema migrations:

Do not use the syncdb management command. It can only be used

the first time and/or if you drop the database and must recreate it from

scratch.

See "Initial Migration" section.

Every time you make changes to the database and data migration is not required

(WARNING: always perform this with extreme care):

    $ ./bin/python manage.py schemamigration db --auto

The above will create the migration script. Now this must be applied to the

live database.

    $ ./bin/python migrate db

Consider this example (adding a field to the SynnefoUser model):

    bkarak@nefarian:~/devel/synnefo$ ./bin/python manage.py schemamigration db --auto

     + Added field new_south_test_field on db.SynnefoUser

     Created 0002_auto__add_field_synnefouser_new_south_test_field.py.

  You can now apply this migration with: ./manage.py migrate db

    $ ./manage.py migrate db

     Running migrations for db:

     - Migrating forwards to 0002_auto__add_field_synnefouser_new_south_test_field.

     > db:0002_auto__add_field_synnefouser_new_south_test_field

     - Loading initial data for db.

    Installing json fixture 'initial_data' from '/home/bkarak/devel/synnefo/../synnefo/db/fixtures'.

    Installed 1 object(s) from 1 fixture(s)

South needs some extra definitions to the model to preserve and migrate the

existing data, for example, if we add a field in a model, we should declare its

default value. If not, South will propably fail, after indicating the error.

    $ ./bin/python manage.py schemamigration db --auto

     ? The field 'SynnefoUser.new_south_field_2' does not have a default specified, yet is NOT NULL.

     ? Since you are adding or removing this field, you MUST specify a default

     ? value to use for existing rows. Would you like to:

     ?  1. Quit now, and add a default to the field in models.py

     ?  2. Specify a one-off value to use for existing columns now

     ? Please select a choice: 1

* Data migrations:

If we need to do data migration as well, for example rename a field, we use tha

'datamigration' management command.

In contrast with schemamigration, to perform complex data migration, we must

write the script manually. The process is the following:

    1. Introduce the changes in the code and fixtures (initial data).

    2. Execute:

    $ ./bin/python manage.py datamigration <migration_name_here>

    For example:

    $ ./bin/python manage.py datamigration db rename_credit_wallet

    Created 0003_rename_credit_wallet.py.

    3. We edit the generated script. It contains two methods: forwards and

    backwards.

    For database operations (column additions, alter tables etc) we use the

    South database API (http://south.aeracode.org/docs/databaseapi.html).

    To access the data, we use the database reference (orm) provided as

    parameter in forwards, backwards method declarations in the migration

    script. For example:

    class Migration(DataMigration):

    def forwards(self, orm):

        orm.SynnefoUser.objects.all()

    4. To migrate the database to the latest version, we execute:

    ./manage.py migrate db

To see which migrations are applied:

    $ ./bin/python manage.py migrate db --list

      db

        (*) 0001_initial

        (*) 0002_auto__add_field_synnefouser_new_south_test_field

        (*) 0003_rename_credit_wallet

More information and more thorough examples can be found in the South web site.

http://south.aeracode.org/

UI Testing

----------

The functional ui tests require the Selenium server and the synnefo app to 

be running.

    $ wget http://selenium.googlecode.com/files/selenium-server-standalone-2.0b2.jar

    $ java -jar selenium-server-standalone-2.0b2.jar &

    $ ./bin/python manage.py runserver &

    $ ./bin/python manage.py test ui

Test coverage

-------------

In order to get code coverage reports you need to install django-test-coverage

   $ ./bin/pip install django-test-coverage

Then edit your settings.py and configure the test runner:

   TEST_RUNNER = 'django-test-coverage.runner.run_tests'