DEVELOP.txt - Information on how to setup a development environment. Dependencies ------------ Synnefo is written in Python 2.6 and depends on the following Python modules [package versions confirmed to be compatible are in braces] - django 1.2 [Django==1.2.4] - simplejson [simplejson==2.1.3] - pycurl [pycurl==7.19.0] - python-dateutil [python-dateutil==1.4.1] WARNING: version python-dateutil==2.0 downloaded by pip known *not* to work with Python 2.6 - south [south==0.7.1] - ampqlib [ampqlib==0.6.1] also, depending on the database engine of choice, on one of the following: - MySQL-python [MySQL-python==1.2.3] - psycopg2 [psycopg2==2.4] to run the user interface tests, selenium must be installed - selenium [?] Preparing the development environment ------------------------------------- 1. Prepare the system The easiest method is to setup a working environment through virtualenv. Alternatively, you can use your system's package manager to install the dependencies (e.g. Macports has them all). *On Snow Leopard and linux (64-bit), you have to set the following environment variable for pip to compile the dependencies correctly. $export ARCHFLAGS="-arch x86_64" *On Ubuntu, a few more packages must be installed before installing the prerequisite Python libraries $sudo aptitude install libcurl3-gnutls libcurl3-gnutls-dev uuid-dev 2. Checkout the code and install the Python prerequisites. This assumes that python is already installed on the host. $ sudo easy_install virtualenv $ git clone https://user@code.grnet.gr/git/synnefo synnefo $ virtualenv --python=python2.6 synnefo --no-site-packages ... $ cd synnefo $ ./bin/pip install 3. At this point you should have all required dependencies installed. Now you have to select a database engine. The choices are: postgres, mysql and sqlite. -SQLite The python sqlite driver is available by default with Python so no additional configuration is required. Also, most self-respecting systems have the sqlite library installed by default. -MySQL MySQL must be installed first *Ubuntu - Debian $sudo apt-get install libmysqlclient-dev *MacPorts $sudo port install mysql5 Install the MySQL python library $ bin/pip install MySQL-python Note: On MacOSX with Mysql install from MacPorts the above command will fail complaining that it cannot find the mysql_config command. Do the following and restart the installation $ echo "mysql_config = /opt/local/bin/mysql_config5" >> ./build/MySQL-python/site.cfg Configure a MySQL db/account for synnefo $ mysql -u root -p mysql> create database synnefo; mysql> show databases; mysql> GRANT ALL on synnefo.* TO username IDENTIFIED BY 'password'; -Postgres #Ubuntu - Debian $ sudo apt-get install postgresql-8.4 libpq-dev #MacPorts $ sudo port install postgresql84 Install the postgres Python library $ bin/pip install psycopg2 Configure a postgres db/account for synnefo Become the postgres user, connect to PostgreSQL: $ sudo su - postgres $ psql Run the following commands: DROP DATABASE synnefo; DROP USER username; CREATE USER username WITH PASSWORD 'password'; CREATE DATABASE synnefo; GRANT ALL PRIVILEGES ON DATABASE synnefo TO username; ALTER DATABASE synnefo OWNER TO username; ALTER USER username CREATEDB; The last line enables the newly created user to create own databases. This is needed for Django to create and drop the test_synnefo database for unit testing. 4. At this point you should have a working DB. Now configure Django to access it: Copy the default configuration file $ cp settings.py.dist settings.py and then copy/edit according to the database used: -SQLite PROJECT_PATH = os.path.dirname(os.path.abspath(__file__)) + '/' DATABASES = { 'default': { 'ENGINE': 'django.db.backends.sqlite3', 'NAME': PROJECT_PATH + 'synnefo.db' #WARN: This must be an absolute path } } -MySQL DATABASES = { 'default': { 'ENGINE': 'django.db.backends.mysql', 'NAME': 'synnefo', 'USER': 'USERNAME', 'PASSWORD': 'PASSWORD', 'HOST': 'HOST', 'PORT': 'PORT', 'OPTIONS': { 'init_command': 'SET storage_engine=INNODB', } } } -Postgres DATABASES = { 'default': { 'ENGINE': 'django.db.backends.postgresql_psycopg2', 'NAME': 'DATABASE', 'USER': 'USERNAME', 'PASSWORD': 'PASSWORD', 'HOST': 'HOST', 'PORT': 'PORT', } } 5. Try it out. The following command will attempt to connect to the DB and print out DDL statements. It should not fail. $ ./bin/python manage.py sql db 6. Create the DB and (optionally) load test data $ ./bin/python manage.py syncdb $ ./bin/python manage.py loaddata db/fixtures/flavors.json $ ./bin/python manage.py loaddata db/fixtures/images.json The following fixtures can be loaded optionally depending on testing requirements: $ ./bin/python manage.py loaddata db/fixtures/vms.json $ ./bin/python manage.py loaddata db/fixtures/disks.json 7. Set the BACKEND_PREFIX_ID variable to some unique prefix, e.g. your commit username in settings.py. Several functional conventions within the system require this variable to include a dash at its end (e.g. snf-) 8. Fix the AMQP-specific settings based on the selected BACKEND_PREFIX_ID. The fix_amqp_settings() function is called once at the end of settings.py.dist, you must call it again if you change BACKEND_PREFIX_ID at some later point. 9. Start the system $ ./bin/python logic/dispatcher.py # DB synch daemon $ ./bin/python manage.py runserver # Django 10. (Hopefully) Done South Database Migrations ------------------------ *Initial Migration First, remember to add the south app to settings.py (it is already included in the settings.py.dist). To initialise south migrations in your database the following commands must be executed: $ ./bin/python manage.py syncdb # Create / update the database with the south tables $ ./bin/python manage.py migrate db # Perform migration in the database Note that syncdb will create the latest models that exist in the db app, so some migrations may fail. If you are sure a migration has already taken place you must use the "--fake" option, to apply it. For example: $ ./bin/python manage.py migrate db 0001 --fake To be sure that all migrations are applied type: $ ./bin/python manage.py migrate db --list All starred migrations are applied. Remember, the migration is performed mainly for the data, not for the database schema. If you do not want to migrate the data, a syncdb and fake migrations for all the migration versions will suffice. *Schema migrations: Do not use the syncdb management command. It can only be used the first time and/or if you drop the database and must recreate it from scratch. See "Initial Migration" section. Each time you make changes to the database and data migration is not required (WARNING: always perform this with extreme care): $ ./bin/python schemamigration db --auto The above will create the migration script. Now this must be applied to the live database. $ ./bin/python migrate db Consider this example (adding a field to the SynnefoUser model): bkarak@nefarian:~/devel/synnefo$ ./bin/python manage.py schemamigration db --auto + Added field new_south_test_field on db.SynnefoUser Created 0002_auto__add_field_synnefouser_new_south_test_field.py. You can now apply this migration with: ./manage.py migrate db $ ./manage.py migrate db Running migrations for db: - Migrating forwards to 0002_auto__add_field_synnefouser_new_south_test_field. > db:0002_auto__add_field_synnefouser_new_south_test_field - Loading initial data for db. Installing json fixture 'initial_data' from '/home/bkarak/devel/synnefo/../synnefo/db/fixtures'. Installed 1 object(s) from 1 fixture(s) South needs some extra definitions to the model to preserve and migrate the existing data, for example, if we add a field in a model, we should declare its default value. If not, South will propably fail, after indicating the error. $ ./bin/python manage.py schemamigration db --auto ? The field 'SynnefoUser.new_south_field_2' does not have a default specified, yet is NOT NULL. ? Since you are adding or removing this field, you MUST specify a default ? value to use for existing rows. Would you like to: ? 1. Quit now, and add a default to the field in models.py ? 2. Specify a one-off value to use for existing columns now ? Please select a choice: 1 *Data migrations: If we need to do data migration as well, for example rename a field, we use tha 'datamigration' management command. In contrast with schemamigration, to perform complex data migration, we must write the script manually. The process is the following: 1. Introduce the changes in the code and fixtures (initial data). 2. Execute: $ ./bin/python manage.py datamigration For example: $ ./bin/python manage.py datamigration db rename_credit_wallet Created 0003_rename_credit_wallet.py. 3. We edit the generated script. It contains two methods: forwards and backwards. For database operations (column additions, alter tables etc) we use the South database API (http://south.aeracode.org/docs/databaseapi.html). To access the data, we use the database reference (orm) provided as parameter in forwards, backwards method declarations in the migration script. For example: class Migration(DataMigration): def forwards(self, orm): orm.SynnefoUser.objects.all() 4. To migrate the database to the latest version, we execute: ./manage.py migrate db To see which migrations are applied: $ ./bin/python manage.py migrate db --list db (*) 0001_initial (*) 0002_auto__add_field_synnefouser_new_south_test_field (*) 0003_rename_credit_wallet More information and more thorough examples can be found in the South web site. http://south.aeracode.org/ UI Testing ---------- The functional ui tests require the Selenium server and the synnefo app to be running. $ wget http://selenium.googlecode.com/files/selenium-server-standalone-2.0b2.jar $ java -jar selenium-server-standalone-2.0b2.jar & $ ./bin/python manage.py runserver & $ ./bin/python manage.py test ui Test coverage ------------- In order to get code coverage reports you need to install django-test-coverage $ ./bin/pip install django-test-coverage Then edit your settings.py and configure the test runner: TEST_RUNNER = 'django-test-coverage.runner.run_tests'