root / README.develop @ 7ca9e930
History | View | Annotate | Download (10.5 kB)
1 |
DEVELOP.txt - Information on how to setup a development environment. |
---|---|
2 |
|
3 |
Dependencies |
4 |
------------ |
5 |
|
6 |
Synnefo is written in Python 2.6 and depends on the following Python modules |
7 |
[package versions confirmed to be compatible are in braces] |
8 |
|
9 |
- django 1.2 [Django==1.2.4] |
10 |
- simplejson [simplejson==2.1.3] |
11 |
- pycurl [pycurl==7.19.0] |
12 |
- python-dateutil [python-dateutil==1.4.1] |
13 |
WARNING: version python-dateutil==2.0 downloaded by pip known *not* to work |
14 |
with Python 2.6 |
15 |
- south [south==0.7.1] |
16 |
- ampqlib [ampqlib==0.6.1] |
17 |
|
18 |
also, depending on the database engine of choice, on one of the following: |
19 |
- MySQL-python [MySQL-python==1.2.3] |
20 |
- psycopg2 [psycopg2==2.4] |
21 |
|
22 |
to run the user interface tests, selenium must be installed |
23 |
- selenium [?] |
24 |
|
25 |
Preparing the development environment |
26 |
------------------------------------- |
27 |
|
28 |
1. Prepare the system |
29 |
The easiest method is to setup a working environment through virtualenv. |
30 |
Alternatively, you can use your system's package manager to install |
31 |
the dependencies (e.g. Macports has them all). |
32 |
|
33 |
*On Snow Leopard and linux (64-bit), you have to set the following environment |
34 |
variable for pip to compile the dependencies correctly. |
35 |
|
36 |
$export ARCHFLAGS="-arch x86_64" |
37 |
|
38 |
*On Ubuntu, a few more packages must be installed before installing the |
39 |
prerequisite Python libraries |
40 |
|
41 |
$sudo aptitude install libcurl3-gnutls libcurl3-gnutls-dev uuid-dev |
42 |
|
43 |
2. Checkout the code and install the Python prerequisites. This assumes |
44 |
that python is already installed on the host. |
45 |
|
46 |
$ sudo easy_install virtualenv |
47 |
$ git clone https://user@code.grnet.gr/git/synnefo synnefo |
48 |
$ virtualenv --python=python2.6 synnefo --no-site-packages |
49 |
... |
50 |
$ cd synnefo |
51 |
$ ./bin/pip install <list_of_prerequisites> |
52 |
|
53 |
3. At this point you should have all required dependencies installed. Now you |
54 |
have to select a database engine. The choices are: postgres, mysql and sqlite. |
55 |
|
56 |
-SQLite |
57 |
The python sqlite driver is available by default with Python so no additional |
58 |
configuration is required. Also, most self-respecting systems have the sqlite |
59 |
library installed by default. |
60 |
|
61 |
-MySQL |
62 |
MySQL must be installed first |
63 |
|
64 |
*Ubuntu - Debian |
65 |
$sudo apt-get install libmysqlclient-dev |
66 |
|
67 |
*MacPorts |
68 |
$sudo port install mysql5 |
69 |
|
70 |
Install the MySQL python library |
71 |
|
72 |
$ bin/pip install MySQL-python |
73 |
|
74 |
Note: On MacOSX with Mysql install from MacPorts the above command will fail |
75 |
complaining that it cannot find the mysql_config command. Do the following and |
76 |
restart the installation |
77 |
|
78 |
$ echo "mysql_config = /opt/local/bin/mysql_config5" >> ./build/MySQL-python/site.cfg |
79 |
|
80 |
Configure a MySQL db/account for synnefo |
81 |
|
82 |
$ mysql -u root -p |
83 |
|
84 |
mysql> create database synnefo; |
85 |
mysql> show databases; |
86 |
mysql> GRANT ALL on synnefo.* TO username IDENTIFIED BY 'password'; |
87 |
|
88 |
-Postgres |
89 |
|
90 |
#Ubuntu - Debian |
91 |
$ sudo apt-get install postgresql-8.4 libpq-dev |
92 |
|
93 |
#MacPorts |
94 |
$ sudo port install postgresql84 |
95 |
|
96 |
Install the postgres Python library |
97 |
|
98 |
$ bin/pip install psycopg2 |
99 |
|
100 |
Configure a postgres db/account for synnefo |
101 |
Become the postgres user, connect to PostgreSQL: |
102 |
$ sudo su - postgres |
103 |
$ psql |
104 |
|
105 |
Run the following commands: |
106 |
DROP DATABASE synnefo; |
107 |
DROP USER username; |
108 |
CREATE USER username WITH PASSWORD 'password'; |
109 |
CREATE DATABASE synnefo; |
110 |
GRANT ALL PRIVILEGES ON DATABASE synnefo TO username; |
111 |
ALTER DATABASE synnefo OWNER TO username; |
112 |
ALTER USER username CREATEDB; |
113 |
|
114 |
The last line enables the newly created user to create own databases. This is |
115 |
needed for Django to create and drop the test_synnefo database for unit |
116 |
testing. |
117 |
|
118 |
4. At this point you should have a working DB. Now configure Django to access it: |
119 |
Copy the default configuration file |
120 |
|
121 |
$ cp settings.py.dist settings.py |
122 |
|
123 |
and then copy/edit according to the database used: |
124 |
|
125 |
-SQLite |
126 |
|
127 |
PROJECT_PATH = os.path.dirname(os.path.abspath(__file__)) + '/' |
128 |
|
129 |
DATABASES = { |
130 |
'default': { |
131 |
'ENGINE': 'django.db.backends.sqlite3', |
132 |
'NAME': PROJECT_PATH + 'synnefo.db' #WARN: This must be an absolute path |
133 |
} |
134 |
} |
135 |
|
136 |
-MySQL |
137 |
DATABASES = { |
138 |
'default': { |
139 |
'ENGINE': 'django.db.backends.mysql', |
140 |
'NAME': 'synnefo', |
141 |
'USER': 'USERNAME', |
142 |
'PASSWORD': 'PASSWORD', |
143 |
'HOST': 'HOST', |
144 |
'PORT': 'PORT', |
145 |
'OPTIONS': { |
146 |
'init_command': 'SET storage_engine=INNODB', |
147 |
} |
148 |
} |
149 |
} |
150 |
|
151 |
-Postgres |
152 |
|
153 |
DATABASES = { |
154 |
'default': { |
155 |
'ENGINE': 'django.db.backends.postgresql_psycopg2', |
156 |
'NAME': 'DATABASE', |
157 |
'USER': 'USERNAME', |
158 |
'PASSWORD': 'PASSWORD', |
159 |
'HOST': 'HOST', |
160 |
'PORT': 'PORT', |
161 |
} |
162 |
} |
163 |
|
164 |
5. Try it out. The following command will attempt to connect to the DB and |
165 |
print out DDL statements. It should not fail. |
166 |
|
167 |
$ ./bin/python manage.py sql db |
168 |
|
169 |
6. Create the DB and (optionally) load test data |
170 |
|
171 |
$ ./bin/python manage.py syncdb |
172 |
$ ./bin/python manage.py loaddata db/fixtures/flavors.json |
173 |
$ ./bin/python manage.py loaddata db/fixtures/images.json |
174 |
|
175 |
The following fixtures can be loaded optionally depending on |
176 |
testing requirements: |
177 |
|
178 |
$ ./bin/python manage.py loaddata db/fixtures/vms.json |
179 |
$ ./bin/python manage.py loaddata db/fixtures/disks.json |
180 |
|
181 |
7. Set the BACKEND_PREFIX_ID variable to some unique prefix, e.g. your commit |
182 |
username in settings.py. Several functional conventions within the system |
183 |
require this variable to include a dash at its end (e.g. snf-) |
184 |
|
185 |
8. Fix the AMQP-specific settings based on the selected BACKEND_PREFIX_ID. |
186 |
The fix_amqp_settings() function is called once at the end of |
187 |
settings.py.dist, you must call it again if you change BACKEND_PREFIX_ID |
188 |
at some later point. |
189 |
|
190 |
9. Start the system |
191 |
$ ./bin/python logic/dispatcher.py # DB synch daemon |
192 |
$ ./bin/python manage.py runserver # Django |
193 |
|
194 |
10. (Hopefully) Done |
195 |
|
196 |
South Database Migrations |
197 |
------------------------ |
198 |
|
199 |
*Initial Migration |
200 |
|
201 |
First, remember to add the south app to settings.py (it is already included in the |
202 |
settings.py.dist). |
203 |
|
204 |
To initialise south migrations in your database the following commands must be executed: |
205 |
|
206 |
$ ./bin/python manage.py syncdb # Create / update the database with the south tables |
207 |
$ ./bin/python manage.py migrate db # Perform migration in the database |
208 |
|
209 |
Note that syncdb will create the latest models that exist in the db app, so some migrations may fail. |
210 |
If you are sure a migration has already taken place you must use the "--fake" option, to apply it. |
211 |
|
212 |
For example: |
213 |
|
214 |
$ ./bin/python manage.py migrate db 0001 --fake |
215 |
|
216 |
To be sure that all migrations are applied type: |
217 |
|
218 |
$ ./bin/python manage.py migrate db --list |
219 |
|
220 |
All starred migrations are applied. |
221 |
|
222 |
Remember, the migration is performed mainly for the data, not for the database schema. If you do not want to migrate the |
223 |
data, a syncdb and fake migrations for all the migration versions will suffice. |
224 |
|
225 |
*Schema migrations: |
226 |
|
227 |
Do not use the syncdb management command. It can only be used |
228 |
the first time and/or if you drop the database and must recreate it from scratch. |
229 |
See "Initial Migration" section. |
230 |
|
231 |
Each time you make changes to the database and data migration is not required (WARNING: always |
232 |
perform this with extreme care): |
233 |
|
234 |
$ ./bin/python schemamigration db --auto |
235 |
|
236 |
The above will create the migration script. Now this must be applied to the live database. |
237 |
|
238 |
$ ./bin/python migrate db |
239 |
|
240 |
Consider this example (adding a field to the SynnefoUser model): |
241 |
|
242 |
bkarak@nefarian:~/devel/synnefo$ ./bin/python manage.py schemamigration db --auto |
243 |
+ Added field new_south_test_field on db.SynnefoUser |
244 |
Created 0002_auto__add_field_synnefouser_new_south_test_field.py. |
245 |
|
246 |
You can now apply this migration with: ./manage.py migrate db |
247 |
|
248 |
$ ./manage.py migrate db |
249 |
Running migrations for db: |
250 |
- Migrating forwards to 0002_auto__add_field_synnefouser_new_south_test_field. |
251 |
> db:0002_auto__add_field_synnefouser_new_south_test_field |
252 |
- Loading initial data for db. |
253 |
Installing json fixture 'initial_data' from '/home/bkarak/devel/synnefo/../synnefo/db/fixtures'. |
254 |
Installed 1 object(s) from 1 fixture(s) |
255 |
|
256 |
South needs some extra definitions to the model to preserve and migrate the existing data, for example, if we add a field |
257 |
in a model, we should declare its default value. If not, South will propably fail, after indicating the error. |
258 |
|
259 |
$ ./bin/python manage.py schemamigration db --auto |
260 |
? The field 'SynnefoUser.new_south_field_2' does not have a default specified, yet is NOT NULL. |
261 |
? Since you are adding or removing this field, you MUST specify a default |
262 |
? value to use for existing rows. Would you like to: |
263 |
? 1. Quit now, and add a default to the field in models.py |
264 |
? 2. Specify a one-off value to use for existing columns now |
265 |
? Please select a choice: 1 |
266 |
|
267 |
*Data migrations: |
268 |
|
269 |
If we need to do data migration as well, for example rename a field, we use tha 'datamigration' management command. |
270 |
|
271 |
In contrast with schemamigration, to perform complex data migration, we must write the script manually. The process is |
272 |
the following: |
273 |
|
274 |
1. Introduce the changes in the code and fixtures (initial data). |
275 |
2. Execute: |
276 |
|
277 |
$ ./bin/python manage.py datamigration <migration_name_here> |
278 |
|
279 |
For example: |
280 |
|
281 |
$ ./bin/python manage.py datamigration db rename_credit_wallet |
282 |
Created 0003_rename_credit_wallet.py. |
283 |
|
284 |
3. We edit the generated script. It contains two methods: forwards and backwards. |
285 |
|
286 |
For database operations (column additions, alter tables etc) we use the South database API |
287 |
(http://south.aeracode.org/docs/databaseapi.html). |
288 |
|
289 |
To access the data, we use the database reference (orm) provided as parameter in forwards, backwards method declarations |
290 |
in the migration script. For example: |
291 |
|
292 |
class Migration(DataMigration): |
293 |
|
294 |
def forwards(self, orm): |
295 |
orm.SynnefoUser.objects.all() |
296 |
|
297 |
4. To migrate the database to the latest version, we execute: |
298 |
|
299 |
./manage.py migrate db |
300 |
|
301 |
To see which migrations are applied: |
302 |
|
303 |
$ ./bin/python manage.py migrate db --list |
304 |
|
305 |
db |
306 |
(*) 0001_initial |
307 |
(*) 0002_auto__add_field_synnefouser_new_south_test_field |
308 |
(*) 0003_rename_credit_wallet |
309 |
|
310 |
More information and more thorough examples can be found in the South web site. |
311 |
|
312 |
http://south.aeracode.org/ |
313 |
|
314 |
UI Testing |
315 |
---------- |
316 |
The functional ui tests require the Selenium server and the synnefo app to |
317 |
be running. |
318 |
|
319 |
$ wget http://selenium.googlecode.com/files/selenium-server-standalone-2.0b2.jar |
320 |
$ java -jar selenium-server-standalone-2.0b2.jar & |
321 |
$ ./bin/python manage.py runserver & |
322 |
$ ./bin/python manage.py test ui |
323 |
|
324 |
Test coverage |
325 |
------------- |
326 |
|
327 |
In order to get code coverage reports you need to install django-test-coverage |
328 |
|
329 |
$ ./bin/pip install django-test-coverage |
330 |
|
331 |
Then edit your settings.py and configure the test runner: |
332 |
|
333 |
TEST_RUNNER = 'django-test-coverage.runner.run_tests' |