root / README.develop @ 13b954b0
History | View | Annotate | Download (10 kB)
1 |
DEVELOP.txt - Information on how to setup a development environment. |
---|---|
2 |
|
3 |
Dependencies |
4 |
------------ |
5 |
|
6 |
Synnefo is written in Python 2.6 and depends on the following Python modules |
7 |
[package versions confirmed to be compatible are in braces] |
8 |
|
9 |
- django 1.2 [Django==1.2.4] |
10 |
- simplejson [simplejson==2.1.3] |
11 |
- selenium [?] |
12 |
- pyzmq-static [pyzmq==2.0.10.1] |
13 |
- pycurl [pycurl==7.19.0] |
14 |
- python-dateutil [python-dateutil==1.4.1] |
15 |
WARNING: version python-dateutil==2.0 downloaded by pip known *not* to work |
16 |
with Python 2.6 |
17 |
- south [south==0.7.1] |
18 |
|
19 |
also, depending on the database engine of choice, on one of the following: |
20 |
- MySQL-python [MySQL-python==1.2.3] |
21 |
- psycopg2 [psycopg2==2.4] |
22 |
|
23 |
|
24 |
Preparing the development environment |
25 |
------------------------------------- |
26 |
|
27 |
1. Prepare the system |
28 |
The easiest method is to setup a working environment through virtualenv. |
29 |
Alternatively, you can use your system's package manager to install |
30 |
the dependencies (e.g. Macports has them all). |
31 |
|
32 |
*On Snow Leopard and linux (64-bit), you have to set the following environment |
33 |
variable for pip to compile the dependencies correctly. |
34 |
|
35 |
$export ARCHFLAGS="-arch x86_64" |
36 |
|
37 |
*On Ubuntu, a few more packages must be installed before installing the |
38 |
prerequisite Python libraries |
39 |
|
40 |
$sudo aptitude install libcurl3-gnutls libcurl3-gnutls-dev uuid-dev |
41 |
|
42 |
2. Checkout the code and install the Python prerequisites. This assumes |
43 |
that python is already installed on the host. |
44 |
|
45 |
$ sudo easy_install virtualenv |
46 |
$ git clone https://user@code.grnet.gr/git/synnefo synnefo |
47 |
$ virtualenv --python=python2.6 synnefo --no-site-packages |
48 |
... |
49 |
$ cd synnefo |
50 |
$ ./bin/pip install <list_of_prerequisites> |
51 |
|
52 |
3. At this point you should have all required dependencies installed. Now you |
53 |
have to select a database engine. The choices are: postgres, mysql and sqlite. |
54 |
|
55 |
-SQLite |
56 |
The python sqlite driver is available by default with Python so no additional |
57 |
configuration is required. Also, most self-respecting systems have the sqlite |
58 |
library installed by default. |
59 |
|
60 |
-MySQL |
61 |
MySQL must be installed first |
62 |
|
63 |
*Ubuntu - Debian |
64 |
$sudo apt-get install libmysqlclient-dev |
65 |
|
66 |
*MacPorts |
67 |
$sudo port install mysql5 |
68 |
|
69 |
Install the MySQL python library |
70 |
|
71 |
$ bin/pip install MySQL-python |
72 |
|
73 |
Note: On MacOSX with Mysql install from MacPorts the above command will fail |
74 |
complaining that it cannot find the mysql_config command. Do the following and |
75 |
restart the installation |
76 |
|
77 |
$ echo "mysql_config = /opt/local/bin/mysql_config5" >> ./build/MySQL-python/site.cfg |
78 |
|
79 |
Configure a MySQL db/account for synnefo |
80 |
|
81 |
$ mysql -u root -p |
82 |
|
83 |
mysql> create database synnefo; |
84 |
mysql> show databases; |
85 |
mysql> GRANT ALL on synnefo.* TO username IDENTIFIED BY 'password'; |
86 |
|
87 |
-Postgres |
88 |
|
89 |
#Ubuntu - Debian |
90 |
$ sudo apt-get install postgresql-8.4 libpq-dev |
91 |
|
92 |
#MacPorts |
93 |
$ sudo port install postgresql84 |
94 |
|
95 |
Install the postgres Python library |
96 |
|
97 |
$ bin/pip install psycopg2 |
98 |
|
99 |
Configure a postgres db/account for synnefo |
100 |
Become the postgres user, connect to PostgreSQL: |
101 |
$ sudo su - postgres |
102 |
$ psql |
103 |
|
104 |
Run the following commands: |
105 |
DROP DATABASE synnefo; |
106 |
DROP USER username; |
107 |
CREATE USER username WITH PASSWORD 'password'; |
108 |
CREATE DATABASE synnefo; |
109 |
GRANT ALL PRIVILEGES ON DATABASE synnefo TO username; |
110 |
ALTER DATABASE synnefo OWNER TO username; |
111 |
ALTER USER username CREATEDB; |
112 |
|
113 |
The last line enables the newly created user to create own databases. This is |
114 |
needed for Django to create and drop the test_synnefo database for unit |
115 |
testing. |
116 |
|
117 |
4. At this point you should have a working DB. Now configure Django to access it: |
118 |
Copy the default configuration file |
119 |
|
120 |
$ cp settings.py.dist settings.py |
121 |
|
122 |
and then copy/edit according to the database used: |
123 |
|
124 |
-SQLite |
125 |
|
126 |
PROJECT_PATH = os.path.dirname(os.path.abspath(__file__)) + '/' |
127 |
|
128 |
DATABASES = { |
129 |
'default': { |
130 |
'ENGINE': 'django.db.backends.sqlite3', |
131 |
'NAME': PROJECT_PATH + 'synnefo.db' #WARN: This must be an absolute path |
132 |
} |
133 |
} |
134 |
|
135 |
-MySQL |
136 |
DATABASES = { |
137 |
'default': { |
138 |
'ENGINE': 'django.db.backends.mysql', |
139 |
'NAME': 'synnefo', |
140 |
'USER': 'USERNAME', |
141 |
'PASSWORD': 'PASSWORD', |
142 |
'HOST': 'HOST', |
143 |
'PORT': 'PORT', |
144 |
'OPTIONS': { |
145 |
'init_command': 'SET storage_engine=INNODB', |
146 |
} |
147 |
} |
148 |
} |
149 |
|
150 |
-Postgres |
151 |
|
152 |
DATABASES = { |
153 |
'default': { |
154 |
'ENGINE': 'django.db.backends.postgresql_psycopg2', |
155 |
'NAME': 'DATABASE', |
156 |
'USER': 'USERNAME', |
157 |
'PASSWORD': 'PASSWORD', |
158 |
'HOST': 'HOST', |
159 |
'PORT': 'PORT', |
160 |
} |
161 |
} |
162 |
|
163 |
5. Try it out. The following command will attempt to connect to the DB and |
164 |
print out DDL statements. It should not fail. |
165 |
|
166 |
$ ./bin/python manage.py sql db |
167 |
|
168 |
6. Create the DB and (optionally) load test data |
169 |
|
170 |
$ ./bin/python manage.py syncdb |
171 |
$ ./bin/python manage.py loaddata db/fixtures/flavors.json |
172 |
$ ./bin/python manage.py loaddata db/fixtures/images.json |
173 |
|
174 |
The following fixtures can be loaded optionally depending on |
175 |
testing requirements: |
176 |
|
177 |
$ ./bin/python manage.py loaddata db/fixtures/vms.json |
178 |
$ ./bin/python manage.py loaddata db/fixtures/disks.json |
179 |
|
180 |
7. Set the BACKEND_PREFIX_ID variable to some unique prefix, e.g. your commit |
181 |
username |
182 |
|
183 |
8. Start the system |
184 |
$ ./bin/python db/db_controller.py # DB synch daemon |
185 |
$ ./bin/python manage.py runserver # Django |
186 |
|
187 |
9. (Hopefully) Done |
188 |
|
189 |
South Database Migrations |
190 |
------------------------ |
191 |
|
192 |
*Initial Migration |
193 |
|
194 |
First, remember to add the south app to settings.py (it is already included in the |
195 |
settings.py.dist). |
196 |
|
197 |
To initialise south migrations in your database the following commands must be executed: |
198 |
|
199 |
$ ./bin/python manage.py syncdb # Create / update the database with the south tables |
200 |
$ ./bin/python manage.py migrate db # Perform migration in the database |
201 |
|
202 |
Note that syncdb will create the latest models that exist in the db app, so some migrations may fail. |
203 |
If you are sure a migration has already taken place you must use the "--fake" option, to apply it. |
204 |
|
205 |
For example: |
206 |
|
207 |
$ ./bin/python manage.py migrate db 0001 --fake |
208 |
|
209 |
To be sure that all migrations are applied type: |
210 |
|
211 |
$ ./bin/python manage.py migrate db --list |
212 |
|
213 |
All starred migrations are applied. |
214 |
|
215 |
Remember, the migration is performed mainly for the data, not for the database schema. If you do not want to migrate the |
216 |
data, a syncdb and fake migrations for all the migration versions will suffice. |
217 |
|
218 |
*Schema migrations: |
219 |
|
220 |
Do not use the syncdb management command. It can only be used |
221 |
the first time and/or if you drop the database and must recreate it from scratch. |
222 |
See "Initial Migration" section. |
223 |
|
224 |
Each time you make changes to the database and data migration is not required (WARNING: always |
225 |
perform this with extreme care): |
226 |
|
227 |
$ ./bin/python schemamigration db --auto |
228 |
|
229 |
The above will create the migration script. Now this must be applied to the live database. |
230 |
|
231 |
$ ./bin/python migrate db |
232 |
|
233 |
Consider this example (adding a field to the SynnefoUser model): |
234 |
|
235 |
bkarak@nefarian:~/devel/synnefo$ ./bin/python manage.py schemamigration db --auto |
236 |
+ Added field new_south_test_field on db.SynnefoUser |
237 |
Created 0002_auto__add_field_synnefouser_new_south_test_field.py. |
238 |
|
239 |
You can now apply this migration with: ./manage.py migrate db |
240 |
|
241 |
$ ./manage.py migrate db |
242 |
Running migrations for db: |
243 |
- Migrating forwards to 0002_auto__add_field_synnefouser_new_south_test_field. |
244 |
> db:0002_auto__add_field_synnefouser_new_south_test_field |
245 |
- Loading initial data for db. |
246 |
Installing json fixture 'initial_data' from '/home/bkarak/devel/synnefo/../synnefo/db/fixtures'. |
247 |
Installed 1 object(s) from 1 fixture(s) |
248 |
|
249 |
South needs some extra definitions to the model to preserve and migrate the existing data, for example, if we add a field |
250 |
in a model, we should declare its default value. If not, South will propably fail, after indicating the error. |
251 |
|
252 |
$ ./bin/python manage.py schemamigration db --auto |
253 |
? The field 'SynnefoUser.new_south_field_2' does not have a default specified, yet is NOT NULL. |
254 |
? Since you are adding or removing this field, you MUST specify a default |
255 |
? value to use for existing rows. Would you like to: |
256 |
? 1. Quit now, and add a default to the field in models.py |
257 |
? 2. Specify a one-off value to use for existing columns now |
258 |
? Please select a choice: 1 |
259 |
|
260 |
*Data migrations: |
261 |
|
262 |
If we need to do data migration as well, for example rename a field, we use tha 'datamigration' management command. |
263 |
|
264 |
In contrast with schemamigration, to perform complex data migration, we must write the script manually. The process is |
265 |
the following: |
266 |
|
267 |
1. Introduce the changes in the code and fixtures (initial data). |
268 |
2. Execute: |
269 |
|
270 |
$ ./bin/python manage.py datamigration <migration_name_here> |
271 |
|
272 |
For example: |
273 |
|
274 |
$ ./bin/python manage.py datamigration db rename_credit_wallet |
275 |
Created 0003_rename_credit_wallet.py. |
276 |
|
277 |
3. We edit the generated script. It contains two methods: forwards and backwards. |
278 |
|
279 |
For database operations (column additions, alter tables etc) we use the South database API |
280 |
(http://south.aeracode.org/docs/databaseapi.html). |
281 |
|
282 |
To access the data, we use the database reference (orm) provided as parameter in forwards, backwards method declarations |
283 |
in the migration script. For example: |
284 |
|
285 |
class Migration(DataMigration): |
286 |
|
287 |
def forwards(self, orm): |
288 |
orm.SynnefoUser.objects.all() |
289 |
|
290 |
4. To migrate the database to the latest version, we execute: |
291 |
|
292 |
./manage.py migrate db |
293 |
|
294 |
To see which migrations are applied: |
295 |
|
296 |
$ ./bin/python manage.py migrate db --list |
297 |
|
298 |
db |
299 |
(*) 0001_initial |
300 |
(*) 0002_auto__add_field_synnefouser_new_south_test_field |
301 |
(*) 0003_rename_credit_wallet |
302 |
|
303 |
More information and more thorough examples can be found in the South web site. |
304 |
|
305 |
http://south.aeracode.org/ |
306 |
|
307 |
UI Testing |
308 |
---------- |
309 |
The functional ui tests require the Selenium server and the synnefo app to |
310 |
be running. |
311 |
|
312 |
$ wget http://selenium.googlecode.com/files/selenium-server-standalone-2.0b2.jar |
313 |
$ java -jar selenium-server-standalone-2.0b2.jar & |
314 |
$ ./bin/python manage.py runserver & |
315 |
$ ./bin/python manage.py test ui |
316 |
|
317 |
Test coverage |
318 |
------------- |
319 |
|
320 |
In order to get code coverage reports you need to install django-test-coverage |
321 |
|
322 |
$ ./bin/pip install django-test-coverage |
323 |
|
324 |
Then edit your settings.py and configure the test runner: |
325 |
|
326 |
TEST_RUNNER = 'django-test-coverage.runner.run_tests' |