root / README.develop @ 6d5f3dd2
History | View | Annotate | Download (9.4 kB)
1 |
DEVELOP.txt - Information on how to setup a development environment. |
---|---|
2 |
|
3 |
Dependencies |
4 |
------------ |
5 |
|
6 |
Synnefo is written in Python 2.6 and depends on the following Python modules |
7 |
[package versions confirmed to be compatible are in braces] |
8 |
|
9 |
- django 1.2 [Django==1.2.4] |
10 |
- simplejson [simplejson==2.1.3] |
11 |
- selenium [?] |
12 |
- pyzmq-static [pyzmq==2.0.10.1] |
13 |
- pycurl [pycurl==7.19.0] |
14 |
- python-dateutil [python-dateutil==1.4.1] |
15 |
WARNING: version python-dateutil==2.0 downloaded by pip known *not* to work |
16 |
with Python 2.6 |
17 |
- south [south==0.7.1] |
18 |
|
19 |
also, depending on the database engine of choice, on one of the following: |
20 |
- MySQL-python [MySQL-python==1.2.3] |
21 |
- psycopg2 [psycopg2==2.4] |
22 |
|
23 |
|
24 |
Preparing the development environment |
25 |
------------------------------------- |
26 |
|
27 |
1. Prepare the system |
28 |
The easiest method is to setup a working environment through virtualenv. |
29 |
Alternatively, you can use your system's package manager to install |
30 |
the dependencies (e.g. Macports has them all). |
31 |
|
32 |
*On Snow Leopard and linux (64-bit), you have to set the following environment |
33 |
variable for pip to compile the dependencies correctly. |
34 |
|
35 |
$export ARCHFLAGS="-arch x86_64" |
36 |
|
37 |
*On Ubuntu, a few more packages must be installed before installing the |
38 |
prerequisite Python libraries |
39 |
|
40 |
$sudo aptitude install libcurl3-gnutls libcurl3-gnutls-dev uuid-dev |
41 |
|
42 |
2. Checkout the code and install the Python prerequisites. This assumes |
43 |
that python is already installed on the host. |
44 |
|
45 |
$ sudo easy_install virtualenv |
46 |
$ git clone https://user@code.grnet.gr/git/synnefo synnefo |
47 |
$ virtualenv --python=python2.6 synnefo --no-site-packages |
48 |
... |
49 |
$ cd synnefo |
50 |
$ ./bin/pip install <list_of_prerequisites> |
51 |
|
52 |
3. At this point you should have all required dependencies installed. Now you |
53 |
have to select a database engine. The choices are: postgres, mysql and sqlite. |
54 |
|
55 |
-SQLite |
56 |
The python sqlite driver is available by default with Python so no additional |
57 |
configuration is required. Also, most self-respecting systems have the sqlite |
58 |
library installed by default. |
59 |
|
60 |
-MySQL |
61 |
MySQL must be installed first |
62 |
|
63 |
*Ubuntu - Debian |
64 |
$sudo apt-get install libmysqlclient-dev |
65 |
|
66 |
*MacPorts |
67 |
$sudo port install mysql5 |
68 |
|
69 |
Install the MySQL python library |
70 |
|
71 |
$ bin/pip install MySQL-python |
72 |
|
73 |
Note: On MacOSX with Mysql install from MacPorts the above command will fail |
74 |
complaining that it cannot find the mysql_config command. Do the following and |
75 |
restart the installation |
76 |
|
77 |
$ echo "mysql_config = /opt/local/bin/mysql_config5" >> ./build/MySQL-python/site.cfg |
78 |
|
79 |
Configure a MySQL db/account for synnefo |
80 |
|
81 |
$ mysql -u root -p |
82 |
|
83 |
mysql> create database synnefo; |
84 |
mysql> show databases; |
85 |
mysql> GRANT ALL on synnefo.* TO username IDENTIFIED BY 'password'; |
86 |
|
87 |
-Postgres |
88 |
|
89 |
#Ubuntu - Debian |
90 |
$ sudo apt-get install postgresql-8.4 libpq-dev |
91 |
|
92 |
#MacPorts |
93 |
$ sudo port install postgresql84 |
94 |
|
95 |
Install the postgres Python library |
96 |
|
97 |
$ bin/pip install psycopg2 |
98 |
|
99 |
Configure a postgres db/account for synnefo |
100 |
Become the postgres user, connect to PostgreSQL: |
101 |
$ sudo su - postgres |
102 |
$ psql |
103 |
|
104 |
Run the following commands: |
105 |
DROP DATABASE synnefo; |
106 |
DROP USER username; |
107 |
CREATE USER username WITH PASSWORD 'password'; |
108 |
CREATE DATABASE synnefo; |
109 |
GRANT ALL PRIVILEGES ON DATABASE synnefo TO username; |
110 |
ALTER DATABASE synnefo OWNER TO username; |
111 |
ALTER USER username CREATEDB; |
112 |
|
113 |
The last line enables the newly created user to create own databases. This is |
114 |
needed for Django to create and drop the test_synnefo database for unit |
115 |
testing. |
116 |
|
117 |
4. At this point you should have a working DB. Now configure Django to access it: |
118 |
Copy the default configuration file |
119 |
|
120 |
$ cp settings.py.dist settings.py |
121 |
|
122 |
and then copy/edit according to the database used: |
123 |
|
124 |
-SQLite |
125 |
|
126 |
PROJECT_PATH = os.path.dirname(os.path.abspath(__file__)) + '/' |
127 |
|
128 |
DATABASES = { |
129 |
'default': { |
130 |
'ENGINE': 'django.db.backends.sqlite3', |
131 |
'NAME': PROJECT_PATH + 'synnefo.db' #WARN: This must be an absolute path |
132 |
} |
133 |
} |
134 |
|
135 |
-MySQL |
136 |
DATABASES = { |
137 |
'default': { |
138 |
'ENGINE': 'django.db.backends.mysql', |
139 |
'NAME': 'synnefo', |
140 |
'USER': 'USERNAME' |
141 |
'PASSWORD': 'PASSWORD', |
142 |
'HOST': 'HOST', |
143 |
'PORT': 'PORT', |
144 |
'OPTIONS': { |
145 |
'init_command': 'SET storage_engine=INNODB', |
146 |
} |
147 |
} |
148 |
} |
149 |
|
150 |
-Postgres |
151 |
|
152 |
DATABASES = { |
153 |
'default': { |
154 |
'ENGINE': 'django.db.backends.postgresql_psycopg2', |
155 |
'NAME': 'DATABASE', |
156 |
'USER': 'USERNAME', |
157 |
'PASSWORD': 'PASSWORD', |
158 |
'HOST': 'HOST', |
159 |
'PORT': 'PORT', |
160 |
} |
161 |
} |
162 |
|
163 |
5. Try it out. The following command will attempt to connect to the DB and |
164 |
print out DDL statements. It should not fail. |
165 |
|
166 |
$ ./bin/python manage.py sql db |
167 |
|
168 |
6. Create the DB and (optionally) load test data |
169 |
|
170 |
$ ./bin/python manage.py syncdb |
171 |
$ ./bin/python manage.py loaddata db/fixtures/flavors.json |
172 |
$ ./bin/python manage.py loaddata db/fixtures/images.json |
173 |
|
174 |
The following fictures can be loaded optionally depending on |
175 |
testing requirements: |
176 |
|
177 |
$ ./bin/python manage.py loaddata db/fixtures/vms.json |
178 |
$ ./bin/python manage.py loaddata db/fixtures/initial_data.json |
179 |
$ ./bin/python manage.py loaddata db/fixtures/disks.json |
180 |
|
181 |
7. Set the BACKEND_PREFIX_ID variable to some unique prefix, e.g. your commit |
182 |
username |
183 |
|
184 |
8. Start the system |
185 |
$ ./bin/python db/db_controller.py #DB synch daemon |
186 |
$ ./bin/python manage.py runserver #Django |
187 |
|
188 |
9. (Hopefully) Done |
189 |
|
190 |
South Database Migrations |
191 |
------------------------ |
192 |
|
193 |
*Initial Migration |
194 |
|
195 |
*Schema migrations: |
196 |
|
197 |
First, remember to add the south app to settings.py (it is already included in the |
198 |
settings.py.dist). |
199 |
|
200 |
In addition, do not use the syncdb management command. It can only be used |
201 |
the first time and/or if you drop the database and must recreate it from scratch. |
202 |
|
203 |
The first schema support for migrations is initialized with the following |
204 |
command: |
205 |
|
206 |
$ ./bin/python schemamigration db --initial |
207 |
|
208 |
Each time you make changes to the database and data migration is not required (WARNING: always |
209 |
perform this with extreme care): |
210 |
|
211 |
$ ./bin/python schemamigration db --auto |
212 |
|
213 |
The above will create the migration script. Now this must be applied to the live database. |
214 |
|
215 |
$ ./bin/python migrate db |
216 |
|
217 |
Consider this example (adding a field to the SynnefoUser model): |
218 |
|
219 |
bkarak@nefarian:~/devel/synnefo$ ./bin/python manage.py schemamigration db --auto |
220 |
+ Added field new_south_test_field on db.SynnefoUser |
221 |
Created 0002_auto__add_field_synnefouser_new_south_test_field.py. |
222 |
|
223 |
You can now apply this migration with: ./manage.py migrate db |
224 |
|
225 |
$ ./manage.py migrate db |
226 |
Running migrations for db: |
227 |
- Migrating forwards to 0002_auto__add_field_synnefouser_new_south_test_field. |
228 |
> db:0002_auto__add_field_synnefouser_new_south_test_field |
229 |
- Loading initial data for db. |
230 |
Installing json fixture 'initial_data' from '/home/bkarak/devel/synnefo/../synnefo/db/fixtures'. |
231 |
Installed 1 object(s) from 1 fixture(s) |
232 |
|
233 |
South needs some extra definitions to the model to preserve and migrate the existing data, for example, if we add a field |
234 |
in a model, we should declare its default value. If not, South will propably fail, after indicating the error. |
235 |
|
236 |
$ ./bin/python manage.py schemamigration db --auto |
237 |
? The field 'SynnefoUser.new_south_field_2' does not have a default specified, yet is NOT NULL. |
238 |
? Since you are adding or removing this field, you MUST specify a default |
239 |
? value to use for existing rows. Would you like to: |
240 |
? 1. Quit now, and add a default to the field in models.py |
241 |
? 2. Specify a one-off value to use for existing columns now |
242 |
? Please select a choice: 1 |
243 |
|
244 |
*Data migrations: |
245 |
|
246 |
If we need to do data migration as well, for example rename a field, we use tha 'datamigration' management command. |
247 |
|
248 |
In contrast with schemamigration, to perform complex data migration, we must write the script manually. The process is |
249 |
the following: |
250 |
|
251 |
1. Introduce the changes in the code and fixtures (initial data). |
252 |
2. Execute: |
253 |
|
254 |
$ ./bin/python manage.py datamigration <migration_name_here> |
255 |
|
256 |
For example: |
257 |
|
258 |
$ ./bin/python manage.py datamigration db rename_credit_wallet |
259 |
Created 0003_rename_credit_wallet.py. |
260 |
|
261 |
3. We edit the generated script. It contains two methods: forwards and backwards. |
262 |
|
263 |
For database operations (column additions, alter tables etc) we use the South database API |
264 |
(http://south.aeracode.org/docs/databaseapi.html). |
265 |
|
266 |
To access the data, we use the database reference (orm) provided as parameter in forwards, backwards method declarations |
267 |
in the migration script. For example: |
268 |
|
269 |
class Migration(DataMigration): |
270 |
|
271 |
def forwards(self, orm): |
272 |
orm.SynnefoUser.objects.all() |
273 |
|
274 |
4. To migrate the database to the latest version, we execute: |
275 |
|
276 |
./manage.py migrate db |
277 |
|
278 |
To see which migrations are applied: |
279 |
|
280 |
$ ./bin/python manage.py migrate db --list |
281 |
|
282 |
db |
283 |
(*) 0001_initial |
284 |
(*) 0002_auto__add_field_synnefouser_new_south_test_field |
285 |
(*) 0003_rename_credit_wallet |
286 |
|
287 |
More information and more thorough examples can be found in the South web site. |
288 |
|
289 |
http://south.aeracode.org/ |
290 |
|
291 |
UI Testing |
292 |
---------- |
293 |
The functional ui tests require the Selenium server and the synnefo app to |
294 |
be running. |
295 |
|
296 |
$ wget http://selenium.googlecode.com/files/selenium-server-standalone-2.0b2.jar |
297 |
$ java -jar selenium-server-standalone-2.0b2.jar & |
298 |
$ ./bin/python manage.py runserver & |
299 |
$ ./bin/python manage.py test ui |
300 |
|
301 |
Test coverage |
302 |
------------- |
303 |
|
304 |
In order to get code coverage reports you need to install django-test-coverage |
305 |
|
306 |
$ ./bin/pip install django-test-coverage |
307 |
|
308 |
Then edit your settings.py and configure the test runner: |
309 |
|
310 |
TEST_RUNNER = 'django-test-coverage.runner.run_tests' |