root / README.develop @ 404ccab2
History | View | Annotate | Download (8.5 kB)
1 | 03353341 | Vangelis Koukis | DEVELOP.txt - Information on how to setup a development environment. |
---|---|---|---|
2 | bbd4f788 | Georgios Gousios | |
3 | 03353341 | Vangelis Koukis | This file documents the installation of a development environment for Synnefo. |
4 | 03353341 | Vangelis Koukis | It should be read alongside README.deploy. |
5 | bbd4f788 | Georgios Gousios | |
6 | 03353341 | Vangelis Koukis | It contains development-specific ammendments to the basic deployment steps |
7 | 03353341 | Vangelis Koukis | outlined in README.deploy, and development-specific notes. |
8 | c807557e | Vangelis Koukis | |
9 | bbd4f788 | Georgios Gousios | |
10 | 03353341 | Vangelis Koukis | Installing the development environment |
11 | 03353341 | Vangelis Koukis | ====================================== |
12 | bbd4f788 | Georgios Gousios | |
13 | 03353341 | Vangelis Koukis | For a basic development environment you need to follow steps 0-15 |
14 | 03353341 | Vangelis Koukis | of README.deploy, which should be read in its entirety *before* this document. |
15 | bbd4f788 | Georgios Gousios | |
16 | 03353341 | Vangelis Koukis | Development-specific guidelines on each step: |
17 | bbd4f788 | Georgios Gousios | |
18 | bbd4f788 | Georgios Gousios | |
19 | 03353341 | Vangelis Koukis | 0. Allocation of physical nodes: |
20 | 03353341 | Vangelis Koukis | Node types DB, APISERVER, LOGIC may all be run on the same physical machine, |
21 | 03353341 | Vangelis Koukis | usually, your development workstation. |
22 | bbd4f788 | Georgios Gousios | |
23 | 03353341 | Vangelis Koukis | Nodes of type GANETI-MASTER, GANETI-NODES and QUEUE are already provided |
24 | 03353341 | Vangelis Koukis | by the development Ganeti backend. Access credentials are provided in |
25 | 03353341 | Vangelis Koukis | settings.py.dist. |
26 | bbd4f788 | Georgios Gousios | |
27 | bbd4f788 | Georgios Gousios | |
28 | 03353341 | Vangelis Koukis | 1. You do not need to install your own Ganeti installation. |
29 | 03353341 | Vangelis Koukis | Use the RAPI endpoint as contained in settings.py.dist. |
30 | 03353341 | Vangelis Koukis | |
31 | 03353341 | Vangelis Koukis | |
32 | 03353341 | Vangelis Koukis | 2. You do not need to setup your own RabbitMQ nodes, use the AMQP endpoints |
33 | 271baf11 | Nikos Skalkotos | contained in settings.py.dist. MAKE SURE to call fix_amqp_settings() after |
34 | 271baf11 | Nikos Skalkotos | settings a custom BACKEND_PREFIX_ID. |
35 | 03353341 | Vangelis Koukis | |
36 | 03353341 | Vangelis Koukis | |
37 | 271baf11 | Nikos Skalkotos | 3. For development purposes, Django's own development |
38 | 271baf11 | Nikos Skalkotos | `server, ./manage.py runserver' will suffice. |
39 | 03353341 | Vangelis Koukis | |
40 | 03353341 | Vangelis Koukis | |
41 | 03353341 | Vangelis Koukis | 4. Use a virtual environment to install the Django project, or packages provided |
42 | 03353341 | Vangelis Koukis | by your distribution. |
43 | 03353341 | Vangelis Koukis | |
44 | 03353341 | Vangelis Koukis | |
45 | 03353341 | Vangelis Koukis | 5. Install a DB of your own, or use the PostgreSQL instance available on the |
46 | 03353341 | Vangelis Koukis | development backend. |
47 | 03353341 | Vangelis Koukis | |
48 | 03353341 | Vangelis Koukis | |
49 | 03353341 | Vangelis Koukis | 6. As is. |
50 | 03353341 | Vangelis Koukis | |
51 | 03353341 | Vangelis Koukis | |
52 | 03353341 | Vangelis Koukis | 7. The following fixtures can be loaded optionally depending on |
53 | 03353341 | Vangelis Koukis | testing/development requirements, and are not needed in a production setup: |
54 | f5e590e4 | Georgios Gousios | |
55 | bbd4f788 | Georgios Gousios | $ ./bin/python manage.py loaddata db/fixtures/vms.json |
56 | f5e590e4 | Georgios Gousios | $ ./bin/python manage.py loaddata db/fixtures/disks.json |
57 | 83d51c7b | Georgios Gousios | |
58 | 83d51c7b | Georgios Gousios | |
59 | 03353341 | Vangelis Koukis | 8. MAKE SURE you setup a distinct BACKEND_PREFIX_ID, e.g., use your commit |
60 | 271baf11 | Nikos Skalkotos | username. Make sure you call fix_amqp_settings() to setup use of individual |
61 | 271baf11 | Nikos Skalkotos | queues on RabbiMQ. |
62 | 03353341 | Vangelis Koukis | |
63 | 03353341 | Vangelis Koukis | |
64 | 03353341 | Vangelis Koukis | 9. The Ganeti monitoring daemon from the latest Synnefo release is already |
65 | 03353341 | Vangelis Koukis | running on the development Ganeti master. You may also run your own, on your |
66 | 03353341 | Vangelis Koukis | own Ganeti backend if you so wish. |
67 | 24435a56 | Georgios Gousios | |
68 | f5e590e4 | Georgios Gousios | |
69 | 03353341 | Vangelis Koukis | 10.As is. |
70 | 03353341 | Vangelis Koukis | |
71 | 03353341 | Vangelis Koukis | 11.The Synnefo Ganeti hook is already running on the development backend, |
72 | 03353341 | Vangelis Koukis | sending notifications over AMQP. |
73 | 03353341 | Vangelis Koukis | |
74 | 03353341 | Vangelis Koukis | |
75 | 03353341 | Vangelis Koukis | 12.The VNC authentication proxy is already running on the Ganeti development |
76 | 271baf11 | Nikos Skalkotos | backend. You *cannot* run your own, unless you install your own Ganeti |
77 | 271baf11 | Nikos Skalkotos | backend, because it needs direct access to the hypervisor's VNC port on |
78 | 271baf11 | Nikos Skalkotos | GANETI-NODEs. |
79 | 03353341 | Vangelis Koukis | |
80 | 11f37f3a | Vangelis Koukis | Note: You still need to install the vncauthproxy package to satisfy |
81 | 11f37f3a | Vangelis Koukis | the dependency of the API on the vncauthproxy client. See Synnefo #807 |
82 | 11f37f3a | Vangelis Koukis | for more details. |
83 | 11f37f3a | Vangelis Koukis | |
84 | 03353341 | Vangelis Koukis | |
85 | 03353341 | Vangelis Koukis | 13.The development Ganeti backend already has a number of OS Images available. |
86 | 03353341 | Vangelis Koukis | |
87 | 03353341 | Vangelis Koukis | |
88 | 271baf11 | Nikos Skalkotos | 14.The development Ganeti backend already has a number of pre-provisioned |
89 | 271baf11 | Nikos Skalkotos | bridges available, per each BACKEND_PREFIX_ID. |
90 | 03353341 | Vangelis Koukis | |
91 | 03353341 | Vangelis Koukis | To setup simple NAT-based networking on a Ganeti backend on your own, |
92 | 03353341 | Vangelis Koukis | please see the provided patches under contrib/patches/. |
93 | 03353341 | Vangelis Koukis | You will need minor patches to the sample KVM ifup hook, kvm-vif-bridge, |
94 | 271baf11 | Nikos Skalkotos | and a small patch to NFDHCPD to enable it to work with bridged tap+ |
95 | 271baf11 | Nikos Skalkotos | interfaces. To support bridged tap interfaces you also need to patch the |
96 | 271baf11 | Nikos Skalkotos | python-nfqueue package, patches against python-nfqueue-0.3 [part of Debian |
97 | 271baf11 | Nikos Skalkotos | Sid] are also provided under contrib/patches/. |
98 | 03353341 | Vangelis Koukis | |
99 | 03353341 | Vangelis Koukis | |
100 | 03353341 | Vangelis Koukis | 15.As is. |
101 | 03353341 | Vangelis Koukis | |
102 | 03353341 | Vangelis Koukis | |
103 | 03353341 | Vangelis Koukis | 16.As is. |
104 | bbd4f788 | Georgios Gousios | |
105 | fc2afa67 | Vangelis Koukis | |
106 | 61868190 | Vangelis Koukis | 17.[OPTIONAL] Create settings.d/99-local.conf and insert local overrides for |
107 | 61868190 | Vangelis Koukis | settings.d/*. This will allow pulling new files without needing to reapply |
108 | 61868190 | Vangelis Koukis | local any local modifications. |
109 | 61868190 | Vangelis Koukis | |
110 | 61868190 | Vangelis Koukis | |
111 | 6fc64f55 | Georgios Gousios | South Database Migrations |
112 | 03353341 | Vangelis Koukis | ========================= |
113 | 9ad94f0a | Vassilios Karakoidas | |
114 | fc2afa67 | Vangelis Koukis | * Initial Migration |
115 | 6d5f3dd2 | Vassilios Karakoidas | |
116 | fc2afa67 | Vangelis Koukis | First, remember to add the south app to settings.py (it is already included in |
117 | fc2afa67 | Vangelis Koukis | the settings.py.dist). |
118 | 9ad94f0a | Vassilios Karakoidas | |
119 | fc2afa67 | Vangelis Koukis | To initialise south migrations in your database the following commands must be |
120 | fc2afa67 | Vangelis Koukis | executed: |
121 | 55175e78 | Vassilios Karakoidas | |
122 | 55175e78 | Vassilios Karakoidas | $ ./bin/python manage.py syncdb # Create / update the database with the south tables |
123 | 55175e78 | Vassilios Karakoidas | $ ./bin/python manage.py migrate db # Perform migration in the database |
124 | 55175e78 | Vassilios Karakoidas | |
125 | 271baf11 | Nikos Skalkotos | Note that syncdb will create the latest models that exist in the db app, so some |
126 | 271baf11 | Nikos Skalkotos | migrations may fail. If you are sure a migration has already taken place you |
127 | 271baf11 | Nikos Skalkotos | must use the "--fake" option, to apply it. |
128 | 55175e78 | Vassilios Karakoidas | |
129 | 55175e78 | Vassilios Karakoidas | For example: |
130 | 55175e78 | Vassilios Karakoidas | |
131 | 55175e78 | Vassilios Karakoidas | $ ./bin/python manage.py migrate db 0001 --fake |
132 | 9ad94f0a | Vassilios Karakoidas | |
133 | 55175e78 | Vassilios Karakoidas | To be sure that all migrations are applied type: |
134 | 9ad94f0a | Vassilios Karakoidas | |
135 | 55175e78 | Vassilios Karakoidas | $ ./bin/python manage.py migrate db --list |
136 | 55175e78 | Vassilios Karakoidas | |
137 | 55175e78 | Vassilios Karakoidas | All starred migrations are applied. |
138 | 55175e78 | Vassilios Karakoidas | |
139 | fc2afa67 | Vangelis Koukis | Remember, the migration is performed mainly for the data, not for the database |
140 | 271baf11 | Nikos Skalkotos | schema. If you do not want to migrate the data, a syncdb and fake migrations for |
141 | 271baf11 | Nikos Skalkotos | all the migration versions will suffice. |
142 | 55175e78 | Vassilios Karakoidas | |
143 | fc2afa67 | Vangelis Koukis | * Schema migrations: |
144 | 55175e78 | Vassilios Karakoidas | |
145 | 271baf11 | Nikos Skalkotos | Do not use the syncdb management command. It can only be used the first time |
146 | 271baf11 | Nikos Skalkotos | and/or if you drop the database and must recreate it from scratch. See |
147 | 271baf11 | Nikos Skalkotos | "Initial Migration" section. |
148 | 9ad94f0a | Vassilios Karakoidas | |
149 | fc2afa67 | Vangelis Koukis | Every time you make changes to the database and data migration is not required |
150 | fc2afa67 | Vangelis Koukis | (WARNING: always perform this with extreme care): |
151 | 9ad94f0a | Vassilios Karakoidas | |
152 | 1ef8ecd4 | Constantinos Venetsanopoulos | $ ./bin/python manage.py schemamigration db --auto |
153 | 9ad94f0a | Vassilios Karakoidas | |
154 | 271baf11 | Nikos Skalkotos | The above will create the migration script. Now this must be applied to the live |
155 | 271baf11 | Nikos Skalkotos | database. |
156 | 9ad94f0a | Vassilios Karakoidas | |
157 | 9ad94f0a | Vassilios Karakoidas | $ ./bin/python migrate db |
158 | 9ad94f0a | Vassilios Karakoidas | |
159 | 9ad94f0a | Vassilios Karakoidas | Consider this example (adding a field to the SynnefoUser model): |
160 | 9ad94f0a | Vassilios Karakoidas | |
161 | 271baf11 | Nikos Skalkotos | $ ./bin/python manage.py schemamigration db --auto |
162 | 9ad94f0a | Vassilios Karakoidas | + Added field new_south_test_field on db.SynnefoUser |
163 | 6fc64f55 | Georgios Gousios | Created 0002_auto__add_field_synnefouser_new_south_test_field.py. |
164 | 6fc64f55 | Georgios Gousios | |
165 | 6fc64f55 | Georgios Gousios | You can now apply this migration with: ./manage.py migrate db |
166 | 6fc64f55 | Georgios Gousios | |
167 | d79d8208 | Vassilios Karakoidas | $ ./manage.py migrate db |
168 | 9ad94f0a | Vassilios Karakoidas | Running migrations for db: |
169 | 9ad94f0a | Vassilios Karakoidas | - Migrating forwards to 0002_auto__add_field_synnefouser_new_south_test_field. |
170 | 9ad94f0a | Vassilios Karakoidas | > db:0002_auto__add_field_synnefouser_new_south_test_field |
171 | 9ad94f0a | Vassilios Karakoidas | - Loading initial data for db. |
172 | 9ad94f0a | Vassilios Karakoidas | Installing json fixture 'initial_data' from '/home/bkarak/devel/synnefo/../synnefo/db/fixtures'. |
173 | 9ad94f0a | Vassilios Karakoidas | Installed 1 object(s) from 1 fixture(s) |
174 | 9ad94f0a | Vassilios Karakoidas | |
175 | fc2afa67 | Vangelis Koukis | South needs some extra definitions to the model to preserve and migrate the |
176 | fc2afa67 | Vangelis Koukis | existing data, for example, if we add a field in a model, we should declare its |
177 | fc2afa67 | Vangelis Koukis | default value. If not, South will propably fail, after indicating the error. |
178 | 6fc64f55 | Georgios Gousios | |
179 | d79d8208 | Vassilios Karakoidas | $ ./bin/python manage.py schemamigration db --auto |
180 | d79d8208 | Vassilios Karakoidas | ? The field 'SynnefoUser.new_south_field_2' does not have a default specified, yet is NOT NULL. |
181 | d79d8208 | Vassilios Karakoidas | ? Since you are adding or removing this field, you MUST specify a default |
182 | d79d8208 | Vassilios Karakoidas | ? value to use for existing rows. Would you like to: |
183 | d79d8208 | Vassilios Karakoidas | ? 1. Quit now, and add a default to the field in models.py |
184 | d79d8208 | Vassilios Karakoidas | ? 2. Specify a one-off value to use for existing columns now |
185 | d79d8208 | Vassilios Karakoidas | ? Please select a choice: 1 |
186 | 6fc64f55 | Georgios Gousios | |
187 | fc2afa67 | Vangelis Koukis | * Data migrations: |
188 | 9ad94f0a | Vassilios Karakoidas | |
189 | 271baf11 | Nikos Skalkotos | If we need to do data migration as well, for example rename a field, we use the |
190 | fc2afa67 | Vangelis Koukis | 'datamigration' management command. |
191 | 9ad94f0a | Vassilios Karakoidas | |
192 | fc2afa67 | Vangelis Koukis | In contrast with schemamigration, to perform complex data migration, we must |
193 | fc2afa67 | Vangelis Koukis | write the script manually. The process is the following: |
194 | 2e19b161 | Vassilios Karakoidas | |
195 | 2e19b161 | Vassilios Karakoidas | 1. Introduce the changes in the code and fixtures (initial data). |
196 | 2e19b161 | Vassilios Karakoidas | 2. Execute: |
197 | 2e19b161 | Vassilios Karakoidas | |
198 | 2e19b161 | Vassilios Karakoidas | $ ./bin/python manage.py datamigration <migration_name_here> |
199 | 2e19b161 | Vassilios Karakoidas | |
200 | 2e19b161 | Vassilios Karakoidas | For example: |
201 | 2e19b161 | Vassilios Karakoidas | |
202 | 2e19b161 | Vassilios Karakoidas | $ ./bin/python manage.py datamigration db rename_credit_wallet |
203 | 2e19b161 | Vassilios Karakoidas | Created 0003_rename_credit_wallet.py. |
204 | 2e19b161 | Vassilios Karakoidas | |
205 | fc2afa67 | Vangelis Koukis | 3. We edit the generated script. It contains two methods: forwards and |
206 | fc2afa67 | Vangelis Koukis | backwards. |
207 | 2e19b161 | Vassilios Karakoidas | |
208 | fc2afa67 | Vangelis Koukis | For database operations (column additions, alter tables etc) we use the |
209 | fc2afa67 | Vangelis Koukis | South database API (http://south.aeracode.org/docs/databaseapi.html). |
210 | 2e19b161 | Vassilios Karakoidas | |
211 | fc2afa67 | Vangelis Koukis | To access the data, we use the database reference (orm) provided as |
212 | fc2afa67 | Vangelis Koukis | parameter in forwards, backwards method declarations in the migration |
213 | fc2afa67 | Vangelis Koukis | script. For example: |
214 | 2e19b161 | Vassilios Karakoidas | |
215 | 2e19b161 | Vassilios Karakoidas | class Migration(DataMigration): |
216 | 2e19b161 | Vassilios Karakoidas | |
217 | 2e19b161 | Vassilios Karakoidas | def forwards(self, orm): |
218 | 2e19b161 | Vassilios Karakoidas | orm.SynnefoUser.objects.all() |
219 | 2e19b161 | Vassilios Karakoidas | |
220 | 2e19b161 | Vassilios Karakoidas | 4. To migrate the database to the latest version, we execute: |
221 | 2e19b161 | Vassilios Karakoidas | |
222 | 2e19b161 | Vassilios Karakoidas | ./manage.py migrate db |
223 | 2e19b161 | Vassilios Karakoidas | |
224 | 2e19b161 | Vassilios Karakoidas | To see which migrations are applied: |
225 | 2e19b161 | Vassilios Karakoidas | |
226 | 2e19b161 | Vassilios Karakoidas | $ ./bin/python manage.py migrate db --list |
227 | 2e19b161 | Vassilios Karakoidas | |
228 | 2e19b161 | Vassilios Karakoidas | db |
229 | 2e19b161 | Vassilios Karakoidas | (*) 0001_initial |
230 | 2e19b161 | Vassilios Karakoidas | (*) 0002_auto__add_field_synnefouser_new_south_test_field |
231 | 2e19b161 | Vassilios Karakoidas | (*) 0003_rename_credit_wallet |
232 | 2e19b161 | Vassilios Karakoidas | |
233 | 2e19b161 | Vassilios Karakoidas | More information and more thorough examples can be found in the South web site. |
234 | 2e19b161 | Vassilios Karakoidas | |
235 | 2e19b161 | Vassilios Karakoidas | http://south.aeracode.org/ |
236 | 2e19b161 | Vassilios Karakoidas | |
237 | fc2afa67 | Vangelis Koukis | |
238 | bbd4f788 | Georgios Gousios | UI Testing |
239 | 03353341 | Vangelis Koukis | ========== |
240 | 03353341 | Vangelis Koukis | The functional ui tests require the Selenium server and the synnefo app to |
241 | 83d51c7b | Georgios Gousios | be running. |
242 | bbd4f788 | Georgios Gousios | |
243 | bbd4f788 | Georgios Gousios | $ wget http://selenium.googlecode.com/files/selenium-server-standalone-2.0b2.jar |
244 | bbd4f788 | Georgios Gousios | $ java -jar selenium-server-standalone-2.0b2.jar & |
245 | bbd4f788 | Georgios Gousios | $ ./bin/python manage.py runserver & |
246 | bbd4f788 | Georgios Gousios | $ ./bin/python manage.py test ui |
247 | bbd4f788 | Georgios Gousios | |
248 | fc2afa67 | Vangelis Koukis | |
249 | bbd4f788 | Georgios Gousios | Test coverage |
250 | 03353341 | Vangelis Koukis | ============= |
251 | bbd4f788 | Georgios Gousios | |
252 | bbd4f788 | Georgios Gousios | In order to get code coverage reports you need to install django-test-coverage |
253 | 03353341 | Vangelis Koukis | |
254 | bbd4f788 | Georgios Gousios | $ ./bin/pip install django-test-coverage |
255 | bbd4f788 | Georgios Gousios | |
256 | bbd4f788 | Georgios Gousios | Then edit your settings.py and configure the test runner: |
257 | bbd4f788 | Georgios Gousios | |
258 | bbd4f788 | Georgios Gousios | TEST_RUNNER = 'django-test-coverage.runner.run_tests' |