Revision fc2afa67

b/README.deploy
5 5
infrastructure and the dependencies among them. This document applies
6 6
to Synnefo v0.3.
7 7

  
8

  
8
    
9 9
- Node types
10 10

  
11 11
  Nodes in a Synnefo deployment belong in one of the following types:
......
44 44
      Any WEBSERVER can issue commands to the GANETI-MASTER, over RAPI, to effect
45 45
      changes in the state of the VMs. The GANETI-MASTER runs the Ganeti request
46 46
      queue.
47
      Services: only on GANETI-MASTER:
47
      Services:
48
      only on GANETI-MASTER:
48 49
        the Synnefo Ganeti monitoring daemon [/ganeti/ganeti-eventd],
49 50
        and the Synnefo Ganeti hook [/ganeti/snf-ganeti-hook.py].
51
      on each GANETI_NODE:
52
        a deployment-specific KVM ifup script
53
        properly configured NFDHCPD
54

  
50 55

  
51
    As of v0.3, the Synnefo Django project needs to be installed on nodes
56
    As of v0.4, the Synnefo Django project needs to be installed on nodes
52 57
    of type WEBSERVER, LOGIC and on the GANETI-MASTER, with a properly configured
53 58
    settings.py. In later revisions, the specific parts of the Django project
54 59
    which need to run on each node type will be identified.
55 60

  
56
    The settings.py file for Django is derived by concatenating the
61
    The settings.py file for Django may be derived by concatenating the
57 62
    settings.py.dist file contained in the Synnefo distribution with a file
58 63
    containing custom modifications. This is recommended to minimize the load
59 64
    of reconstructing settings.py from scratch, since each release currently
......
64 69

  
65 70
    This section describes each of the required service dependencies.
66 71

  
72
    * API implementation
73

  
74
      You need to load fixtures db/fixtures/{flavors,images}.json,
75
      which make the API usable by end users by defining a sample
76
      set of hardware configurations (flavors) and OS images.
77

  
67 78
    * RabbitMQ
68 79

  
69 80
      RabbitMQ is used as a generic message broker for the system. It should
......
111 122
      hooks in Ganeti.  It resides in the ganeti/ directory of the Synnefo
112 123
      project root.
113 124

  
114
      The hook needs to be enabled for the post-start phase, by *symlinking*
115
      in  /etc/ganeti/hooks/instance-start-post.d on GANETI-MASTER:
125
      The hook needs to be enabled for phases
126
      post-{add,modify,reboot,start,stop} by *symlinking*
127
      in  /etc/ganeti/hooks/instance-{add,modify,reboot,start,stop}-post.d
128
      on GANETI-MASTER, e.g.:
116 129

  
117 130
      root@ganeti-master:/etc/ganeti/hooks/instance-start-post.d# ls -l
118 131
      lrwxrwxrwx 1 root root 45 May   3 13:45 00-snf-ganeti-hook -> /home/devel/synnefo/ganeti/snf-ganeti-hook.py*
......
150 163
      repository, at https://code.grnet.gr/git/gnt-instance-image
151 164

  
152 165
      After installing gnt-instance-image do the following:
153
      1.root@ganeti-master:/path-to-repo# cp ./defaults /etc/default/ganeti-instance-image
154
      2.Uncomment the following in /etc/default/ganeti-instance-image:
155
        SWAP=no
156
        ARCH="x86_64"
157
      3.Add to /etc/default/ganeti-instance-image (so that make-dump makes no /boot partition):
158
        KERNEL_PATH="True"
159
      4.Change the path in make-dump line 22 according to your installation
160
        (/usr/share/ganeti/os/image/common.sh --> /srv/ganeti/os/image/common.sh)
161
      5.In common.sh, comment out the KERNEL_PATH variable initialization.
162
        (#KERNEL_PATH="$INSTANCE_HV_kernel_path")
163
      6.In /etc/ganeti/instance-image/hooks, make sure the hooks you want to
164
        run during instance creation process have execute permission. At least
165
        `grub' and `root_passwd' should be triggered to make the instance
166
        usable:
167
            chmod +x /etc/ganeti/instance-image/hooks/{grub,root_passwd}
166
      1. root@ganeti-master:/path-to-repo# cp ./defaults /etc/default/ganeti-instance-image
167
      2. Uncomment the following in /etc/default/ganeti-instance-image:
168
         SWAP=no
169
         ARCH="x86_64"
170
      3. Add to /etc/default/ganeti-instance-image (so that make-dump makes no /boot partition):
171
         KERNEL_PATH="True"
172
      4. Change the path in make-dump line 22 according to your installation
173
         (/usr/share/ganeti/os/image/common.sh --> /srv/ganeti/os/image/common.sh)
174
      5. In common.sh, comment out the KERNEL_PATH variable initialization.
175
         (#KERNEL_PATH="$INSTANCE_HV_kernel_path")
176
      6. In /etc/ganeti/instance-image/hooks, make sure the hooks you want to
177
         run during instance creation process have execute permission. At least
178
         `grub' and `root_passwd' should be triggered to make the instance
179
         usable:
180
             chmod +x /etc/ganeti/instance-image/hooks/{grub,root_passwd}
168 181

  
169 182
      Your Custom Images should be stored in a dump format under /var/cache/ganeti-instance-image
170 183
      and their filenames should have the following format:
......
172 185
      e.g. debian-6.0.1a-x86_64-root.dump (backend_id = "debian-6.0.1a")
173 186

  
174 187

  
175
-Administration
188

  
189
- Administration
190

  
176 191
    * Reconciliation process: On certain occasions, such as a Ganeti or
177 192
      RabbitMQ failure, the VM state in the system's database may differ from
178 193
      that in the Ganeti installation. The reconciliation process is designed
......
197 212

  
198 213
      On each invocation of the reconcile command, the system will trigger a
199 214
      reconciliation for ((num_all_vms/RECONCILIATION_MIN) * interval)
200
      machines. Obviously the less the interval value and the more the
201
      RECONCILIATION_MIN setting, the less load is going to be put on Ganeti.
215
      machines. Obviously the lower the interval value and the higher the
216
      setting of RECONCILIATION_MIN, the less load is going to be put on Ganeti.
202 217

  
203 218

  
204 219
- OS Specific instructions
b/README.develop
23 23
if the invitations application is deployed, the following
24 24
dependencies should be installed
25 25

  
26
-pycrypto==2.1.0
26
- pycrypto==2.1.0
27 27

  
28
if you want to test the ganeti 
28
if you want to test the ganeti (FIXME)
29 29

  
30 30
to run the user interface tests, selenium must be installed
31 31
- selenium [?]
......
33 33
Preparing the development environment
34 34
-------------------------------------
35 35

  
36
1. Prepare the system 
37
The easiest method is to setup a working environment through virtualenv. 
38
Alternatively, you can use your system's package manager to install
39
the dependencies (e.g. Macports has them all).
36
1. Prepare the system: The easiest method is to setup a working environment
37
through virtualenv. Alternatively, you can use your system's package manager
38
to install the dependencies (e.g. Macports has them all).
40 39

  
41
*On Snow Leopard and linux (64-bit), you have to set the following environment
42
variable for pip to compile the dependencies correctly.
40
* On Snow Leopard and linux (64-bit), you have to set the following environment
41
  variable for pip to compile the dependencies correctly.
43 42

  
44
	$export ARCHFLAGS="-arch x86_64"
43
	$ export ARCHFLAGS="-arch x86_64"
45 44

  
46
*On Ubuntu, a few more packages must be installed before installing the
47
prerequisite Python libraries
45
* On Ubuntu, a few more packages must be installed before installing the
46
  prerequisite Python libraries
48 47

  
49
	$sudo aptitude install libcurl3-gnutls libcurl3-gnutls-dev uuid-dev 
48
	$ sudo aptitude install libcurl3-gnutls libcurl3-gnutls-dev uuid-dev 
50 49

  
51 50
2. Checkout the code and install the Python prerequisites. This assumes
52 51
that python is already installed on the host.
......
61 60
3. At this point you should have all required dependencies installed. Now you
62 61
have to select a database engine. The choices are: postgres, mysql and sqlite.
63 62

  
64
-SQLite
63
- SQLite
65 64
The python sqlite driver is available by default with Python so no additional 
66 65
configuration is required. Also, most self-respecting systems have the sqlite 
67 66
library installed by default.   
68 67

  
69
-MySQL
68
- MySQL
70 69
MySQL must be installed first
71 70

  
72
*Ubuntu - Debian
73
	$sudo apt-get install libmysqlclient-dev
71
* Ubuntu - Debian
72
	$ sudo apt-get install libmysqlclient-dev
74 73

  
75
*MacPorts
74
* MacPorts
76 75
	$sudo port install mysql5
77 76

  
78 77
Install the MySQL python library
......
93 92
	mysql> show databases;
94 93
	mysql> GRANT ALL on synnefo.* TO username IDENTIFIED BY 'password';
95 94
 
96
-Postgres
95
- Postgres
97 96

  
98
#Ubuntu - Debian
97
# Ubuntu - Debian
99 98
	$ sudo apt-get install postgresql-8.4 libpq-dev
100 99

  
101
#MacPorts
100
# MacPorts
102 101
	$ sudo port install postgresql84
103 102

  
104 103
Install the postgres Python library
......
130 129
    
131 130
and then copy/edit according to the database used:  
132 131
    
133
-SQLite
132
- SQLite
134 133

  
135 134
	PROJECT_PATH = os.path.dirname(os.path.abspath(__file__)) + '/'
136 135

  
......
141 140
	    }
142 141
	}
143 142

  
144
-MySQL
143
- MySQL
145 144
	DATABASES = {
146 145
	    'default': {
147 146
		'ENGINE': 'django.db.backends.mysql', 
......
156 155
	    }
157 156
	}
158 157

  
159
-Postgres    
158
- Postgres    
160 159

  
161 160
    DATABASES = {
162 161
	    'default': {
......
174 173

  
175 174
	$ ./bin/python manage.py sql db
176 175
	
177
6. Create the DB and (optionally) load test data
176
6. Create the DB and load initial data:
178 177
    
179 178
	$ ./bin/python manage.py syncdb
180 179
	$ ./bin/python manage.py migrate db
......
204 203

  
205 204
10. (Hopefully) Done
206 205

  
206

  
207 207
South Database Migrations
208 208
------------------------
209 209

  
210
*Initial Migration
210
* Initial Migration
211 211

  
212
First, remember to add the south app to settings.py (it is already included in the
213
settings.py.dist).
212
First, remember to add the south app to settings.py (it is already included in
213
the settings.py.dist).
214 214

  
215
To initialise south migrations in your database the following commands must be executed:
215
To initialise south migrations in your database the following commands must be
216
executed:
216 217

  
217 218
    $ ./bin/python manage.py syncdb       # Create / update the database with the south tables
218 219
    $ ./bin/python manage.py migrate db   # Perform migration in the database
219 220

  
220
Note that syncdb will create the latest models that exist in the db app, so some migrations may fail.
221
If you are sure a migration has already taken place you must use the "--fake" option, to apply it.
221
Note that syncdb will create the latest models that exist in the db app, so
222
some migrations may fail.  If you are sure a migration has already taken place
223
you must use the "--fake" option, to apply it.
222 224

  
223 225
For example:
224 226

  
......
230 232

  
231 233
All starred migrations are applied.
232 234

  
233
Remember, the migration is performed mainly for the data, not for the database schema. If you do not want to migrate the
234
data, a syncdb and fake migrations for all the migration versions will suffice.
235
Remember, the migration is performed mainly for the data, not for the database
236
schema. If you do not want to migrate the data, a syncdb and fake migrations
237
for all the migration versions will suffice.
235 238

  
236
*Schema migrations:
239
* Schema migrations:
237 240

  
238 241
Do not use the syncdb management command. It can only be used
239
the first time and/or if you drop the database and must recreate it from scratch.
242
the first time and/or if you drop the database and must recreate it from
243
scratch.
240 244
See "Initial Migration" section.
241 245

  
242
Each time you make changes to the database and data migration is not required (WARNING: always
243
perform this with extreme care):
246
Every time you make changes to the database and data migration is not required
247
(WARNING: always perform this with extreme care):
244 248

  
245 249
    $ ./bin/python manage.py schemamigration db --auto
246 250

  
247
The above will create the migration script. Now this must be applied to the live database.
251
The above will create the migration script. Now this must be applied to the
252
live database.
248 253

  
249 254
    $ ./bin/python migrate db
250 255

  
......
264 269
    Installing json fixture 'initial_data' from '/home/bkarak/devel/synnefo/../synnefo/db/fixtures'.
265 270
    Installed 1 object(s) from 1 fixture(s)
266 271

  
267
South needs some extra definitions to the model to preserve and migrate the existing data, for example, if we add a field
268
in a model, we should declare its default value. If not, South will propably fail, after indicating the error.
272
South needs some extra definitions to the model to preserve and migrate the
273
existing data, for example, if we add a field in a model, we should declare its
274
default value. If not, South will propably fail, after indicating the error.
269 275

  
270 276
    $ ./bin/python manage.py schemamigration db --auto
271 277
     ? The field 'SynnefoUser.new_south_field_2' does not have a default specified, yet is NOT NULL.
......
275 281
     ?  2. Specify a one-off value to use for existing columns now
276 282
     ? Please select a choice: 1
277 283

  
278
*Data migrations:
284
* Data migrations:
279 285

  
280
If we need to do data migration as well, for example rename a field, we use tha 'datamigration' management command.
286
If we need to do data migration as well, for example rename a field, we use tha
287
'datamigration' management command.
281 288

  
282
In contrast with schemamigration, to perform complex data migration, we must write the script manually. The process is
283
the following:
289
In contrast with schemamigration, to perform complex data migration, we must
290
write the script manually. The process is the following:
284 291

  
285 292
    1. Introduce the changes in the code and fixtures (initial data).
286 293
    2. Execute:
......
292 299
    $ ./bin/python manage.py datamigration db rename_credit_wallet
293 300
    Created 0003_rename_credit_wallet.py.
294 301

  
295
    3. We edit the generated script. It contains two methods: forwards and backwards.
302
    3. We edit the generated script. It contains two methods: forwards and
303
    backwards.
296 304

  
297
    For database operations (column additions, alter tables etc) we use the South database API
298
    (http://south.aeracode.org/docs/databaseapi.html).
305
    For database operations (column additions, alter tables etc) we use the
306
    South database API (http://south.aeracode.org/docs/databaseapi.html).
299 307

  
300
    To access the data, we use the database reference (orm) provided as parameter in forwards, backwards method declarations
301
    in the migration script. For example:
308
    To access the data, we use the database reference (orm) provided as
309
    parameter in forwards, backwards method declarations in the migration
310
    script. For example:
302 311

  
303 312
    class Migration(DataMigration):
304 313

  
......
322 331

  
323 332
http://south.aeracode.org/
324 333

  
334

  
325 335
UI Testing
326 336
----------
327 337
The functional ui tests require the Selenium server and the synnefo app to 
......
332 342
    $ ./bin/python manage.py runserver &
333 343
    $ ./bin/python manage.py test ui
334 344

  
345

  
335 346
Test coverage
336 347
-------------
337 348

  

Also available in: Unified diff