root / docs / upgrade-0.13.rst @ 5c088197
History | View | Annotate | Download (18.5 kB)
1 |
Upgrade to Synnefo v0.13 |
---|---|
2 |
^^^^^^^^^^^^^^^^^^^^^^^^ |
3 |
|
4 |
The bulk of the upgrade to v0.13 is about user and quota migrations. |
5 |
In summary, the migration process has 3 steps: |
6 |
|
7 |
1. Run some commands and scripts to diagnose and extract some migration data |
8 |
while the OLD code is running, and BEFORE any changes are made. |
9 |
|
10 |
2. Bring down services, upgrade packages, configure services, and perform |
11 |
django database migrations. These migrations do not need any interaction |
12 |
between services. |
13 |
|
14 |
3. Initialize the Astakos quota system and bring the Astakos service up, since |
15 |
it will be needed during a second-phase of UUID and quota migrations, that |
16 |
also uses data extracted from step 1. |
17 |
|
18 |
|
19 |
.. warning:: |
20 |
|
21 |
It is strongly suggested that you keep separate database backups |
22 |
for each service after the completion of each of step. |
23 |
|
24 |
1. Bring web services down, backup databases |
25 |
============================================ |
26 |
|
27 |
1. All web services must be brought down so that the database maintains a |
28 |
predictable and consistent state during the migration process. |
29 |
|
30 |
2. Backup databases for recovery to a pre-migration state. |
31 |
|
32 |
3. Keep the database servers running during the migration process |
33 |
|
34 |
|
35 |
2. Prepare astakos user migration to case insensitive emails |
36 |
============================================================ |
37 |
|
38 |
It is possible that two or more users have been registered with emails that |
39 |
differ only in the case of its letters. There can only be one of those |
40 |
accounts after the migration, so the rest must be deleted. |
41 |
|
42 |
Note that even if the users are deleted in Astakos, there still are duplicate |
43 |
entries in Cyclades and Pithos. For each service we need to reduce those |
44 |
multiple accounts into one, either merging them together, or deleting and |
45 |
discarding data from all but one. |
46 |
|
47 |
.. _find_duplicate_emails: |
48 |
|
49 |
2.1 Find duplicate email entries in Astakos |
50 |
------------------------------------------- |
51 |
(script: ``find_astakos_users_with_conflicting_emails.py``):: |
52 |
|
53 |
astakos-host$ cat << EOF > find_astakos_users_with_conflicting_emails.py |
54 |
#!/usr/bin/env python |
55 |
import os |
56 |
import sys |
57 |
|
58 |
os.environ['DJANGO_SETTINGS_MODULE'] = 'synnefo.settings' |
59 |
|
60 |
import astakos |
61 |
from astakos.im.models import AstakosUser as A |
62 |
|
63 |
|
64 |
def user_filter(user): |
65 |
return A.objects.filter(email__iexact=user.email).count() > 1 |
66 |
|
67 |
all_users = list(A.objects.all()) |
68 |
userlist = [(str(u.pk) + ': ' + str(u.email) + ' (' + str(u.is_active) + ', ' + |
69 |
str(u.date_joined) + ')') for u in filter(user_filter, all_users)] |
70 |
|
71 |
sys.stderr.write("id email (is_active, creation date)\n") |
72 |
print "\n".join(userlist) |
73 |
EOF |
74 |
|
75 |
astakos-host$ python ./find_astakos_users_with_conflicting_emails.py |
76 |
|
77 |
.. _remove_astakos_duplicate: |
78 |
|
79 |
2.1 Remove duplicate users in Astakos by their id |
80 |
------------------------------------------------- |
81 |
(script: ``delete_astakos_users.py``):: |
82 |
|
83 |
astakos-host$ cat << EOF > delete_astakos_users.py |
84 |
#!/usr/bin/env python |
85 |
|
86 |
import os |
87 |
import sys |
88 |
from time import sleep |
89 |
|
90 |
os.environ['DJANGO_SETTINGS_MODULE'] = 'synnefo.settings' |
91 |
|
92 |
import astakos |
93 |
from astakos.im.models import AstakosUser as A |
94 |
|
95 |
|
96 |
def user_filter(user): |
97 |
return A.objects.filter(email__iexact=user.email).count() > 1 |
98 |
|
99 |
argv = sys.argv |
100 |
argc = len(sys.argv) |
101 |
|
102 |
if argc < 2: |
103 |
print "Usage: ./delete_astakos_users.py <id>..." |
104 |
raise SystemExit() |
105 |
|
106 |
id_list = [int(x) for x in argv[1:]] |
107 |
|
108 |
print "" |
109 |
print "This will permanently delete the following users:\n" |
110 |
print "id email (is_active, creation date)" |
111 |
print "-- --------------------------------" |
112 |
|
113 |
users = A.objects.filter(id__in=id_list) |
114 |
for user in users: |
115 |
print "%s: %s (%s, %s)" % (user.id, user.email, user.is_active, |
116 |
user.date_joined) |
117 |
|
118 |
print "\nExecute? (yes/no): ", |
119 |
line = raw_input().rstrip() |
120 |
if line != 'yes': |
121 |
print "\nCancelled" |
122 |
raise SystemExit() |
123 |
|
124 |
print "\nConfirmed." |
125 |
sleep(2) |
126 |
for user in users: |
127 |
print "deleting %s: %s" % (user.id, user.email) |
128 |
user.delete() |
129 |
|
130 |
EOF |
131 |
|
132 |
astakos-host$ python ./delete_astakos_users.py 30 40 |
133 |
|
134 |
.. warning:: |
135 |
|
136 |
After deleting users with the ``delete_astakos_users.py`` script, |
137 |
check again with ``find_astakos_users_with_conflicting_emails.py`` |
138 |
(as in :ref:`find_duplicate_emails`) |
139 |
to make sure that no duplicate email conflicts remain. |
140 |
|
141 |
|
142 |
3. Upgrade Synnefo and configure settings |
143 |
========================================= |
144 |
|
145 |
3.1 Install the new versions of packages |
146 |
---------------------------------------- |
147 |
|
148 |
:: |
149 |
|
150 |
astakos.host$ apt-get install \ |
151 |
snf-common \ |
152 |
snf-webproject \ |
153 |
snf-quotaholder-app \ |
154 |
snf-astakos-app \ |
155 |
kamaki \ |
156 |
|
157 |
|
158 |
cyclades.host$ apt-get install \ |
159 |
snf-common \ |
160 |
snf-webproject |
161 |
snf-pithos-backend \ |
162 |
snf-cyclades-app \ |
163 |
kamaki \ |
164 |
|
165 |
|
166 |
pithos.host$ apt-get install \ |
167 |
snf-common \ |
168 |
snf-webproject |
169 |
snf-pithos-backend \ |
170 |
snf-pithos-app \ |
171 |
snf-pithos-webclient \ |
172 |
kamaki \ |
173 |
|
174 |
.. note:: |
175 |
|
176 |
If you get questioned about stale content types during the |
177 |
migration process, answer ``no`` and let the migration finish. |
178 |
|
179 |
|
180 |
3.2 Sync and migrate Django DB |
181 |
------------------------------ |
182 |
|
183 |
.. note:: |
184 |
|
185 |
If you are asked about stale content types during the migration process, |
186 |
answer 'no' and let the migration finish. |
187 |
|
188 |
:: |
189 |
|
190 |
astakos-host$ snf-manage syncdb |
191 |
astakos-host$ snf-manage migrate |
192 |
|
193 |
cyclades-host$ snf-manage syncdb |
194 |
cyclades-host$ snf-manage migrate |
195 |
|
196 |
.. note:: |
197 |
|
198 |
After the migration, Astakos has created uuids for all users, |
199 |
and has set the uuid as the public identifier of a user. |
200 |
This uuid is to be used both at other services (Cyclades, Pithos) |
201 |
and at the clientside (kamaki client settings). |
202 |
|
203 |
Duplicate-email users have been deleted earlier in |
204 |
:ref:`remove_astakos_duplicate` |
205 |
|
206 |
3.3 Setup quota settings for all services |
207 |
----------------------------------------- |
208 |
|
209 |
Generally: |
210 |
|
211 |
:: |
212 |
|
213 |
# Service Setting Value |
214 |
# quotaholder: QUOTAHOLDER_TOKEN = <random string> |
215 |
|
216 |
# astakos: ASTAKOS_QUOTAHOLDER_TOKEN = <the same random string> |
217 |
# astakos: ASTAKOS_QUOTAHOLDER_URL = https://quotaholder.host/quotaholder/v |
218 |
|
219 |
# cyclades: CYCLADES_QUOTAHOLDER_TOKEN = <the same random string> |
220 |
# cyclades: CYCLADES_QUOTAHOLDER_URL = http://quotaholder.host/quotaholder/v |
221 |
# cyclades: CYCLADES_USE_QUOTAHOLDER = True |
222 |
|
223 |
|
224 |
# pithos: PITHOS_QUOTAHOLDER_TOKEN = <the same random string> |
225 |
# pithos: PITHOS_QUOTAHOLDER_URL = http://quotaholder.host/quotaholder/v |
226 |
# All services must match the quotaholder token and url configured for quotaholder. |
227 |
|
228 |
Specifically: |
229 |
|
230 |
On the Astakos host, edit ``/etc/synnefo/20-snf-astakos-app-settings.conf``: |
231 |
|
232 |
:: |
233 |
|
234 |
QUOTAHOLDER_TOKEN = 'aExampleTokenJbFm12w' |
235 |
ASTAKOS_QUOTAHOLDER_TOKEN = 'aExampleTokenJbFm12w' |
236 |
ASTAKOS_QUOTAHOLDER_URL = 'https://accounts.synnefo.local/quotaholder/v' |
237 |
|
238 |
On the Cyclades host, edit ``/etc/synnefo/20-snf-cyclades-app-quotas.conf``: |
239 |
|
240 |
:: |
241 |
|
242 |
CYCLADES_USE_QUOTAHOLDER = True |
243 |
CYCLADES_QUOTAHOLDER_URL = 'https://accounts.synnefo.local/quotaholder/v' |
244 |
CYCLADES_QUOTAHOLDER_TOKEN = 'aExampleTokenJbFm12w' |
245 |
|
246 |
On the Pithos host, edit ``/etc/synnefo/20-snf-pithos-app-settings.conf``: |
247 |
|
248 |
:: |
249 |
|
250 |
PITHOS_QUOTAHOLDER_URL = 'https://accounts.synnefo.local/quotaholder/v' |
251 |
PITHOS_QUOTAHOLDER_TOKEN = 'aExampleTokenJbFm12w' |
252 |
|
253 |
3.4 Setup astakos |
254 |
----------------- |
255 |
|
256 |
- **Remove** this redirection from astakos front-end web server :: |
257 |
|
258 |
RewriteRule ^/login(.*) /im/login/redirect$1 [PT,NE] |
259 |
|
260 |
(see `<http://docs.dev.grnet.gr/synnefo/latest/quick-install-admin-guide.html#apache2-setup>`_) |
261 |
|
262 |
- Enable users to change their contact email. Edit |
263 |
``/etc/synnefo/20-snf-astakos-app-settings.conf`` :: |
264 |
|
265 |
ASTAKOS_EMAILCHANGE_ENABLED = True |
266 |
|
267 |
3.5 Setup Cyclades |
268 |
------------------ |
269 |
|
270 |
- Run on the Astakos host :: |
271 |
|
272 |
# snf-manage service-list |
273 |
|
274 |
- Set the Cyclades service token in ``/etc/synnefo/20-snf-cyclades-app-api.conf`` :: |
275 |
|
276 |
CYCLADES_ASTAKOS_SERVICE_TOKEN = 'asfasdf_CycladesServiceToken_iknl' |
277 |
|
278 |
- Since version 0.13, Synnefo uses **VMAPI** in order to prevent sensitive data |
279 |
needed by 'snf-image' to be stored in Ganeti configuration (e.g. VM |
280 |
password). This is achieved by storing all sensitive information to a CACHE |
281 |
backend and exporting it via VMAPI. The cache entries are invalidated after |
282 |
the first request. Synnefo uses **memcached** as a django cache backend. |
283 |
To install, run on the Cyclades host:: |
284 |
|
285 |
apt-get install memcached |
286 |
apt-get install python-memcache |
287 |
|
288 |
You will also need to configure Cyclades to use the memcached cache backend. |
289 |
Namely, you need to set IP address and port of the memcached daemon, and the |
290 |
default timeout (seconds tha value is stored in the cache). Edit |
291 |
``/etc/synnefo/20-snf-cyclades-app-vmapi.conf`` :: |
292 |
|
293 |
VMAPI_CACHE_BACKEND = "memcached://127.0.0.1:11211/?timeout=3600" |
294 |
|
295 |
|
296 |
Finally, set the BASE_URL for the VMAPI, which is actually the base URL of |
297 |
Cyclades, again in ``/etc/synnefo/20-snf-cyclades-app-vmapi.conf``:: |
298 |
|
299 |
VMAPI_BASE_URL = "https://cyclades.okeanos.grnet.gr/" |
300 |
|
301 |
.. note:: |
302 |
|
303 |
- These settings are needed in all Cyclades workers. |
304 |
|
305 |
- VMAPI_CACHE_BACKEND just overrides django's CACHE_BACKEND setting |
306 |
|
307 |
- memcached must be reachable from all Cyclades workers. |
308 |
|
309 |
- For more information about configuring django to use memcached: |
310 |
https://docs.djangoproject.com/en/1.2/topics/cache |
311 |
|
312 |
3.6 Setup Pithos |
313 |
---------------- |
314 |
|
315 |
- Pithos forwards user catalog services to Astakos so that web clients may |
316 |
access them for uuid-displayname translations. Edit on the Pithos host |
317 |
``/etc/synnefo/20-snf-pithos-app-settings.conf`` :: |
318 |
|
319 |
PITHOS_USER_CATALOG_URL = https://accounts.synnefo.local/user_catalogs/ |
320 |
PITHOS_USER_FEEDBACK_URL = https://accounts.synnefo.local/feedback/ |
321 |
PITHOS_USER_LOGIN_URL = https://accounts.synnefo.local/login/ |
322 |
#PITHOS_PROXY_USER_SERVICES = True # Set False if astakos & pithos are on the same host |
323 |
|
324 |
|
325 |
4. Start astakos and quota services |
326 |
=================================== |
327 |
Start the webserver and gunicorn on the Astakos host. E.g.:: |
328 |
|
329 |
# service apache2 start |
330 |
# service gunicorn start |
331 |
|
332 |
.. warning:: |
333 |
|
334 |
To ensure consistency, prevent public access to astakos during migrations. |
335 |
This can be done via firewall or webserver access control. |
336 |
|
337 |
.. _astakos-load-resources: |
338 |
|
339 |
5. Load resource definitions into Astakos |
340 |
========================================= |
341 |
|
342 |
First, set the corresponding values on the following dict in |
343 |
``/etc/synnefo/20-snf-astakos-app-settings.conf`` :: |
344 |
|
345 |
# Set the cloud service properties |
346 |
ASTAKOS_SERVICES = { |
347 |
'cyclades': { |
348 |
#This can also be set from a management command |
349 |
'url': 'https://cyclades.host/ui/', |
350 |
'order': 0, |
351 |
'resources': [{ |
352 |
'name':'disk', |
353 |
'group':'compute', |
354 |
'uplimit':300*1024*1024*1024, |
355 |
'unit':'bytes', |
356 |
'desc': 'Virtual machine disk size' |
357 |
},{ |
358 |
'name':'cpu', |
359 |
'group':'compute', |
360 |
'uplimit':24, |
361 |
'desc': 'Number of virtual machine processors' |
362 |
},{ |
363 |
'name':'ram', |
364 |
'group':'compute', |
365 |
'uplimit':40*1024*1024*1024, |
366 |
'unit':'bytes', |
367 |
'desc': 'Virtual machines' |
368 |
},{ |
369 |
'name':'vm', |
370 |
'group':'compute', |
371 |
'uplimit':5, |
372 |
'desc': 'Number of virtual machines' |
373 |
},{ |
374 |
'name':'network.private', |
375 |
'group':'network', |
376 |
'uplimit':5, |
377 |
'desc': 'Private networks' |
378 |
} |
379 |
] |
380 |
}, |
381 |
'pithos+': { |
382 |
'url': 'https://pithos.host/ui/', |
383 |
'order': 1, |
384 |
'resources':[{ |
385 |
'name':'diskspace', |
386 |
'group':'storage', |
387 |
'uplimit':20 * 1024 * 1024 * 1024, |
388 |
'unit':'bytes', |
389 |
'desc': 'Pithos account diskspace' |
390 |
}] |
391 |
} |
392 |
} |
393 |
|
394 |
Then, configure and load the available resources per service |
395 |
and associated default limits into Astakos. On the Astakos host run :: |
396 |
|
397 |
# snf-manage astakos-init --load-service-resources |
398 |
|
399 |
|
400 |
.. note:: |
401 |
|
402 |
Before v0.13, only `cyclades.vm`, `cyclades.network.private`, |
403 |
and `pithos+.diskspace` existed (not with this names, of course). |
404 |
However, limits to the new resources must also be set. |
405 |
|
406 |
If the intetion is to keep a resource unlimited, (counting on that VM |
407 |
creation will be limited by other resources' limit) it is best to calculate |
408 |
a value that is too large to be reached because of other limits (and |
409 |
available flavours), but not much larger than needed because this might |
410 |
confuse users who do not readily understand that multiple limits apply and |
411 |
flavors are limited. |
412 |
|
413 |
|
414 |
6. Migrate Services user names to uuids |
415 |
======================================= |
416 |
|
417 |
|
418 |
6.1 Double-check cyclades before user case/uuid migration |
419 |
--------------------------------------------------------- |
420 |
|
421 |
:: |
422 |
|
423 |
cyclades.host$ snf-manage cyclades-astakos-migrate-013 --validate |
424 |
|
425 |
Duplicate user found? |
426 |
|
427 |
- either *merge* (merge will merge all resources to one user):: |
428 |
|
429 |
cyclades.host$ snf-manage cyclades-astakos-migrate-013 --merge-user=kpap@grnet.gr |
430 |
|
431 |
- or *delete* :: |
432 |
|
433 |
cyclades.host$ snf-manage cyclades-astakos-migrate-013 --delete-user=KPap@grnet.gr |
434 |
# (only KPap will be deleted not kpap) |
435 |
|
436 |
6.2 Double-check pithos before user case/uuid migration |
437 |
--------------------------------------------------------- |
438 |
|
439 |
:: |
440 |
|
441 |
pithos.host$ snf-manage pithos-manage-accounts --list-duplicate |
442 |
|
443 |
Duplicate user found? |
444 |
|
445 |
If you want to migrate files first: |
446 |
|
447 |
- *merge* (merge will merge all resources to one user):: |
448 |
|
449 |
pithos.host$ snf-manage pithos-manage-accounts --merge-accounts --src-account=SPapagian@grnet.gr --dest-account=spapagian@grnet.gr |
450 |
# (SPapagian@grnet.gr's contents will be merged into spapagian@grnet.gr, but SPapagian@grnet.gr account will still exist) |
451 |
|
452 |
- and then *delete* :: |
453 |
|
454 |
pithos.host$ snf-manage pithos-manage-accounts --delete-account=SPapagian@grnet.gr |
455 |
# (only SPapagian@grnet.gr will be deleted not spapagian@grnet.gr) |
456 |
|
457 |
If you do *NOT* want to migrate files just run the second step and delete |
458 |
the duplicate account. |
459 |
|
460 |
6.3 Migrate Cyclades users (email case/uuid) |
461 |
-------------------------------------------- |
462 |
|
463 |
:: |
464 |
|
465 |
cyclades.host$ snf-manage cyclades-astakos-migrate-013 --migrate-users |
466 |
|
467 |
- if invalid usernames are found, verify that they do not exist in astakos:: |
468 |
|
469 |
astakos.host$ snf-manage user-list |
470 |
|
471 |
- if no user exists:: |
472 |
|
473 |
cyclades.host$ snf-manage cyclades-astakos-migrate-013 --delete-user=<userid> |
474 |
|
475 |
Finally, if you have set manually quotas for specific users inside |
476 |
``/etc/synnefo/20-snf-cyclades-app-api.conf`` (in ``VMS_USER_QUOTA``, |
477 |
``NETWORKS_USER_QUOTA`` make sure to update them so that: |
478 |
|
479 |
1. There are no double entries wrt case sensitivity |
480 |
2. Replace all user email addresses with the corresponding UUIDs |
481 |
|
482 |
To find the UUIDs for step 2 run on the Astakos host :: |
483 |
|
484 |
# snf-manage user-list |
485 |
|
486 |
6.4 Migrate Pithos user names |
487 |
----------------------------- |
488 |
|
489 |
Check if alembic has not been initialized :: |
490 |
|
491 |
pithos.host$ pithos-migrate current |
492 |
|
493 |
- If alembic current is None (e.g. okeanos.io) :: |
494 |
|
495 |
pithos.host$ pithos-migrate stamp 3dd56e750a3 |
496 |
|
497 |
Finally, migrate pithos account name to uuid:: |
498 |
|
499 |
pithos.host$ pithos-migrate upgrade head |
500 |
|
501 |
7. Migrate old quota limits |
502 |
=========================== |
503 |
|
504 |
7.1 Migrate Pithos quota limits to Astakos |
505 |
------------------------------------------ |
506 |
|
507 |
Migrate from pithos native to astakos/quotaholder. |
508 |
This requires a file to be transfered from Cyclades to Astakos:: |
509 |
|
510 |
pithos.host$ snf-manage pithos-export-quota --location=pithos-quota.txt |
511 |
pithos.host$ rsync -avP pithos-quota.txt astakos.host: |
512 |
astakos.host$ snf-manage user-set-initial-quota pithos-quota.txt |
513 |
|
514 |
.. _export-quota-note: |
515 |
|
516 |
.. note:: |
517 |
|
518 |
`pithos-export-quota` will only export quotas that are not equal to the |
519 |
defaults in Pithos. Therefore, it is possible to both change or maintain |
520 |
the default quotas across the migration. To maintain quotas the new default |
521 |
pithos+.diskpace limit in Astakos must be equal to the (old) default quota |
522 |
limit in Pithos. Change either one of them make them equal. |
523 |
|
524 |
see :ref:`astakos-load-resources` on how to set the (new) default quotas in Astakos. |
525 |
|
526 |
7.2 Migrate Cyclades quota limits to Astakos |
527 |
-------------------------------------------- |
528 |
|
529 |
:: |
530 |
|
531 |
cyclades.host$ snf-manage cyclades-export-quota --location=cyclades-quota.txt |
532 |
cyclades.host$ rsync -avP cyclades-quota.txt astakos.host: |
533 |
astakos.host$ snf-manage user-set-initial-quota cyclades-quota.txt |
534 |
|
535 |
`cyclades-export-quota` will only export quotas that are not equal to the defaults. |
536 |
See :ref:`note above <export-quota-note>`. |
537 |
|
538 |
8. Enforce the new quota limits migrated to Astakos |
539 |
=================================================== |
540 |
The following should report all users not having quota limits set |
541 |
because the effective quota database has not been initialized yet. :: |
542 |
|
543 |
astakos.host$ snf-manage astakos-quota --verify |
544 |
|
545 |
Initialize the effective quota database:: |
546 |
|
547 |
astakos.host$ snf-manage astakos-quota --sync |
548 |
|
549 |
This procedure may be used to verify and re-synchronize the effective quota |
550 |
database with the quota limits that are derived from policies in Astakos |
551 |
(initial quotas, project memberships, etc.) |
552 |
|
553 |
9. Initialize resource usage |
554 |
============================ |
555 |
|
556 |
The effective quota database (quotaholder) has just been initialized and knows |
557 |
nothing of the current resource usage. Therefore, each service must send it in. |
558 |
|
559 |
9.1 Initialize Pithos resource usage |
560 |
------------------------------------ |
561 |
|
562 |
:: |
563 |
|
564 |
pithos.host$ snf-manage pithos-reset-usage |
565 |
|
566 |
9.2 Initialize Cyclades resource usage |
567 |
-------------------------------------- |
568 |
|
569 |
:: |
570 |
|
571 |
cyclades.host$ snf-manage cyclades-reset-usage |
572 |
|
573 |
10. Install periodic project maintainance checks |
574 |
================================================ |
575 |
In order to detect and effect project expiration, |
576 |
a management command has to be run periodically |
577 |
(depending on the required granularity, e.g. once a day/hour):: |
578 |
|
579 |
astakos.host$ snf-manage project-control --terminate-expired |
580 |
|
581 |
A list of expired projects can be extracted with:: |
582 |
|
583 |
astakos.host$ snf-manage project-control --list-expired |
584 |
|