Statistics
| Branch: | Tag: | Revision:

root / docs / upgrade / upgrade-0.13.rst @ 1cd3daa1

History | View | Annotate | Download (21.2 kB)

1
Upgrade to Synnefo v0.13
2
^^^^^^^^^^^^^^^^^^^^^^^^
3

    
4
The bulk of the upgrade to v0.13 is about user and quota migrations.
5
In summary, the migration process has 3 steps:
6

    
7
1. Run some commands and scripts to diagnose and extract some migration data
8
   while the OLD code is running, and BEFORE any changes are made.
9

    
10
2. Bring down services, upgrade packages, configure services, and perform
11
   django database migrations. These migrations do not need any interaction
12
   between services.
13

    
14
3. Initialize the Astakos quota system and bring the Astakos service up, since
15
   it will be needed during a second-phase of UUID and quota migrations, that
16
   also uses data extracted from step 1.
17

    
18

    
19
.. warning::
20
 
21
    It is strongly suggested that you keep separate database backups
22
    for each service after the completion of each of step.
23

    
24
1. Bring web services down, backup databases
25
============================================
26

    
27
1. All web services must be brought down so that the database maintains a
28
   predictable and consistent state during the migration process::
29

    
30
    # service gunicorn stop
31
    # service snf-dispatcher stop
32
    # etc.
33

    
34
2. Backup databases for recovery to a pre-migration state.
35

    
36
3. Keep the database servers running during the migration process
37

    
38

    
39
2. Prepare astakos user migration to case insensitive emails
40
============================================================
41

    
42
It is possible that two or more users have been registered with emails that
43
differ only in the case of its letters.  There can only be one of those
44
accounts after the migration, so the rest must be deleted.
45

    
46
Note that even if the users are deleted in Astakos, there still are duplicate
47
entries in Cyclades and Pithos.  For each service we need to reduce those
48
multiple accounts into one, either merging them together, or deleting and
49
discarding data from all but one.
50

    
51
.. _find_duplicate_emails:
52

    
53
2.1 Find duplicate email entries in Astakos
54
-------------------------------------------
55
(script: ``find_astakos_users_with_conflicting_emails.py``)::
56

    
57
    astakos-host$ cat << EOF > find_astakos_users_with_conflicting_emails.py
58
    #!/usr/bin/env python
59
    import os
60
    import sys
61

    
62
    os.environ['DJANGO_SETTINGS_MODULE'] = 'synnefo.settings'
63

    
64
    import astakos
65
    from astakos.im.models import AstakosUser as A
66

    
67

    
68
    def user_filter(user):
69
        return A.objects.filter(email__iexact=user.email).count() > 1
70

    
71
    all_users = list(A.objects.all())
72
    userlist = [(str(u.pk) + ': ' + str(u.email) + ' (' + str(u.is_active) + ', ' +
73
                 str(u.date_joined) + ')') for u in filter(user_filter, all_users)]
74

    
75
    sys.stderr.write("id email (is_active, creation date)\n")
76
    print "\n".join(userlist)
77
    EOF
78

    
79
    astakos-host$ python ./find_astakos_users_with_conflicting_emails.py
80

    
81
.. _remove_astakos_duplicate:
82

    
83
2.1 Remove duplicate users in Astakos by their id
84
-------------------------------------------------
85
(script: ``delete_astakos_users.py``)::
86

    
87
    astakos-host$ cat << EOF > delete_astakos_users.py
88
    #!/usr/bin/env python
89

    
90
    import os
91
    import sys
92
    from time import sleep
93

    
94
    os.environ['DJANGO_SETTINGS_MODULE'] = 'synnefo.settings'
95

    
96
    import astakos
97
    from astakos.im.models import AstakosUser as A
98

    
99

    
100
    def user_filter(user):
101
        return A.objects.filter(email__iexact=user.email).count() > 1
102

    
103
    argv = sys.argv
104
    argc = len(sys.argv)
105

    
106
    if argc < 2:
107
        print "Usage: ./delete_astakos_users.py <id>..."
108
        raise SystemExit()
109

    
110
    id_list = [int(x) for x in argv[1:]]
111

    
112
    print ""
113
    print "This will permanently delete the following users:\n"
114
    print "id  email (is_active, creation date)"
115
    print "--  --------------------------------"
116

    
117
    users = A.objects.filter(id__in=id_list)
118
    for user in users:
119
        print "%s: %s (%s, %s)" % (user.id, user.email, user.is_active,
120
                                   user.date_joined)
121

    
122
    print "\nExecute? (yes/no): ",
123
    line = raw_input().rstrip()
124
    if line != 'yes':
125
        print "\nCancelled"
126
        raise SystemExit()
127

    
128
    print "\nConfirmed."
129
    sleep(2)
130
    for user in users:
131
        print "deleting %s: %s" % (user.id, user.email)
132
        user.delete()
133

    
134
    EOF
135

    
136
    astakos-host$ python ./delete_astakos_users.py 30 40
137

    
138
.. warning::
139

    
140
    After deleting users with the ``delete_astakos_users.py`` script,
141
    check again with ``find_astakos_users_with_conflicting_emails.py``
142
    (as in :ref:`find_duplicate_emails`)
143
    to make sure that no duplicate email conflicts remain.
144

    
145

    
146
3. Upgrade Synnefo and configure settings
147
=========================================
148

    
149
3.1 Install the new versions of packages
150
----------------------------------------
151

    
152
::
153

    
154
    astakos.host$ apt-get install \
155
                            kamaki \
156
                            snf-common \
157
                            snf-webproject \
158
                            snf-quotaholder-app \
159
                            snf-astakos-app \
160

    
161

    
162
    cyclades.host$ apt-get install \
163
                            kamaki \
164
                            snf-common \
165
                            snf-webproject
166
                            snf-pithos-backend \
167
                            snf-cyclades-app \
168

    
169
                           
170
    pithos.host$ apt-get install \
171
                            kamaki \
172
                            snf-common \
173
                            snf-webproject
174
                            snf-pithos-backend \
175
                            snf-pithos-app \
176
                            snf-pithos-webclient \
177

    
178

    
179
    ganeti.node$ apt-get install \
180
                            kamaki \
181
                            snf-common \
182
                            snf-cyclades-gtools \
183
                            snf-pithos-backend \
184

    
185
.. note::
186

    
187
    Installing the packages will cause services to start. Make sure you bring
188
    them down again (at least ``gunicorn``, ``snf-dispatcher``)
189

    
190
3.2 Sync and migrate Django DB
191
------------------------------
192

    
193
.. note::
194

    
195
   If you are asked about stale content types during the migration process,
196
   answer 'no' and let the migration finish.
197

    
198
::
199

    
200
    astakos-host$ snf-manage syncdb
201
    astakos-host$ snf-manage migrate
202

    
203
    cyclades-host$ snf-manage syncdb
204
    cyclades-host$ snf-manage migrate
205

    
206
.. note::
207

    
208
    After the migration, Astakos has created uuids for all users,
209
    and has set the uuid as the public identifier of a user.
210
    This uuid is to be used both at other services (Cyclades, Pithos)
211
    and at the clientside (kamaki client settings).
212

    
213
    Duplicate-email users have been deleted earlier in
214
    :ref:`remove_astakos_duplicate`
215

    
216
3.3 Setup quota settings for all services
217
-----------------------------------------
218

    
219
Generally:
220

    
221
::
222

    
223
    # Service       Setting                       Value
224
    # quotaholder:  QUOTAHOLDER_URL            = https://quotaholder.host/quotaholder/v
225
    # quotaholder:  QUOTAHOLDER_TOKEN          = <random string>
226

    
227
    # astakos:      ASTAKOS_QUOTAHOLDER_TOKEN  = <the same random string>
228
    # astakos:      ASTAKOS_QUOTAHOLDER_URL    = https://quotaholder.host/quotaholder/v
229

    
230
    # cyclades:     CYCLADES_QUOTAHOLDER_TOKEN = <the same random string>
231
    # cyclades:     CYCLADES_QUOTAHOLDER_URL   = https://quotaholder.host/quotaholder/v
232
    # cyclades:     CYCLADES_USE_QUOTAHOLDER   = True
233

    
234

    
235
    # pithos:       PITHOS_QUOTAHOLDER_TOKEN   = <the same random string>
236
    # pithos:       PITHOS_QUOTAHOLDER_URL     = https://quotaholder.host/quotaholder/v
237
    # pithos:       PITHOS_USE_QUOTAHOLDER     = True
238
    # All services must match the quotaholder token and url configured for quotaholder.
239

    
240
Specifically:
241

    
242
On the Astakos host, edit ``/etc/synnefo/20-snf-astakos-app-settings.conf``:
243

    
244
::
245

    
246
    QUOTAHOLDER_URL = 'https://accounts.example.synnefo.org/quotaholder/v'
247
    QUOTAHOLDER_TOKEN = 'aExampleTokenJbFm12w'
248
    ASTAKOS_QUOTAHOLDER_TOKEN = 'aExampleTokenJbFm12w'
249
    ASTAKOS_QUOTAHOLDER_URL = 'https://accounts.example.synnefo.org/quotaholder/v'
250

    
251
On the Cyclades host, edit ``/etc/synnefo/20-snf-cyclades-app-quotas.conf``:
252

    
253
::
254

    
255
    CYCLADES_USE_QUOTAHOLDER = True
256
    CYCLADES_QUOTAHOLDER_URL = 'https://accounts.example.synnefo.org/quotaholder/v'
257
    CYCLADES_QUOTAHOLDER_TOKEN = 'aExampleTokenJbFm12w'
258

    
259
    # Set to False if astakos & cyclades are on the same host
260
    #CYCLADES_PROXY_USER_SERVICES = True
261

    
262
.. note::
263

    
264
    If Cylcades and Astakos are installed on the same server,
265
    set ``CYCLADES_PROXY_USER_SERVICES = False``
266

    
267

    
268
On the Pithos host, edit ``/etc/synnefo/20-snf-pithos-app-settings.conf``:
269

    
270
::
271

    
272
    PITHOS_QUOTAHOLDER_URL = 'https://accounts.example.synnefo.org/quotaholder/v'
273
    PITHOS_QUOTAHOLDER_TOKEN = 'aExampleTokenJbFm12w'
274
    PITHOS_USE_QUOTAHOLDER = False # will set to True after migration
275

    
276
.. note::
277

    
278
    During the migration it must be set, ``PITHOS_USE_QUOTAHOLDER = False``.
279
    Set to ``True`` once the migration is over.
280

    
281
3.4 Setup astakos
282
-----------------
283

    
284
- **Remove** this redirection from astakos front-end web server ::
285

    
286
        RewriteRule ^/login(.*) /im/login/redirect$1 [PT,NE]
287

    
288
    (see `<http://docs.dev.grnet.gr/synnefo/latest/quick-install-admin-guide.html#apache2-setup>`_)
289

    
290
- Enable users to change their contact email. Edit
291
``/etc/synnefo/20-snf-astakos-app-settings.conf`` ::
292

    
293
    ASTAKOS_EMAILCHANGE_ENABLED = True
294

    
295
- Rename the following (Astakos-specific) setting::
296

    
297
    ASTAKOS_DEFAULT_FROM_EMAIL
298
  
299
  to this (Django-specific) name::
300

    
301
    SERVER_EMAIL
302

    
303
- Instead of using the following (Astakos-specific) setting::
304

    
305
    ASTAKOS_DEFAULT_ADMIN_EMAIL
306

    
307
  include one or more entries in this (Django-specific) setting::
308

    
309
    ADMINS = (
310
        ('Joe Doe', 'doe@example.net'),
311
        ('Mary Jean', 'mary@example.net'),
312
    ) 
313

    
314
.. note::
315

    
316
    The ``SERVER_EMAIL`` and ``ADMINS`` settings are Django-specific.
317
    As such they will be the shared for any two (or more) services that happen
318
    to be collocated within the same application server (e.g. astakos &
319
    cyclades within the same gunicorn)
320

    
321
3.5 Setup Cyclades
322
------------------
323

    
324
- Run on the Astakos host ::
325

    
326
    # snf-manage service-list
327

    
328
- Set the Cyclades service token in
329
  ``/etc/synnefo/20-snf-cyclades-app-api.conf`` ::
330

    
331
    CYCLADES_ASTAKOS_SERVICE_TOKEN = 'asfasdf_CycladesServiceToken_iknl'
332

    
333
- Since version 0.13, Synnefo uses **VMAPI** in order to prevent sensitive data
334
  needed by 'snf-image' to be stored in Ganeti configuration (e.g. VM
335
  password). This is achieved by storing all sensitive information to a CACHE
336
  backend and exporting it via VMAPI. The cache entries are invalidated after
337
  the first request. Synnefo uses **memcached** as a django cache backend.
338
  To install, run on the Cyclades host::
339

    
340
        apt-get install memcached
341
        apt-get install python-memcache
342

    
343
  You will also need to configure Cyclades to use the memcached cache backend.
344
  Namely, you need to set IP address and port of the memcached daemon, and the
345
  default timeout (seconds tha value is stored in the cache). Edit
346
  ``/etc/synnefo/20-snf-cyclades-app-vmapi.conf`` ::
347

    
348
    VMAPI_CACHE_BACKEND = "memcached://127.0.0.1:11211/?timeout=3600"
349

    
350

    
351
  Finally, set the BASE_URL for the VMAPI, which is actually the base URL of
352
  Cyclades, again in ``/etc/synnefo/20-snf-cyclades-app-vmapi.conf``. Make sure
353
  the domain is exaclty the same, so that no re-directs happen ::
354

    
355
    VMAPI_BASE_URL = "https://cyclades.example.synnefo.org"
356

    
357
  .. note::
358

    
359
    - These settings are needed in all Cyclades workers.
360

    
361
    - VMAPI_CACHE_BACKEND just overrides django's CACHE_BACKEND setting
362

    
363
    - memcached must be reachable from all Cyclades workers.
364

    
365
    - For more information about configuring django to use memcached:
366
      https://docs.djangoproject.com/en/1.2/topics/cache
367

    
368
3.6 Setup Pithos
369
----------------
370

    
371
- Pithos forwards user catalog services to Astakos so that web clients may
372
  access them for uuid-displayname translations. Edit on the Pithos host
373
  ``/etc/synnefo/20-snf-pithos-app-settings.conf`` ::
374

    
375
    PITHOS_USER_CATALOG_URL    = https://accounts.example.synnefo.org/user_catalogs/
376
    PITHOS_USER_FEEDBACK_URL   = https://accounts.example.synnefo.org/feedback/
377
    PITHOS_USER_LOGIN_URL      = https://accounts.example.synnefo.org/login/
378

    
379
    # Set to False if astakos & pithos are on the same host
380
    #PITHOS_PROXY_USER_SERVICES = True
381

    
382
.. note::
383

    
384
    If Pithos and Astakos are installed on the same server,
385
    set ``PITHOS_PROXY_USER_SERVICES = False``
386

    
387
4. Start astakos and quota services
388
===================================
389
.. warning::
390

    
391
    To ensure consistency, prevent public access to astakos during migrations.
392
    This can be done via firewall or webserver access control.
393

    
394
Start (or restart, if running) the webserver and gunicorn on the Astakos host.
395
E.g.::
396

    
397
    # service apache2 start
398
    # service gunicorn start
399

    
400
.. _astakos-load-resources:
401

    
402
5. Load resource definitions into Astakos
403
=========================================
404

    
405
First, set the corresponding values on the following dict in
406
``/etc/synnefo/20-snf-astakos-app-settings.conf`` ::
407

    
408
    # Set the cloud service properties
409
    ASTAKOS_SERVICES = {
410
        'cyclades': {
411
    #        # Specifying the key 'url' will overwrite it.
412
    #        # Use this to (re)set service URL.
413
    #        'url': 'https://cyclades.example.synnefo.org/ui/',
414
    #        # order services in listings, cloudbar, etc.
415
    #        'order' : 1
416
            'resources': [{
417
                'name': 'disk',
418
                'group': 'compute',
419
                'uplimit': 30*1024*1024*1024,
420
                'unit': 'bytes',
421
                'desc': 'Virtual machine disk size'
422
                }, {
423
                'name': 'cpu',
424
                'group': 'compute',
425
                'uplimit': 6,
426
                'desc': 'Number of virtual machine processors'
427
                }, {
428
                'name': 'ram',
429
                'group': 'compute',
430
                'uplimit': 6*1024*1024*1024,
431
                'unit': 'bytes',
432
                'desc': 'Virtual machines'
433
                }, {
434
                'name': 'vm',
435
                'group': 'compute',
436
                'uplimit': 2,
437
                'desc': 'Number of virtual machines'
438
                }, {
439
                'name': 'network.private',
440
                'group': 'network',
441
                'uplimit': 1,
442
                'desc': 'Private networks'
443
                }
444
            ]
445
        },
446
        'pithos+': {
447
    #        # Use this to (re)set service URL.
448
    #        'url': 'https://pithos.example.synnefo.org/ui/',
449
    #        # order services in listings, cloudbar, etc.
450
    #        'order' : 2
451
            'resources':[{
452
                'name': 'diskspace',
453
                'group': 'storage',
454
                'uplimit': 5*1024*1024*1024,
455
                'unit': 'bytes',
456
                'desc': 'Pithos account diskspace'
457
                }]
458
        }
459
    }
460

    
461
.. note::
462

    
463
    The name of the Pithos service is ``pithos+``.
464
    If you have named your pithos service ``pithos``, without ``+``,
465
    then you must rename it::
466

    
467
        $ snf-manage service-list | grep pithos # find service id
468
        $ snf-manage service-update --name='pithos+' <service id> 
469

    
470
Then, configure and load the available resources per service
471
and associated default limits into Astakos. On the Astakos host run ::
472

    
473
     # snf-manage astakos-init --load-service-resources
474

    
475

    
476
.. note::
477

    
478
    Before v0.13, only `cyclades.vm`, `cyclades.network.private`,
479
    and `pithos+.diskspace` existed (not with these names,
480
    there were per-service settings).
481
    However, limits to the new resources must also be set.
482

    
483
    If the intetion is to keep a resource unlimited, (counting on that VM
484
    creation will be limited by other resources' limit) it is best to calculate
485
    a value that is too large to be reached because of other limits (and
486
    available flavours), but not much larger than needed because this might
487
    confuse users who do not readily understand that multiple limits apply and
488
    flavors are limited.
489

    
490

    
491
6. Migrate Services user names to uuids
492
=======================================
493

    
494

    
495
6.1 Double-check cyclades before user case/uuid migration
496
---------------------------------------------------------
497

    
498
::
499

    
500
    cyclades.host$ snf-manage cyclades-astakos-migrate-013 --validate
501

    
502
Duplicate user found?
503

    
504
- either *merge* (merge will merge all resources to one user)::
505

    
506
    cyclades.host$ snf-manage cyclades-astakos-migrate-013 --merge-user=kpap@grnet.gr
507

    
508
- or *delete* ::
509

    
510
    cyclades.host$ snf-manage cyclades-astakos-migrate-013 --delete-user=KPap@grnet.gr
511
    # (only KPap will be deleted not kpap)
512

    
513
6.2 Double-check pithos before user case/uuid migration
514
---------------------------------------------------------
515

    
516
::
517

    
518
    pithos.host$ snf-manage pithos-manage-accounts --list-duplicate
519

    
520
Duplicate user found?
521

    
522
If you want to migrate files first:
523

    
524
- *merge* (merge will merge all resources to one user)::
525

    
526
    pithos.host$ snf-manage pithos-manage-accounts --merge-accounts --src-account=SPapagian@grnet.gr --dest-account=spapagian@grnet.gr
527
    # (SPapagian@grnet.gr's contents will be merged into spapagian@grnet.gr, but SPapagian@grnet.gr account will still exist)
528

    
529
- and then *delete* ::
530

    
531
    pithos.host$ snf-manage pithos-manage-accounts --delete-account=SPapagian@grnet.gr
532
    # (only SPapagian@grnet.gr will be deleted not spapagian@grnet.gr)
533

    
534
If you do *NOT* want to migrate files just run the second step and delete
535
the duplicate account.
536

    
537
6.3 Migrate Cyclades users (email case/uuid)
538
--------------------------------------------
539

    
540
::
541

    
542
    cyclades.host$ snf-manage cyclades-astakos-migrate-013 --migrate-users
543

    
544
- if invalid usernames are found, verify that they do not exist in astakos::
545

    
546
    astakos.host$ snf-manage user-list
547

    
548
- if no user exists::
549

    
550
    cyclades.host$ snf-manage cyclades-astakos-migrate-013 --delete-user=<userid>
551

    
552
Finally, if you have set manually quotas for specific users inside
553
``/etc/synnefo/20-snf-cyclades-app-api.conf`` (in ``VMS_USER_QUOTA``,
554
``NETWORKS_USER_QUOTA`` make sure to update them so that:
555

    
556
1. There are no double entries wrt case sensitivity
557
2. Replace all user email addresses with the corresponding UUIDs
558

    
559
To find the UUIDs for step 2 run on the Astakos host ::
560

    
561
     # snf-manage user-list
562

    
563
6.4 Migrate Pithos user names
564
-----------------------------
565

    
566
Check if alembic has not been initialized ::
567

    
568
    pithos.host$ pithos-migrate current
569

    
570
- If alembic current is None (e.g. okeanos.io) ::
571

    
572
    pithos.host$ pithos-migrate stamp 3dd56e750a3
573

    
574
Then, migrate pithos account name to uuid::
575

    
576
    pithos.host$ pithos-migrate upgrade head
577

    
578
Finally, set this setting to ``True``::
579

    
580
    PITHOS_USE_QUOTAHOLDER = True
581

    
582

    
583
7. Migrate old quota limits
584
===========================
585

    
586
7.1 Migrate Pithos quota limits to Astakos
587
------------------------------------------
588

    
589
Migrate from pithos native to astakos/quotaholder.
590
This requires a file to be transfered from Cyclades to Astakos::
591

    
592
    pithos.host$ snf-manage pithos-export-quota --location=pithos-quota.txt
593
    pithos.host$ scp pithos-quota.txt astakos.host:
594
    astakos.host$ snf-manage user-set-initial-quota pithos-quota.txt
595

    
596
.. _export-quota-note:
597

    
598
.. note::
599

    
600
    `pithos-export-quota` will only export quotas that are not equal to the
601
    defaults in Pithos. Therefore, it is possible to both change or maintain
602
    the default quotas across the migration. To maintain quotas the new default
603
    pithos+.diskpace limit in Astakos must be equal to the (old) default quota
604
    limit in Pithos. Change either one of them make them equal.
605

    
606
    see :ref:`astakos-load-resources` on how to set the (new) default quotas in Astakos.
607

    
608
7.2 Migrate Cyclades quota limits to Astakos
609
--------------------------------------------
610

    
611
::
612

    
613
    cyclades.host$ snf-manage cyclades-export-quota --location=cyclades-quota.txt
614
    cyclades.host$ scp cyclades-quota.txt astakos.host:
615
    astakos.host$ snf-manage user-set-initial-quota cyclades-quota.txt
616

    
617
`cyclades-export-quota` will only export quotas that are not equal to the defaults.
618
See :ref:`note above <export-quota-note>`.
619

    
620
8. Enforce the new quota limits migrated to Astakos
621
===================================================
622
The following should report all users not having quota limits set
623
because the effective quota database has not been initialized yet. ::
624

    
625
    astakos.host$ snf-manage astakos-quota --verify
626

    
627
Initialize the effective quota database::
628

    
629
    astakos.host$ snf-manage astakos-quota --sync
630

    
631
This procedure may be used to verify and re-synchronize the effective quota
632
database with the quota limits that are derived from policies in Astakos
633
(initial quotas, project memberships, etc.)
634

    
635
9. Initialize resource usage
636
============================
637

    
638
The effective quota database (quotaholder) has just been initialized and knows
639
nothing of the current resource usage. Therefore, each service must send it in.
640

    
641
9.1 Initialize Pithos resource usage
642
------------------------------------
643

    
644
::
645

    
646
    pithos.host$ snf-manage pithos-reset-usage
647

    
648
9.2 Initialize Cyclades resource usage
649
--------------------------------------
650

    
651
::
652

    
653
    cyclades.host$ snf-manage cyclades-reset-usage
654

    
655
10. Install periodic project maintainance checks
656
================================================
657
In order to detect and effect project expiration,
658
a management command has to be run periodically
659
(depending on the required granularity, e.g. once a day/hour)::
660

    
661
    astakos.host$ snf-manage project-control --terminate-expired
662

    
663
A list of expired projects can be extracted with::
664

    
665
    astakos.host$ snf-manage project-control --list-expired
666

    
667

    
668
11. Restart all services
669
========================
670

    
671
Start (or restart, if running) all Synnefo services on all hosts.
672

    
673
::
674

    
675
    # service gunicorn restart
676
    # service snf-dispatcher restart
677
    # etc.