Revision 1cd3daa1

b/docs/admin-guide.rst
1166 1166
.. toctree::
1167 1167
   :maxdepth: 1
1168 1168

  
1169
   upgrade-0.13
1169
   v0.12 -> v0.13 <upgrade/upgrade-0.13>
1170 1170

  
1171
Older Cyclades upgrade notes
1171
Older Cyclades Upgrade Notes
1172 1172
============================
1173 1173

  
1174 1174
.. toctree::
1175 1175
   :maxdepth: 2
1176 1176

  
1177
   cyclades-upgrade
1177
   upgrade/cyclades-upgrade
1178 1178

  
1179 1179
Changelog
1180 1180
=========
/dev/null
1
Upgrade
2
=======
3

  
4
This file documents the upgrade to newer versions of the Synnefo software.
5
For more information, please see deployment guide.
6

  
7

  
8
v0.7.4 -> v0.8
9
--------------
10

  
11
Synnefo is now distributed packaged in python/debian packages. You should
12
consider the following when migrating from projects previously deployed from
13
source:
14
    
15
    * To keep consistency with future changes, change all ``python manage.py``
16
      calls to refer to the shipped ``snf-manage`` console script.
17
      This includes automation scripts, service scripts etc.
18

  
19
      Same applies for calls to ``dispatcher.py``, ``snf-tools/admin.py``,
20
      ``snf-tools/burnin.py`` and ``snf-tools/cloud.py`` which are replaced
21
      by ``snf-dispatcher``, ``snf-admin``, ``snf-burnin`` and ``snf-cloud``,
22
      respectively.
23

  
24
    * Copy custom settings files located in project's ``settings.d`` directory
25
      to ``/etc/synnefo/`` directory.
26

  
27
    * Migrate location from which :ref:`static files <static-files>` are served from.
28

  
29
.. warning::
30
   
31
   Synnefo settings have been refactored as part of the
32
   :ref:`snf-common <snf-common>` component. File locations may be inaccurate
33
   and the list of changed settings incomplete.  Please consult the actual
34
   files installed under ``/etc/synnefo`` as the authoritative source for new
35
   settings.
36

  
37
NEW APPS
38
    * The ``synnefo.ui.userdata`` application has been added in
39
      :file:`settings.d/00-apps.conf`. Application urls appended in
40
      :file:`ui/urls.py`.
41
      If no custom ``ROOT_URLCONF`` module is used, no url change is needed.
42
    * The new app has migrations defined.
43
      Use ``snf-manage migrate`` to migrate *all* apps.
44

  
45
NEW/UPDATED SETTINGS
46
    * ``BYPASS_AUTHENTICATION_TOKEN_SECRET`` replaces ``BYPASS_AUTHENTICATION_TOKEN``
47
      in :file:`settings/common/aai.py`.
48
    * New config file :file:`31-userdata.conf`, containing userdata app settings
49
    * ``USERDATA_SSH_KEY_LENGTH`` in :file:`31-userdata.conf`:
50
      Key length in bits for generated ssh keys
51
    * ``USERDATA_SSH_KEY_EXPONENT`` in :file:`31-userdata.conf`:
52
      Generated SSH key exponent
53
    * ``USERDATA_MAX_SSH_KEYS_PER_USER`` in :file:`31-userdata.conf`:
54
      Maximum number of ssh keys a user is allowed to have
55
    * ``UI_SUPPORT_SSH_OS_LIST``, in :file:`30-ui.conf`:
56
      A list of os names that support ssh public key assignment
57
    * ``UI_OS_DEFAULT_USER_MAP``, in :file:`30-ui.conf`:
58
      OS/username map to identify default user name for a specific os
59
    * ``VM_CREATE_NAME_TPL``, in :file:`30-ui.conf`:
60
      Template to be used for suggesting the user a default name for newly
61
      created VMs. String {0} gets replaced by the value of metadata key "os"
62
      for the Image.
63
    * ``UI_FLAVORS_DISK_TEMPLATES_INFO`` added in :file:`30-ui.conf`:
64
      Name/description metadata for the available flavor disk templates
65
    * ``VM_CREATE_SUGGESTED_FLAVORS`` in :file:`30-ui.conf`:
66
      Updated flavor data to include disk_template value.
67
    * ``GANETI_DISK_TEMPLATES`` and ``DEFAULT_GANETI_DISK_TEMPLATE`` in :file:`20-api.conf`:
68
      The list of disk templates supported by the Ganeti backend.
69
      The default template to be used when migrating Flavors with no value for
70
      disk_template (i.e., 'drbd').
71
    * ``UI_COMPUTE_URL``, ``UI_GLANCE_URL`` in ui app:
72
      Configurable API endpoints
73
    * ``UI_ENABLE_GLANCE`` in ui app:
74
      Whether or not UI should retrieve images from the Glance API endpoint
75
      set in ``UI_GLANCE_API_URL``. If setting is set to False, ui will request
76
      images using calls to the Cyclades API.
77
UI
78
    * ``synnefo.ui.userdata`` application has been added in ``INSTALLED_APPS``.
79
      Database migration is needed for the creation of application db tables.
80

  
81
API
82
    * A new ``disk_template`` attribute has been added to Flavors.
83
      ``GANETI_DISK_TEMPLATES`` and ``DEFAULT_GANETI_DISK_TEMPLATE`` have been
84
      added in :file:`20-api.conf` to control its value. A database migration is
85
      needed.
86

  
87
PLANKTON
88
    Plankton is a new image service that has been added as a separate app. The
89
    images are stored in backend of Pithos and thus it must be configured to
90
    access the DB and directory that Pithos uses to store its data. These
91
    settings are defined in the new plankton.py file in settings.
92
    
93
    Plankton stores and looks for images in the Pithos container named
94
    ``PITHOS_IMAGE_CONTAINER``.
95
    
96
    There is a Pithos account that is reserved for system images. This account
97
    is defined in ``SYSTEM_IMAGES_OWNER``.
98

  
99
ADMIN TOOLS
100
    * A new ``--disk-template`` flag has been added to snf-admin to choose a
101
      disk template when creating flavors. Similarly, ``disk_template`` support
102
      in flavors has beed added to the admin web interface.
103

  
104

  
105
v0.7.3 -> v0.7.4
106
----------------
107

  
108
OKEANOS INTRO
109
    * News section added. News content can be remotely retrieved using
110
      OKEANOS_NEWS_CONTENT_URL settings option.
111

  
112

  
113
v0.6.2 -> v0.7
114
--------------
115

  
116
HTML TEMPLATES
117
    * Included a generic service unavailable template based on
118
      generic_info_tpl.html, as ui/static/service_unavailable.html.
119

  
120
NEW DEPENDENCIES
121
    * python-unittest2, python-paramiko needed by the new integration
122
      test suite snf-tools/burnin. Paramiko shipped with Squeeze uses
123
      broken RandomPool, produces warnings, see #576697.
124
    * snf-image must be installed on all Ganeti hosts, see README.deploy,
125
      Step 13.
126

  
127
COMPONENTS
128
    * snf-admin has been updated with new functionality, be sure to upgrade any
129
      locally installed versions.
130
    * snf-image replaces snf-ganeti-instance-image as the Ganeti OS provider
131
      used by Synnefo, and can live alongside snf-ganeti-instance-image.
132
      Once snf-image has been deployed on all Ganeti nodes, be sure to modify
133
      the value of settings.d/10-backend.conf:GANETI_CREATEINSTANCE_KWARGS
134
      to use 'snf-image+default'.
135

  
136
UI STATIC FILES
137
    * A major reorganization of static files has been commited. All ui and
138
      invitations static files has been moved in their own separate directory
139
      (ui/static/snf, ui/static/invitations).
140
    * UI static files base url is now configurable via UI_MEDIA_URL.
141
    * A symlink named after the current version of the ui application has been
142
      committed in ui/static directory. The symlink will get updated after each
143
      version upgrade to allow us provide unique urls of static files between
144
      after each upgrade, see #1460.
145

  
146
NEW/UPDATED SETTINGS
147
    * EMAIL_SUBJECT_PREFIX:
148
      Prepended to automated emails, set to someting that uniquely identifies
149
      the deployment.
150
    * MAX_PERSONALITY and MAX_PERSONALITY_SIZE in 20-api.conf:
151
      Maximum number of files to be injected in newly created servers,
152
      maximum total size of encoded file contents.
153
    * Renamed SUGGESTED_FLAVORS to VM_CREATE_SUGGESTED_FLAVORS in 30-ui.conf
154
    * VM_CREATE_SUGGESTED_ROLES in 30-ui.conf:
155
      A list of suggested vm roles to display to user on create wizard.
156
    * UI_DELAY_ON_BLUR in 30-ui.conf:
157
      Whether to increase the intervals of recurrent requests (networks/vms 
158
      update) if window loses its focus.
159
    * UI_BLUR_DELAY in 30-ui.conf:
160
      The value of update intervals if window loses its focus.
161
      Considered only if `UI_DELAY_ON_BLUR` is set to True.
162
    * UI_UPDATE_HIDDEN_VIEWS in 30-ui.conf:
163
      Whether not visible vm views will update their content if vm changes.
164
    * UI_SKIP_TIMEOUTS in 30-ui.conf:
165
      After how many timeouts of reccurent ajax requests to display the timeout
166
      error overlay.
167
    * UI_HANDLE_WINDOW_EXCEPTIONS in 30-ui.conf:
168
      Whether UI should display error overlay for all Javascript exceptions.
169
    * UI_MEDIA_URL in 30-ui.conf:
170
      Base url for ui static files, 
171
      defaults to MEDIA_URL + 'snf-<latest_ui_version>/'.
172
    * MEDIA_URL changed in 00-site.conf:
173
      Changed to '/static/' since it is now used in ui app.
174
    * TEMPLATE_CONTEXT_PROCESSORS changed in 00-apps.conf:
175
      added 'django.core.context_processors.media' to allow access of MEDIA_URL
176
      in template files.
177
    * GANETI_CREATEINSTANCE_KWARGS in 10-backend.conf:
178
      Must be updated to use snf-image as the Ganeti OS provider.
179
      
180
    
181
DB MIGRATION
182
    * Uniqueness constraints have been added to the metadata models.
183
      A database migration is needed.
184

  
185
LOGGING
186
    * A new logging mechanism has been implemeted. Please see 00-logging.conf
187
      under settings.d/ and read the relevant section in README.admin for more
188
      info.
189

  
190

  
191
v0.6.1 -> v0.6.2
192
----------------
193

  
194
ADMIN INTERFACE
195
    * The models were changed so that Flavors and SynnefoUsers now have
196
      a deleted state. The admin tools were updated so that models are
197
      now marked as deleted instead of actually being deleted from the DB.
198
      A database migration is needed.
199

  
200
COMPONENTS
201
    * Only the Django webapp is affected, must restart the logic dispatcher
202
      due to DB migration taking place.
203

  
204

  
205
v0.5.5 -> v0.6
206
--------------
207

  
208
ADMIN INTERFACE
209
    * A new Web-based admin interface is available under /admin.
210
      It is accessible by users of type 'ADMIN' in the DB, with
211
      their authentication token. "snf-admin user modify" may be used
212
      to change the type of a specific user.
213

  
214
RECONCILIATION
215
    * Implemented new reconciliation management command, please see
216
      ./manage.py reconcile --help and README.admin for more info.
217
      Recommended to run ./manage.py reconcile --detect-all periodically,
218
      via cron.
219

  
220
GANETI-INSTANCE-IMAGE
221
    * A new version of ganeti-instance-image is required (v0.5.1-1-snf1).
222
      The new version is available for download as a debian package:
223
      https://code.grnet.gr/projects/gnt-instance-image/files
224

  
225
COMPONENTS
226
    * snf-cyclades-gtools must be rebuilt, see snf-cyclades-gtools/debian/
227
      for Debian packaging.
228

  
229
NEW SETTINGS
230
    * 30-ui.conf:SUGGESTED_FLAVORS
231
    * 30-ui.conf:VM_IMAGE_COMMON_METADATA
232

  
233

  
234
v0.5.4 -> v0.5.5
235
----------------
236

  
237
LOGGING
238
    * Changed the default logging settings for the dispatcher to also log
239
      to /var/log/synnefo/dispatcher.log, redirecting stderr and stdout there
240

  
241
QUEUES
242
    * Changed default routing key naming for queues. Queues must be redeclared.
243
      The suggested upgrade path is to delete ALL (even obsolete) existing
244
      queues and restart the dispatcher. To do so, download amqp-utils from
245

  
246
          https://github.com/dougbarth/amqp-utils
247

  
248
      and run the amqp-deleteq tool for each declared queue. Alternatively,
249
      amqp-utils can be installed with: sudo gem install amqp-utils.
250

  
251
UI
252
    * Feedback form now uses django native send_mail for sending emails.
253
      Proper django settings should be set for feedback mails to work
254
      (https://docs.djangoproject.com/en/dev/topics/email/)
255

  
256
COMPONENTS
257
    * snf-cyclades-gtools must be rebuilt, see snf-cyclades-gtools/debian/
258
      for Debian packaging.
259

  
260

  
261
v0.5.3 -> v0.5.4
262
----------------
263

  
264
REPOSITORY
265
    * Split Synnefo Ganeti tools to snf-cyclades-gtools, with Debian packaging
266

  
267
REMOVED APPS
268
    * The ganeti/ app has been removed from the Django project completely.
269
      Any explicit references to it in Django settings must be removed.
270

  
271
DJANGO SETTINGS
272
    * snf-cyclades-gtools is configured independently from Django,
273
      need to add proper /etc/synnefo/settings.conf
274
    * Removed 15-queues.conf: fix_amqp_settings (no need to call it anywhere)
275
    * Removed settings.d/98-ganeti-* due to split of snf-cyclades-gtools
276
    * ~okeanos intro: OKEANOS_VIDEO_URL: Changed from string to dict
277
    * ~okeanos intro: OKEANOS_VIDEO_POSTER_IMAGE_URL: New setting
278
    * ~okeanos intro: OKEANOS_VIDEO_FLOWPLAYER_URL: New setting
279

  
280
DB MIGRATIONS
281
    * 0018_auto__add_field_virtualmachine_buildpercentage
282

  
283
PACKAGING
284
    * Split Synnefo Ganeti tools to separate snf-cyclades-gtools Debian package
285

  
286
NEW DEPENDENCIES
287
    * python-prctl: Needed by the snf-progress-monitor,
288
      specified as a dependency of the snf-cyclades-gtools Debian package.
289

  
290
EXTERNAL COMPONENTS
291
    * Ganeti Instance Image must be upgraded to support progress monitoring,
292
      please see README.deploy.
293

  
294

  
295
v0.5.2 -> v0.5.3
296
----------------
297

  
298
NEW SETTINGS
299
    * 30-ui.conf:LOGOUT_URL
300
    * 00-admins.conf:DEFAULT_FROM_EMAIL
301
    * 90-okeanos.conf.sample:LOGOUT_URL
302

  
303
REMOVED SETTINGS
304
    * 00-admins.conf:SYSTEM_EMAIL_ADDR
305
    * 90-okeanos.conf.sample:APP_INSTALL_URL
306

  
307

  
308
v0.5.1 -> v0.5.2
309
----------------
310

  
311
NEW SETTINGS
312
    * 10-backend.py:GANETI_CREATEINSTANCE_KWARGS
313

  
314
REMOVED SETTINGS
315
    * 10-backend.conf:GANETI_OS_PROVIDER
316
    * 20-api.conf:GANETI_DISK_TEMPLATE
317

  
318
BACKEND CHANGES
319
    * Need to patch Ganeti, file:
320
      lib/python2.6/site-packages/ganeti/rapi/rlib2.py
321
      to honor the wait_for_sync flag, see Synnefo #835.
322
      Patch provided under contrib/patches/ganeti-rlib2.py-v0.5.2
323

  
/dev/null
1
Upgrade to Synnefo v0.13
2
^^^^^^^^^^^^^^^^^^^^^^^^
3

  
4
The bulk of the upgrade to v0.13 is about user and quota migrations.
5
In summary, the migration process has 3 steps:
6

  
7
1. Run some commands and scripts to diagnose and extract some migration data
8
   while the OLD code is running, and BEFORE any changes are made.
9

  
10
2. Bring down services, upgrade packages, configure services, and perform
11
   django database migrations. These migrations do not need any interaction
12
   between services.
13

  
14
3. Initialize the Astakos quota system and bring the Astakos service up, since
15
   it will be needed during a second-phase of UUID and quota migrations, that
16
   also uses data extracted from step 1.
17

  
18

  
19
.. warning::
20
 
21
    It is strongly suggested that you keep separate database backups
22
    for each service after the completion of each of step.
23

  
24
1. Bring web services down, backup databases
25
============================================
26

  
27
1. All web services must be brought down so that the database maintains a
28
   predictable and consistent state during the migration process::
29

  
30
    # service gunicorn stop
31
    # service snf-dispatcher stop
32
    # etc.
33

  
34
2. Backup databases for recovery to a pre-migration state.
35

  
36
3. Keep the database servers running during the migration process
37

  
38

  
39
2. Prepare astakos user migration to case insensitive emails
40
============================================================
41

  
42
It is possible that two or more users have been registered with emails that
43
differ only in the case of its letters.  There can only be one of those
44
accounts after the migration, so the rest must be deleted.
45

  
46
Note that even if the users are deleted in Astakos, there still are duplicate
47
entries in Cyclades and Pithos.  For each service we need to reduce those
48
multiple accounts into one, either merging them together, or deleting and
49
discarding data from all but one.
50

  
51
.. _find_duplicate_emails:
52

  
53
2.1 Find duplicate email entries in Astakos
54
-------------------------------------------
55
(script: ``find_astakos_users_with_conflicting_emails.py``)::
56

  
57
    astakos-host$ cat << EOF > find_astakos_users_with_conflicting_emails.py
58
    #!/usr/bin/env python
59
    import os
60
    import sys
61

  
62
    os.environ['DJANGO_SETTINGS_MODULE'] = 'synnefo.settings'
63

  
64
    import astakos
65
    from astakos.im.models import AstakosUser as A
66

  
67

  
68
    def user_filter(user):
69
        return A.objects.filter(email__iexact=user.email).count() > 1
70

  
71
    all_users = list(A.objects.all())
72
    userlist = [(str(u.pk) + ': ' + str(u.email) + ' (' + str(u.is_active) + ', ' +
73
                 str(u.date_joined) + ')') for u in filter(user_filter, all_users)]
74

  
75
    sys.stderr.write("id email (is_active, creation date)\n")
76
    print "\n".join(userlist)
77
    EOF
78

  
79
    astakos-host$ python ./find_astakos_users_with_conflicting_emails.py
80

  
81
.. _remove_astakos_duplicate:
82

  
83
2.1 Remove duplicate users in Astakos by their id
84
-------------------------------------------------
85
(script: ``delete_astakos_users.py``)::
86

  
87
    astakos-host$ cat << EOF > delete_astakos_users.py
88
    #!/usr/bin/env python
89

  
90
    import os
91
    import sys
92
    from time import sleep
93

  
94
    os.environ['DJANGO_SETTINGS_MODULE'] = 'synnefo.settings'
95

  
96
    import astakos
97
    from astakos.im.models import AstakosUser as A
98

  
99

  
100
    def user_filter(user):
101
        return A.objects.filter(email__iexact=user.email).count() > 1
102

  
103
    argv = sys.argv
104
    argc = len(sys.argv)
105

  
106
    if argc < 2:
107
        print "Usage: ./delete_astakos_users.py <id>..."
108
        raise SystemExit()
109

  
110
    id_list = [int(x) for x in argv[1:]]
111

  
112
    print ""
113
    print "This will permanently delete the following users:\n"
114
    print "id  email (is_active, creation date)"
115
    print "--  --------------------------------"
116

  
117
    users = A.objects.filter(id__in=id_list)
118
    for user in users:
119
        print "%s: %s (%s, %s)" % (user.id, user.email, user.is_active,
120
                                   user.date_joined)
121

  
122
    print "\nExecute? (yes/no): ",
123
    line = raw_input().rstrip()
124
    if line != 'yes':
125
        print "\nCancelled"
126
        raise SystemExit()
127

  
128
    print "\nConfirmed."
129
    sleep(2)
130
    for user in users:
131
        print "deleting %s: %s" % (user.id, user.email)
132
        user.delete()
133

  
134
    EOF
135

  
136
    astakos-host$ python ./delete_astakos_users.py 30 40
137

  
138
.. warning::
139

  
140
    After deleting users with the ``delete_astakos_users.py`` script,
141
    check again with ``find_astakos_users_with_conflicting_emails.py``
142
    (as in :ref:`find_duplicate_emails`)
143
    to make sure that no duplicate email conflicts remain.
144

  
145

  
146
3. Upgrade Synnefo and configure settings
147
=========================================
148

  
149
3.1 Install the new versions of packages
150
----------------------------------------
151

  
152
::
153

  
154
    astakos.host$ apt-get install \
155
                            kamaki \
156
                            snf-common \
157
                            snf-webproject \
158
                            snf-quotaholder-app \
159
                            snf-astakos-app \
160

  
161

  
162
    cyclades.host$ apt-get install \
163
                            kamaki \
164
                            snf-common \
165
                            snf-webproject
166
                            snf-pithos-backend \
167
                            snf-cyclades-app \
168

  
169
                           
170
    pithos.host$ apt-get install \
171
                            kamaki \
172
                            snf-common \
173
                            snf-webproject
174
                            snf-pithos-backend \
175
                            snf-pithos-app \
176
                            snf-pithos-webclient \
177

  
178

  
179
    ganeti.node$ apt-get install \
180
                            kamaki \
181
                            snf-common \
182
                            snf-cyclades-gtools \
183
                            snf-pithos-backend \
184

  
185
.. note::
186

  
187
    Installing the packages will cause services to start. Make sure you bring
188
    them down again (at least ``gunicorn``, ``snf-dispatcher``)
189

  
190
3.2 Sync and migrate Django DB
191
------------------------------
192

  
193
.. note::
194

  
195
   If you are asked about stale content types during the migration process,
196
   answer 'no' and let the migration finish.
197

  
198
::
199

  
200
    astakos-host$ snf-manage syncdb
201
    astakos-host$ snf-manage migrate
202

  
203
    cyclades-host$ snf-manage syncdb
204
    cyclades-host$ snf-manage migrate
205

  
206
.. note::
207

  
208
    After the migration, Astakos has created uuids for all users,
209
    and has set the uuid as the public identifier of a user.
210
    This uuid is to be used both at other services (Cyclades, Pithos)
211
    and at the clientside (kamaki client settings).
212

  
213
    Duplicate-email users have been deleted earlier in
214
    :ref:`remove_astakos_duplicate`
215

  
216
3.3 Setup quota settings for all services
217
-----------------------------------------
218

  
219
Generally:
220

  
221
::
222

  
223
    # Service       Setting                       Value
224
    # quotaholder:  QUOTAHOLDER_URL            = https://quotaholder.host/quotaholder/v
225
    # quotaholder:  QUOTAHOLDER_TOKEN          = <random string>
226

  
227
    # astakos:      ASTAKOS_QUOTAHOLDER_TOKEN  = <the same random string>
228
    # astakos:      ASTAKOS_QUOTAHOLDER_URL    = https://quotaholder.host/quotaholder/v
229

  
230
    # cyclades:     CYCLADES_QUOTAHOLDER_TOKEN = <the same random string>
231
    # cyclades:     CYCLADES_QUOTAHOLDER_URL   = https://quotaholder.host/quotaholder/v
232
    # cyclades:     CYCLADES_USE_QUOTAHOLDER   = True
233

  
234

  
235
    # pithos:       PITHOS_QUOTAHOLDER_TOKEN   = <the same random string>
236
    # pithos:       PITHOS_QUOTAHOLDER_URL     = https://quotaholder.host/quotaholder/v
237
    # pithos:       PITHOS_USE_QUOTAHOLDER     = True
238
    # All services must match the quotaholder token and url configured for quotaholder.
239

  
240
Specifically:
241

  
242
On the Astakos host, edit ``/etc/synnefo/20-snf-astakos-app-settings.conf``:
243

  
244
::
245

  
246
    QUOTAHOLDER_URL = 'https://accounts.example.synnefo.org/quotaholder/v'
247
    QUOTAHOLDER_TOKEN = 'aExampleTokenJbFm12w'
248
    ASTAKOS_QUOTAHOLDER_TOKEN = 'aExampleTokenJbFm12w'
249
    ASTAKOS_QUOTAHOLDER_URL = 'https://accounts.example.synnefo.org/quotaholder/v'
250

  
251
On the Cyclades host, edit ``/etc/synnefo/20-snf-cyclades-app-quotas.conf``:
252

  
253
::
254

  
255
    CYCLADES_USE_QUOTAHOLDER = True
256
    CYCLADES_QUOTAHOLDER_URL = 'https://accounts.example.synnefo.org/quotaholder/v'
257
    CYCLADES_QUOTAHOLDER_TOKEN = 'aExampleTokenJbFm12w'
258

  
259
    # Set to False if astakos & cyclades are on the same host
260
    #CYCLADES_PROXY_USER_SERVICES = True
261

  
262
.. note::
263

  
264
    If Cylcades and Astakos are installed on the same server,
265
    set ``CYCLADES_PROXY_USER_SERVICES = False``
266

  
267

  
268
On the Pithos host, edit ``/etc/synnefo/20-snf-pithos-app-settings.conf``:
269

  
270
::
271

  
272
    PITHOS_QUOTAHOLDER_URL = 'https://accounts.example.synnefo.org/quotaholder/v'
273
    PITHOS_QUOTAHOLDER_TOKEN = 'aExampleTokenJbFm12w'
274
    PITHOS_USE_QUOTAHOLDER = False # will set to True after migration
275

  
276
.. note::
277

  
278
    During the migration it must be set, ``PITHOS_USE_QUOTAHOLDER = False``.
279
    Set to ``True`` once the migration is over.
280

  
281
3.4 Setup astakos
282
-----------------
283

  
284
- **Remove** this redirection from astakos front-end web server ::
285

  
286
        RewriteRule ^/login(.*) /im/login/redirect$1 [PT,NE]
287

  
288
    (see `<http://docs.dev.grnet.gr/synnefo/latest/quick-install-admin-guide.html#apache2-setup>`_)
289

  
290
- Enable users to change their contact email. Edit
291
``/etc/synnefo/20-snf-astakos-app-settings.conf`` ::
292

  
293
    ASTAKOS_EMAILCHANGE_ENABLED = True
294

  
295
- Rename the following (Astakos-specific) setting::
296

  
297
    ASTAKOS_DEFAULT_FROM_EMAIL
298
  
299
  to this (Django-specific) name::
300

  
301
    SERVER_EMAIL
302

  
303
- Instead of using the following (Astakos-specific) setting::
304

  
305
    ASTAKOS_DEFAULT_ADMIN_EMAIL
306

  
307
  include one or more entries in this (Django-specific) setting::
308

  
309
    ADMINS = (
310
        ('Joe Doe', 'doe@example.net'),
311
        ('Mary Jean', 'mary@example.net'),
312
    ) 
313

  
314
.. note::
315

  
316
    The ``SERVER_EMAIL`` and ``ADMINS`` settings are Django-specific.
317
    As such they will be the shared for any two (or more) services that happen
318
    to be collocated within the same application server (e.g. astakos &
319
    cyclades within the same gunicorn)
320

  
321
3.5 Setup Cyclades
322
------------------
323

  
324
- Run on the Astakos host ::
325

  
326
    # snf-manage service-list
327

  
328
- Set the Cyclades service token in
329
  ``/etc/synnefo/20-snf-cyclades-app-api.conf`` ::
330

  
331
    CYCLADES_ASTAKOS_SERVICE_TOKEN = 'asfasdf_CycladesServiceToken_iknl'
332

  
333
- Since version 0.13, Synnefo uses **VMAPI** in order to prevent sensitive data
334
  needed by 'snf-image' to be stored in Ganeti configuration (e.g. VM
335
  password). This is achieved by storing all sensitive information to a CACHE
336
  backend and exporting it via VMAPI. The cache entries are invalidated after
337
  the first request. Synnefo uses **memcached** as a django cache backend.
338
  To install, run on the Cyclades host::
339

  
340
        apt-get install memcached
341
        apt-get install python-memcache
342

  
343
  You will also need to configure Cyclades to use the memcached cache backend.
344
  Namely, you need to set IP address and port of the memcached daemon, and the
345
  default timeout (seconds tha value is stored in the cache). Edit
346
  ``/etc/synnefo/20-snf-cyclades-app-vmapi.conf`` ::
347

  
348
    VMAPI_CACHE_BACKEND = "memcached://127.0.0.1:11211/?timeout=3600"
349

  
350

  
351
  Finally, set the BASE_URL for the VMAPI, which is actually the base URL of
352
  Cyclades, again in ``/etc/synnefo/20-snf-cyclades-app-vmapi.conf``. Make sure
353
  the domain is exaclty the same, so that no re-directs happen ::
354

  
355
    VMAPI_BASE_URL = "https://cyclades.example.synnefo.org"
356

  
357
  .. note::
358

  
359
    - These settings are needed in all Cyclades workers.
360

  
361
    - VMAPI_CACHE_BACKEND just overrides django's CACHE_BACKEND setting
362

  
363
    - memcached must be reachable from all Cyclades workers.
364

  
365
    - For more information about configuring django to use memcached:
366
      https://docs.djangoproject.com/en/1.2/topics/cache
367

  
368
3.6 Setup Pithos
369
----------------
370

  
371
- Pithos forwards user catalog services to Astakos so that web clients may
372
  access them for uuid-displayname translations. Edit on the Pithos host
373
  ``/etc/synnefo/20-snf-pithos-app-settings.conf`` ::
374

  
375
    PITHOS_USER_CATALOG_URL    = https://accounts.example.synnefo.org/user_catalogs/
376
    PITHOS_USER_FEEDBACK_URL   = https://accounts.example.synnefo.org/feedback/
377
    PITHOS_USER_LOGIN_URL      = https://accounts.example.synnefo.org/login/
378

  
379
    # Set to False if astakos & pithos are on the same host
380
    #PITHOS_PROXY_USER_SERVICES = True
381

  
382
.. note::
383

  
384
    If Pithos and Astakos are installed on the same server,
385
    set ``PITHOS_PROXY_USER_SERVICES = False``
386

  
387
4. Start astakos and quota services
388
===================================
389
.. warning::
390

  
391
    To ensure consistency, prevent public access to astakos during migrations.
392
    This can be done via firewall or webserver access control.
393

  
394
Start (or restart, if running) the webserver and gunicorn on the Astakos host.
395
E.g.::
396

  
397
    # service apache2 start
398
    # service gunicorn start
399

  
400
.. _astakos-load-resources:
401

  
402
5. Load resource definitions into Astakos
403
=========================================
404

  
405
First, set the corresponding values on the following dict in
406
``/etc/synnefo/20-snf-astakos-app-settings.conf`` ::
407

  
408
    # Set the cloud service properties
409
    ASTAKOS_SERVICES = {
410
        'cyclades': {
411
    #        # Specifying the key 'url' will overwrite it.
412
    #        # Use this to (re)set service URL.
413
    #        'url': 'https://cyclades.example.synnefo.org/ui/',
414
    #        # order services in listings, cloudbar, etc.
415
    #        'order' : 1
416
            'resources': [{
417
                'name': 'disk',
418
                'group': 'compute',
419
                'uplimit': 30*1024*1024*1024,
420
                'unit': 'bytes',
421
                'desc': 'Virtual machine disk size'
422
                }, {
423
                'name': 'cpu',
424
                'group': 'compute',
425
                'uplimit': 6,
426
                'desc': 'Number of virtual machine processors'
427
                }, {
428
                'name': 'ram',
429
                'group': 'compute',
430
                'uplimit': 6*1024*1024*1024,
431
                'unit': 'bytes',
432
                'desc': 'Virtual machines'
433
                }, {
434
                'name': 'vm',
435
                'group': 'compute',
436
                'uplimit': 2,
437
                'desc': 'Number of virtual machines'
438
                }, {
439
                'name': 'network.private',
440
                'group': 'network',
441
                'uplimit': 1,
442
                'desc': 'Private networks'
443
                }
444
            ]
445
        },
446
        'pithos+': {
447
    #        # Use this to (re)set service URL.
448
    #        'url': 'https://pithos.example.synnefo.org/ui/',
449
    #        # order services in listings, cloudbar, etc.
450
    #        'order' : 2
451
            'resources':[{
452
                'name': 'diskspace',
453
                'group': 'storage',
454
                'uplimit': 5*1024*1024*1024,
455
                'unit': 'bytes',
456
                'desc': 'Pithos account diskspace'
457
                }]
458
        }
459
    }
460

  
461
.. note::
462

  
463
    The name of the Pithos service is ``pithos+``.
464
    If you have named your pithos service ``pithos``, without ``+``,
465
    then you must rename it::
466

  
467
        $ snf-manage service-list | grep pithos # find service id
468
        $ snf-manage service-update --name='pithos+' <service id> 
469

  
470
Then, configure and load the available resources per service
471
and associated default limits into Astakos. On the Astakos host run ::
472

  
473
     # snf-manage astakos-init --load-service-resources
474

  
475

  
476
.. note::
477

  
478
    Before v0.13, only `cyclades.vm`, `cyclades.network.private`,
479
    and `pithos+.diskspace` existed (not with these names,
480
    there were per-service settings).
481
    However, limits to the new resources must also be set.
482

  
483
    If the intetion is to keep a resource unlimited, (counting on that VM
484
    creation will be limited by other resources' limit) it is best to calculate
485
    a value that is too large to be reached because of other limits (and
486
    available flavours), but not much larger than needed because this might
487
    confuse users who do not readily understand that multiple limits apply and
488
    flavors are limited.
489

  
490

  
491
6. Migrate Services user names to uuids
492
=======================================
493

  
494

  
495
6.1 Double-check cyclades before user case/uuid migration
496
---------------------------------------------------------
497

  
498
::
499

  
500
    cyclades.host$ snf-manage cyclades-astakos-migrate-013 --validate
501

  
502
Duplicate user found?
503

  
504
- either *merge* (merge will merge all resources to one user)::
505

  
506
    cyclades.host$ snf-manage cyclades-astakos-migrate-013 --merge-user=kpap@grnet.gr
507

  
508
- or *delete* ::
509

  
510
    cyclades.host$ snf-manage cyclades-astakos-migrate-013 --delete-user=KPap@grnet.gr
511
    # (only KPap will be deleted not kpap)
512

  
513
6.2 Double-check pithos before user case/uuid migration
514
---------------------------------------------------------
515

  
516
::
517

  
518
    pithos.host$ snf-manage pithos-manage-accounts --list-duplicate
519

  
520
Duplicate user found?
521

  
522
If you want to migrate files first:
523

  
524
- *merge* (merge will merge all resources to one user)::
525

  
526
    pithos.host$ snf-manage pithos-manage-accounts --merge-accounts --src-account=SPapagian@grnet.gr --dest-account=spapagian@grnet.gr
527
    # (SPapagian@grnet.gr's contents will be merged into spapagian@grnet.gr, but SPapagian@grnet.gr account will still exist)
528

  
529
- and then *delete* ::
530

  
531
    pithos.host$ snf-manage pithos-manage-accounts --delete-account=SPapagian@grnet.gr
532
    # (only SPapagian@grnet.gr will be deleted not spapagian@grnet.gr)
533

  
534
If you do *NOT* want to migrate files just run the second step and delete
535
the duplicate account.
536

  
537
6.3 Migrate Cyclades users (email case/uuid)
538
--------------------------------------------
539

  
540
::
541

  
542
    cyclades.host$ snf-manage cyclades-astakos-migrate-013 --migrate-users
543

  
544
- if invalid usernames are found, verify that they do not exist in astakos::
545

  
546
    astakos.host$ snf-manage user-list
547

  
548
- if no user exists::
549

  
550
    cyclades.host$ snf-manage cyclades-astakos-migrate-013 --delete-user=<userid>
551

  
552
Finally, if you have set manually quotas for specific users inside
553
``/etc/synnefo/20-snf-cyclades-app-api.conf`` (in ``VMS_USER_QUOTA``,
554
``NETWORKS_USER_QUOTA`` make sure to update them so that:
555

  
556
1. There are no double entries wrt case sensitivity
557
2. Replace all user email addresses with the corresponding UUIDs
558

  
559
To find the UUIDs for step 2 run on the Astakos host ::
560

  
561
     # snf-manage user-list
562

  
563
6.4 Migrate Pithos user names
564
-----------------------------
565

  
566
Check if alembic has not been initialized ::
567

  
568
    pithos.host$ pithos-migrate current
569

  
570
- If alembic current is None (e.g. okeanos.io) ::
571

  
572
    pithos.host$ pithos-migrate stamp 3dd56e750a3
573

  
574
Then, migrate pithos account name to uuid::
575

  
576
    pithos.host$ pithos-migrate upgrade head
577

  
578
Finally, set this setting to ``True``::
579

  
580
    PITHOS_USE_QUOTAHOLDER = True
581

  
582

  
583
7. Migrate old quota limits
584
===========================
585

  
586
7.1 Migrate Pithos quota limits to Astakos
587
------------------------------------------
588

  
589
Migrate from pithos native to astakos/quotaholder.
590
This requires a file to be transfered from Cyclades to Astakos::
591

  
592
    pithos.host$ snf-manage pithos-export-quota --location=pithos-quota.txt
593
    pithos.host$ scp pithos-quota.txt astakos.host:
594
    astakos.host$ snf-manage user-set-initial-quota pithos-quota.txt
595

  
596
.. _export-quota-note:
597

  
598
.. note::
599

  
600
    `pithos-export-quota` will only export quotas that are not equal to the
601
    defaults in Pithos. Therefore, it is possible to both change or maintain
602
    the default quotas across the migration. To maintain quotas the new default
603
    pithos+.diskpace limit in Astakos must be equal to the (old) default quota
604
    limit in Pithos. Change either one of them make them equal.
605

  
606
    see :ref:`astakos-load-resources` on how to set the (new) default quotas in Astakos.
607

  
608
7.2 Migrate Cyclades quota limits to Astakos
609
--------------------------------------------
610

  
611
::
612

  
613
    cyclades.host$ snf-manage cyclades-export-quota --location=cyclades-quota.txt
614
    cyclades.host$ scp cyclades-quota.txt astakos.host:
615
    astakos.host$ snf-manage user-set-initial-quota cyclades-quota.txt
616

  
617
`cyclades-export-quota` will only export quotas that are not equal to the defaults.
618
See :ref:`note above <export-quota-note>`.
619

  
620
8. Enforce the new quota limits migrated to Astakos
621
===================================================
622
The following should report all users not having quota limits set
623
because the effective quota database has not been initialized yet. ::
624

  
625
    astakos.host$ snf-manage astakos-quota --verify
626

  
627
Initialize the effective quota database::
628

  
629
    astakos.host$ snf-manage astakos-quota --sync
630

  
631
This procedure may be used to verify and re-synchronize the effective quota
632
database with the quota limits that are derived from policies in Astakos
633
(initial quotas, project memberships, etc.)
634

  
635
9. Initialize resource usage
636
============================
637

  
638
The effective quota database (quotaholder) has just been initialized and knows
639
nothing of the current resource usage. Therefore, each service must send it in.
640

  
641
9.1 Initialize Pithos resource usage
642
------------------------------------
643

  
644
::
645

  
646
    pithos.host$ snf-manage pithos-reset-usage
647

  
648
9.2 Initialize Cyclades resource usage
649
--------------------------------------
650

  
651
::
652

  
653
    cyclades.host$ snf-manage cyclades-reset-usage
654

  
655
10. Install periodic project maintainance checks
656
================================================
657
In order to detect and effect project expiration,
658
a management command has to be run periodically
659
(depending on the required granularity, e.g. once a day/hour)::
660

  
661
    astakos.host$ snf-manage project-control --terminate-expired
662

  
663
A list of expired projects can be extracted with::
664

  
665
    astakos.host$ snf-manage project-control --list-expired
666

  
667

  
668
11. Restart all services
669
========================
670

  
671
Start (or restart, if running) all Synnefo services on all hosts.
672

  
673
::
674

  
675
    # service gunicorn restart
676
    # service snf-dispatcher restart
677
    # etc.
b/docs/upgrade/cyclades-upgrade.rst
1
Upgrade
2
=======
3

  
4
This file documents the upgrade to newer versions of the Synnefo software.
5
For more information, please see deployment guide.
6

  
7

  
8
v0.7.4 -> v0.8
9
--------------
10

  
11
Synnefo is now distributed packaged in python/debian packages. You should
12
consider the following when migrating from projects previously deployed from
13
source:
14
    
15
    * To keep consistency with future changes, change all ``python manage.py``
16
      calls to refer to the shipped ``snf-manage`` console script.
17
      This includes automation scripts, service scripts etc.
18

  
19
      Same applies for calls to ``dispatcher.py``, ``snf-tools/admin.py``,
20
      ``snf-tools/burnin.py`` and ``snf-tools/cloud.py`` which are replaced
21
      by ``snf-dispatcher``, ``snf-admin``, ``snf-burnin`` and ``snf-cloud``,
22
      respectively.
23

  
24
    * Copy custom settings files located in project's ``settings.d`` directory
25
      to ``/etc/synnefo/`` directory.
26

  
27
    * Migrate location from which :ref:`static files <static-files>` are served from.
28

  
29
.. warning::
30
   
31
   Synnefo settings have been refactored as part of the
32
   :ref:`snf-common <snf-common>` component. File locations may be inaccurate
33
   and the list of changed settings incomplete.  Please consult the actual
34
   files installed under ``/etc/synnefo`` as the authoritative source for new
35
   settings.
36

  
37
NEW APPS
38
    * The ``synnefo.ui.userdata`` application has been added in
39
      :file:`settings.d/00-apps.conf`. Application urls appended in
40
      :file:`ui/urls.py`.
41
      If no custom ``ROOT_URLCONF`` module is used, no url change is needed.
42
    * The new app has migrations defined.
43
      Use ``snf-manage migrate`` to migrate *all* apps.
44

  
45
NEW/UPDATED SETTINGS
46
    * ``BYPASS_AUTHENTICATION_TOKEN_SECRET`` replaces ``BYPASS_AUTHENTICATION_TOKEN``
47
      in :file:`settings/common/aai.py`.
48
    * New config file :file:`31-userdata.conf`, containing userdata app settings
49
    * ``USERDATA_SSH_KEY_LENGTH`` in :file:`31-userdata.conf`:
50
      Key length in bits for generated ssh keys
51
    * ``USERDATA_SSH_KEY_EXPONENT`` in :file:`31-userdata.conf`:
52
      Generated SSH key exponent
53
    * ``USERDATA_MAX_SSH_KEYS_PER_USER`` in :file:`31-userdata.conf`:
54
      Maximum number of ssh keys a user is allowed to have
55
    * ``UI_SUPPORT_SSH_OS_LIST``, in :file:`30-ui.conf`:
56
      A list of os names that support ssh public key assignment
57
    * ``UI_OS_DEFAULT_USER_MAP``, in :file:`30-ui.conf`:
58
      OS/username map to identify default user name for a specific os
59
    * ``VM_CREATE_NAME_TPL``, in :file:`30-ui.conf`:
60
      Template to be used for suggesting the user a default name for newly
61
      created VMs. String {0} gets replaced by the value of metadata key "os"
62
      for the Image.
63
    * ``UI_FLAVORS_DISK_TEMPLATES_INFO`` added in :file:`30-ui.conf`:
64
      Name/description metadata for the available flavor disk templates
65
    * ``VM_CREATE_SUGGESTED_FLAVORS`` in :file:`30-ui.conf`:
66
      Updated flavor data to include disk_template value.
67
    * ``GANETI_DISK_TEMPLATES`` and ``DEFAULT_GANETI_DISK_TEMPLATE`` in :file:`20-api.conf`:
68
      The list of disk templates supported by the Ganeti backend.
69
      The default template to be used when migrating Flavors with no value for
70
      disk_template (i.e., 'drbd').
71
    * ``UI_COMPUTE_URL``, ``UI_GLANCE_URL`` in ui app:
72
      Configurable API endpoints
73
    * ``UI_ENABLE_GLANCE`` in ui app:
74
      Whether or not UI should retrieve images from the Glance API endpoint
75
      set in ``UI_GLANCE_API_URL``. If setting is set to False, ui will request
76
      images using calls to the Cyclades API.
77
UI
78
    * ``synnefo.ui.userdata`` application has been added in ``INSTALLED_APPS``.
79
      Database migration is needed for the creation of application db tables.
80

  
81
API
82
    * A new ``disk_template`` attribute has been added to Flavors.
83
      ``GANETI_DISK_TEMPLATES`` and ``DEFAULT_GANETI_DISK_TEMPLATE`` have been
84
      added in :file:`20-api.conf` to control its value. A database migration is
85
      needed.
86

  
87
PLANKTON
88
    Plankton is a new image service that has been added as a separate app. The
89
    images are stored in backend of Pithos and thus it must be configured to
90
    access the DB and directory that Pithos uses to store its data. These
91
    settings are defined in the new plankton.py file in settings.
92
    
93
    Plankton stores and looks for images in the Pithos container named
94
    ``PITHOS_IMAGE_CONTAINER``.
95
    
96
    There is a Pithos account that is reserved for system images. This account
97
    is defined in ``SYSTEM_IMAGES_OWNER``.
98

  
99
ADMIN TOOLS
100
    * A new ``--disk-template`` flag has been added to snf-admin to choose a
101
      disk template when creating flavors. Similarly, ``disk_template`` support
102
      in flavors has beed added to the admin web interface.
103

  
104

  
105
v0.7.3 -> v0.7.4
106
----------------
107

  
108
OKEANOS INTRO
109
    * News section added. News content can be remotely retrieved using
110
      OKEANOS_NEWS_CONTENT_URL settings option.
111

  
112

  
113
v0.6.2 -> v0.7
114
--------------
115

  
116
HTML TEMPLATES
117
    * Included a generic service unavailable template based on
118
      generic_info_tpl.html, as ui/static/service_unavailable.html.
119

  
120
NEW DEPENDENCIES
121
    * python-unittest2, python-paramiko needed by the new integration
122
      test suite snf-tools/burnin. Paramiko shipped with Squeeze uses
123
      broken RandomPool, produces warnings, see #576697.
124
    * snf-image must be installed on all Ganeti hosts, see README.deploy,
125
      Step 13.
126

  
127
COMPONENTS
128
    * snf-admin has been updated with new functionality, be sure to upgrade any
129
      locally installed versions.
130
    * snf-image replaces snf-ganeti-instance-image as the Ganeti OS provider
131
      used by Synnefo, and can live alongside snf-ganeti-instance-image.
132
      Once snf-image has been deployed on all Ganeti nodes, be sure to modify
133
      the value of settings.d/10-backend.conf:GANETI_CREATEINSTANCE_KWARGS
134
      to use 'snf-image+default'.
135

  
136
UI STATIC FILES
137
    * A major reorganization of static files has been commited. All ui and
138
      invitations static files has been moved in their own separate directory
139
      (ui/static/snf, ui/static/invitations).
140
    * UI static files base url is now configurable via UI_MEDIA_URL.
141
    * A symlink named after the current version of the ui application has been
142
      committed in ui/static directory. The symlink will get updated after each
143
      version upgrade to allow us provide unique urls of static files between
144
      after each upgrade, see #1460.
145

  
146
NEW/UPDATED SETTINGS
147
    * EMAIL_SUBJECT_PREFIX:
148
      Prepended to automated emails, set to someting that uniquely identifies
149
      the deployment.
150
    * MAX_PERSONALITY and MAX_PERSONALITY_SIZE in 20-api.conf:
151
      Maximum number of files to be injected in newly created servers,
152
      maximum total size of encoded file contents.
153
    * Renamed SUGGESTED_FLAVORS to VM_CREATE_SUGGESTED_FLAVORS in 30-ui.conf
154
    * VM_CREATE_SUGGESTED_ROLES in 30-ui.conf:
155
      A list of suggested vm roles to display to user on create wizard.
156
    * UI_DELAY_ON_BLUR in 30-ui.conf:
157
      Whether to increase the intervals of recurrent requests (networks/vms 
158
      update) if window loses its focus.
159
    * UI_BLUR_DELAY in 30-ui.conf:
160
      The value of update intervals if window loses its focus.
161
      Considered only if `UI_DELAY_ON_BLUR` is set to True.
162
    * UI_UPDATE_HIDDEN_VIEWS in 30-ui.conf:
163
      Whether not visible vm views will update their content if vm changes.
164
    * UI_SKIP_TIMEOUTS in 30-ui.conf:
165
      After how many timeouts of reccurent ajax requests to display the timeout
166
      error overlay.
167
    * UI_HANDLE_WINDOW_EXCEPTIONS in 30-ui.conf:
168
      Whether UI should display error overlay for all Javascript exceptions.
169
    * UI_MEDIA_URL in 30-ui.conf:
170
      Base url for ui static files, 
171
      defaults to MEDIA_URL + 'snf-<latest_ui_version>/'.
172
    * MEDIA_URL changed in 00-site.conf:
173
      Changed to '/static/' since it is now used in ui app.
174
    * TEMPLATE_CONTEXT_PROCESSORS changed in 00-apps.conf:
175
      added 'django.core.context_processors.media' to allow access of MEDIA_URL
176
      in template files.
177
    * GANETI_CREATEINSTANCE_KWARGS in 10-backend.conf:
178
      Must be updated to use snf-image as the Ganeti OS provider.
179
      
180
    
181
DB MIGRATION
182
    * Uniqueness constraints have been added to the metadata models.
183
      A database migration is needed.
184

  
185
LOGGING
186
    * A new logging mechanism has been implemeted. Please see 00-logging.conf
187
      under settings.d/ and read the relevant section in README.admin for more
188
      info.
189

  
190

  
191
v0.6.1 -> v0.6.2
192
----------------
193

  
194
ADMIN INTERFACE
195
    * The models were changed so that Flavors and SynnefoUsers now have
196
      a deleted state. The admin tools were updated so that models are
197
      now marked as deleted instead of actually being deleted from the DB.
198
      A database migration is needed.
199

  
200
COMPONENTS
201
    * Only the Django webapp is affected, must restart the logic dispatcher
202
      due to DB migration taking place.
203

  
204

  
205
v0.5.5 -> v0.6
206
--------------
207

  
208
ADMIN INTERFACE
209
    * A new Web-based admin interface is available under /admin.
210
      It is accessible by users of type 'ADMIN' in the DB, with
211
      their authentication token. "snf-admin user modify" may be used
212
      to change the type of a specific user.
213

  
214
RECONCILIATION
215
    * Implemented new reconciliation management command, please see
216
      ./manage.py reconcile --help and README.admin for more info.
217
      Recommended to run ./manage.py reconcile --detect-all periodically,
218
      via cron.
219

  
220
GANETI-INSTANCE-IMAGE
221
    * A new version of ganeti-instance-image is required (v0.5.1-1-snf1).
222
      The new version is available for download as a debian package:
223
      https://code.grnet.gr/projects/gnt-instance-image/files
224

  
225
COMPONENTS
226
    * snf-cyclades-gtools must be rebuilt, see snf-cyclades-gtools/debian/
227
      for Debian packaging.
228

  
229
NEW SETTINGS
230
    * 30-ui.conf:SUGGESTED_FLAVORS
231
    * 30-ui.conf:VM_IMAGE_COMMON_METADATA
232

  
233

  
234
v0.5.4 -> v0.5.5
235
----------------
236

  
237
LOGGING
238
    * Changed the default logging settings for the dispatcher to also log
239
      to /var/log/synnefo/dispatcher.log, redirecting stderr and stdout there
240

  
241
QUEUES
242
    * Changed default routing key naming for queues. Queues must be redeclared.
243
      The suggested upgrade path is to delete ALL (even obsolete) existing
244
      queues and restart the dispatcher. To do so, download amqp-utils from
245

  
246
          https://github.com/dougbarth/amqp-utils
247

  
248
      and run the amqp-deleteq tool for each declared queue. Alternatively,
249
      amqp-utils can be installed with: sudo gem install amqp-utils.
250

  
251
UI
252
    * Feedback form now uses django native send_mail for sending emails.
253
      Proper django settings should be set for feedback mails to work
254
      (https://docs.djangoproject.com/en/dev/topics/email/)
255

  
256
COMPONENTS
257
    * snf-cyclades-gtools must be rebuilt, see snf-cyclades-gtools/debian/
258
      for Debian packaging.
259

  
260

  
261
v0.5.3 -> v0.5.4
262
----------------
263

  
264
REPOSITORY
265
    * Split Synnefo Ganeti tools to snf-cyclades-gtools, with Debian packaging
266

  
267
REMOVED APPS
268
    * The ganeti/ app has been removed from the Django project completely.
269
      Any explicit references to it in Django settings must be removed.
270

  
271
DJANGO SETTINGS
272
    * snf-cyclades-gtools is configured independently from Django,
273
      need to add proper /etc/synnefo/settings.conf
274
    * Removed 15-queues.conf: fix_amqp_settings (no need to call it anywhere)
275
    * Removed settings.d/98-ganeti-* due to split of snf-cyclades-gtools
276
    * ~okeanos intro: OKEANOS_VIDEO_URL: Changed from string to dict
277
    * ~okeanos intro: OKEANOS_VIDEO_POSTER_IMAGE_URL: New setting
278
    * ~okeanos intro: OKEANOS_VIDEO_FLOWPLAYER_URL: New setting
279

  
280
DB MIGRATIONS
281
    * 0018_auto__add_field_virtualmachine_buildpercentage
282

  
283
PACKAGING
284
    * Split Synnefo Ganeti tools to separate snf-cyclades-gtools Debian package
285

  
286
NEW DEPENDENCIES
287
    * python-prctl: Needed by the snf-progress-monitor,
288
      specified as a dependency of the snf-cyclades-gtools Debian package.
289

  
290
EXTERNAL COMPONENTS
291
    * Ganeti Instance Image must be upgraded to support progress monitoring,
292
      please see README.deploy.
293

  
294

  
295
v0.5.2 -> v0.5.3
296
----------------
297

  
298
NEW SETTINGS
299
    * 30-ui.conf:LOGOUT_URL
300
    * 00-admins.conf:DEFAULT_FROM_EMAIL
301
    * 90-okeanos.conf.sample:LOGOUT_URL
302

  
303
REMOVED SETTINGS
304
    * 00-admins.conf:SYSTEM_EMAIL_ADDR
305
    * 90-okeanos.conf.sample:APP_INSTALL_URL
306

  
307

  
308
v0.5.1 -> v0.5.2
309
----------------
310

  
311
NEW SETTINGS
312
    * 10-backend.py:GANETI_CREATEINSTANCE_KWARGS
313

  
314
REMOVED SETTINGS
315
    * 10-backend.conf:GANETI_OS_PROVIDER
316
    * 20-api.conf:GANETI_DISK_TEMPLATE
317

  
318
BACKEND CHANGES
319
    * Need to patch Ganeti, file:
320
      lib/python2.6/site-packages/ganeti/rapi/rlib2.py
321
      to honor the wait_for_sync flag, see Synnefo #835.
322
      Patch provided under contrib/patches/ganeti-rlib2.py-v0.5.2
323

  
b/docs/upgrade/upgrade-0.13.rst
1
Upgrade to Synnefo v0.13
2
^^^^^^^^^^^^^^^^^^^^^^^^
3

  
4
The bulk of the upgrade to v0.13 is about user and quota migrations.
5
In summary, the migration process has 3 steps:
6

  
7
1. Run some commands and scripts to diagnose and extract some migration data
8
   while the OLD code is running, and BEFORE any changes are made.
9

  
10
2. Bring down services, upgrade packages, configure services, and perform
11
   django database migrations. These migrations do not need any interaction
12
   between services.
13

  
14
3. Initialize the Astakos quota system and bring the Astakos service up, since
15
   it will be needed during a second-phase of UUID and quota migrations, that
16
   also uses data extracted from step 1.
17

  
18

  
19
.. warning::
20
 
21
    It is strongly suggested that you keep separate database backups
22
    for each service after the completion of each of step.
23

  
24
1. Bring web services down, backup databases
25
============================================
26

  
27
1. All web services must be brought down so that the database maintains a
28
   predictable and consistent state during the migration process::
29

  
30
    # service gunicorn stop
31
    # service snf-dispatcher stop
32
    # etc.
33

  
34
2. Backup databases for recovery to a pre-migration state.
35

  
36
3. Keep the database servers running during the migration process
37

  
38

  
39
2. Prepare astakos user migration to case insensitive emails
40
============================================================
41

  
42
It is possible that two or more users have been registered with emails that
43
differ only in the case of its letters.  There can only be one of those
44
accounts after the migration, so the rest must be deleted.
45

  
46
Note that even if the users are deleted in Astakos, there still are duplicate
47
entries in Cyclades and Pithos.  For each service we need to reduce those
48
multiple accounts into one, either merging them together, or deleting and
49
discarding data from all but one.
50

  
51
.. _find_duplicate_emails:
52

  
53
2.1 Find duplicate email entries in Astakos
54
-------------------------------------------
55
(script: ``find_astakos_users_with_conflicting_emails.py``)::
56

  
57
    astakos-host$ cat << EOF > find_astakos_users_with_conflicting_emails.py
58
    #!/usr/bin/env python
59
    import os
60
    import sys
61

  
62
    os.environ['DJANGO_SETTINGS_MODULE'] = 'synnefo.settings'
63

  
64
    import astakos
65
    from astakos.im.models import AstakosUser as A
66

  
67

  
68
    def user_filter(user):
69
        return A.objects.filter(email__iexact=user.email).count() > 1
70

  
71
    all_users = list(A.objects.all())
72
    userlist = [(str(u.pk) + ': ' + str(u.email) + ' (' + str(u.is_active) + ', ' +
73
                 str(u.date_joined) + ')') for u in filter(user_filter, all_users)]
74

  
75
    sys.stderr.write("id email (is_active, creation date)\n")
76
    print "\n".join(userlist)
77
    EOF
78

  
79
    astakos-host$ python ./find_astakos_users_with_conflicting_emails.py
80

  
81
.. _remove_astakos_duplicate:
82

  
83
2.1 Remove duplicate users in Astakos by their id
84
-------------------------------------------------
85
(script: ``delete_astakos_users.py``)::
86

  
87
    astakos-host$ cat << EOF > delete_astakos_users.py
88
    #!/usr/bin/env python
89

  
90
    import os
91
    import sys
92
    from time import sleep
93

  
94
    os.environ['DJANGO_SETTINGS_MODULE'] = 'synnefo.settings'
95

  
96
    import astakos
97
    from astakos.im.models import AstakosUser as A
98

  
99

  
100
    def user_filter(user):
101
        return A.objects.filter(email__iexact=user.email).count() > 1
102

  
103
    argv = sys.argv
104
    argc = len(sys.argv)
105

  
106
    if argc < 2:
107
        print "Usage: ./delete_astakos_users.py <id>..."
108
        raise SystemExit()
109

  
110
    id_list = [int(x) for x in argv[1:]]
111

  
112
    print ""
113
    print "This will permanently delete the following users:\n"
114
    print "id  email (is_active, creation date)"
115
    print "--  --------------------------------"
116

  
117
    users = A.objects.filter(id__in=id_list)
118
    for user in users:
119
        print "%s: %s (%s, %s)" % (user.id, user.email, user.is_active,
120
                                   user.date_joined)
121

  
122
    print "\nExecute? (yes/no): ",
123
    line = raw_input().rstrip()
124
    if line != 'yes':
125
        print "\nCancelled"
... This diff was truncated because it exceeds the maximum size that can be displayed.

Also available in: Unified diff