root / docs / upgrade / upgrade-0.13.rst @ d8e8581b
History | View | Annotate | Download (21.1 kB)
1 | 7e38e668 | Georgios D. Tsoukalas | Upgrade to Synnefo v0.13 |
---|---|---|---|
2 | 7e38e668 | Georgios D. Tsoukalas | ^^^^^^^^^^^^^^^^^^^^^^^^ |
3 | 7e38e668 | Georgios D. Tsoukalas | |
4 | 457fe4fc | Georgios D. Tsoukalas | The bulk of the upgrade to v0.13 is about user and quota migrations. |
5 | 7e38e668 | Georgios D. Tsoukalas | In summary, the migration process has 3 steps: |
6 | 7e38e668 | Georgios D. Tsoukalas | |
7 | 7e38e668 | Georgios D. Tsoukalas | 1. Run some commands and scripts to diagnose and extract some migration data |
8 | 7e38e668 | Georgios D. Tsoukalas | while the OLD code is running, and BEFORE any changes are made. |
9 | 7e38e668 | Georgios D. Tsoukalas | |
10 | 7e38e668 | Georgios D. Tsoukalas | 2. Bring down services, upgrade packages, configure services, and perform |
11 | eeb3f40b | Georgios D. Tsoukalas | django database migrations. These migrations do not need any interaction |
12 | 7e38e668 | Georgios D. Tsoukalas | between services. |
13 | 7e38e668 | Georgios D. Tsoukalas | |
14 | 7e38e668 | Georgios D. Tsoukalas | 3. Initialize the Astakos quota system and bring the Astakos service up, since |
15 | 7e38e668 | Georgios D. Tsoukalas | it will be needed during a second-phase of UUID and quota migrations, that |
16 | 7e38e668 | Georgios D. Tsoukalas | also uses data extracted from step 1. |
17 | 7e38e668 | Georgios D. Tsoukalas | |
18 | e38ad44f | Georgios D. Tsoukalas | |
19 | e6df0da5 | Kostas Papadimitriou | .. warning:: |
20 | e38ad44f | Georgios D. Tsoukalas | |
21 | e38ad44f | Georgios D. Tsoukalas | It is strongly suggested that you keep separate database backups |
22 | e38ad44f | Georgios D. Tsoukalas | for each service after the completion of each of step. |
23 | e6df0da5 | Kostas Papadimitriou | |
24 | e38ad44f | Georgios D. Tsoukalas | 1. Bring web services down, backup databases |
25 | e38ad44f | Georgios D. Tsoukalas | ============================================ |
26 | e6df0da5 | Kostas Papadimitriou | |
27 | e38ad44f | Georgios D. Tsoukalas | 1. All web services must be brought down so that the database maintains a |
28 | eeb3f40b | Georgios D. Tsoukalas | predictable and consistent state during the migration process:: |
29 | eeb3f40b | Georgios D. Tsoukalas | |
30 | eeb3f40b | Georgios D. Tsoukalas | # service gunicorn stop |
31 | eeb3f40b | Georgios D. Tsoukalas | # service snf-dispatcher stop |
32 | eeb3f40b | Georgios D. Tsoukalas | # etc. |
33 | 7e38e668 | Georgios D. Tsoukalas | |
34 | e38ad44f | Georgios D. Tsoukalas | 2. Backup databases for recovery to a pre-migration state. |
35 | 7e38e668 | Georgios D. Tsoukalas | |
36 | e38ad44f | Georgios D. Tsoukalas | 3. Keep the database servers running during the migration process |
37 | 7e38e668 | Georgios D. Tsoukalas | |
38 | 7e38e668 | Georgios D. Tsoukalas | |
39 | 7e38e668 | Georgios D. Tsoukalas | 2. Prepare astakos user migration to case insensitive emails |
40 | 7e38e668 | Georgios D. Tsoukalas | ============================================================ |
41 | 7e38e668 | Georgios D. Tsoukalas | |
42 | 7e38e668 | Georgios D. Tsoukalas | It is possible that two or more users have been registered with emails that |
43 | 7e38e668 | Georgios D. Tsoukalas | differ only in the case of its letters. There can only be one of those |
44 | 7e38e668 | Georgios D. Tsoukalas | accounts after the migration, so the rest must be deleted. |
45 | 7e38e668 | Georgios D. Tsoukalas | |
46 | 7e38e668 | Georgios D. Tsoukalas | Note that even if the users are deleted in Astakos, there still are duplicate |
47 | 7e38e668 | Georgios D. Tsoukalas | entries in Cyclades and Pithos. For each service we need to reduce those |
48 | 7e38e668 | Georgios D. Tsoukalas | multiple accounts into one, either merging them together, or deleting and |
49 | 7e38e668 | Georgios D. Tsoukalas | discarding data from all but one. |
50 | 7e38e668 | Georgios D. Tsoukalas | |
51 | e38ad44f | Georgios D. Tsoukalas | .. _find_duplicate_emails: |
52 | e38ad44f | Georgios D. Tsoukalas | |
53 | 457fe4fc | Georgios D. Tsoukalas | 2.1 Find duplicate email entries in Astakos |
54 | 457fe4fc | Georgios D. Tsoukalas | ------------------------------------------- |
55 | 7e38e668 | Georgios D. Tsoukalas | (script: ``find_astakos_users_with_conflicting_emails.py``):: |
56 | 7e38e668 | Georgios D. Tsoukalas | |
57 | e38ad44f | Georgios D. Tsoukalas | astakos-host$ cat << EOF > find_astakos_users_with_conflicting_emails.py |
58 | e38ad44f | Georgios D. Tsoukalas | #!/usr/bin/env python |
59 | e38ad44f | Georgios D. Tsoukalas | import os |
60 | e38ad44f | Georgios D. Tsoukalas | import sys |
61 | e38ad44f | Georgios D. Tsoukalas | |
62 | e38ad44f | Georgios D. Tsoukalas | os.environ['DJANGO_SETTINGS_MODULE'] = 'synnefo.settings' |
63 | e38ad44f | Georgios D. Tsoukalas | |
64 | e38ad44f | Georgios D. Tsoukalas | import astakos |
65 | e38ad44f | Georgios D. Tsoukalas | from astakos.im.models import AstakosUser as A |
66 | 7e38e668 | Georgios D. Tsoukalas | |
67 | 457fe4fc | Georgios D. Tsoukalas | |
68 | e38ad44f | Georgios D. Tsoukalas | def user_filter(user): |
69 | e38ad44f | Georgios D. Tsoukalas | return A.objects.filter(email__iexact=user.email).count() > 1 |
70 | e38ad44f | Georgios D. Tsoukalas | |
71 | e38ad44f | Georgios D. Tsoukalas | all_users = list(A.objects.all()) |
72 | e38ad44f | Georgios D. Tsoukalas | userlist = [(str(u.pk) + ': ' + str(u.email) + ' (' + str(u.is_active) + ', ' + |
73 | e38ad44f | Georgios D. Tsoukalas | str(u.date_joined) + ')') for u in filter(user_filter, all_users)] |
74 | e38ad44f | Georgios D. Tsoukalas | |
75 | e38ad44f | Georgios D. Tsoukalas | sys.stderr.write("id email (is_active, creation date)\n") |
76 | e38ad44f | Georgios D. Tsoukalas | print "\n".join(userlist) |
77 | e38ad44f | Georgios D. Tsoukalas | EOF |
78 | e38ad44f | Georgios D. Tsoukalas | |
79 | e38ad44f | Georgios D. Tsoukalas | astakos-host$ python ./find_astakos_users_with_conflicting_emails.py |
80 | e38ad44f | Georgios D. Tsoukalas | |
81 | 457fe4fc | Georgios D. Tsoukalas | .. _remove_astakos_duplicate: |
82 | 457fe4fc | Georgios D. Tsoukalas | |
83 | 457fe4fc | Georgios D. Tsoukalas | 2.1 Remove duplicate users in Astakos by their id |
84 | 457fe4fc | Georgios D. Tsoukalas | ------------------------------------------------- |
85 | 7e38e668 | Georgios D. Tsoukalas | (script: ``delete_astakos_users.py``):: |
86 | 7e38e668 | Georgios D. Tsoukalas | |
87 | e38ad44f | Georgios D. Tsoukalas | astakos-host$ cat << EOF > delete_astakos_users.py |
88 | e38ad44f | Georgios D. Tsoukalas | #!/usr/bin/env python |
89 | e38ad44f | Georgios D. Tsoukalas | |
90 | e38ad44f | Georgios D. Tsoukalas | import os |
91 | e38ad44f | Georgios D. Tsoukalas | import sys |
92 | e38ad44f | Georgios D. Tsoukalas | from time import sleep |
93 | e38ad44f | Georgios D. Tsoukalas | |
94 | e38ad44f | Georgios D. Tsoukalas | os.environ['DJANGO_SETTINGS_MODULE'] = 'synnefo.settings' |
95 | e38ad44f | Georgios D. Tsoukalas | |
96 | e38ad44f | Georgios D. Tsoukalas | import astakos |
97 | e38ad44f | Georgios D. Tsoukalas | from astakos.im.models import AstakosUser as A |
98 | e38ad44f | Georgios D. Tsoukalas | |
99 | e38ad44f | Georgios D. Tsoukalas | |
100 | e38ad44f | Georgios D. Tsoukalas | def user_filter(user): |
101 | e38ad44f | Georgios D. Tsoukalas | return A.objects.filter(email__iexact=user.email).count() > 1 |
102 | e38ad44f | Georgios D. Tsoukalas | |
103 | e38ad44f | Georgios D. Tsoukalas | argv = sys.argv |
104 | e38ad44f | Georgios D. Tsoukalas | argc = len(sys.argv) |
105 | e38ad44f | Georgios D. Tsoukalas | |
106 | e38ad44f | Georgios D. Tsoukalas | if argc < 2: |
107 | e38ad44f | Georgios D. Tsoukalas | print "Usage: ./delete_astakos_users.py <id>..." |
108 | e38ad44f | Georgios D. Tsoukalas | raise SystemExit() |
109 | e38ad44f | Georgios D. Tsoukalas | |
110 | e38ad44f | Georgios D. Tsoukalas | id_list = [int(x) for x in argv[1:]] |
111 | e38ad44f | Georgios D. Tsoukalas | |
112 | e38ad44f | Georgios D. Tsoukalas | print "" |
113 | e38ad44f | Georgios D. Tsoukalas | print "This will permanently delete the following users:\n" |
114 | e38ad44f | Georgios D. Tsoukalas | print "id email (is_active, creation date)" |
115 | e38ad44f | Georgios D. Tsoukalas | print "-- --------------------------------" |
116 | e38ad44f | Georgios D. Tsoukalas | |
117 | e38ad44f | Georgios D. Tsoukalas | users = A.objects.filter(id__in=id_list) |
118 | e38ad44f | Georgios D. Tsoukalas | for user in users: |
119 | e38ad44f | Georgios D. Tsoukalas | print "%s: %s (%s, %s)" % (user.id, user.email, user.is_active, |
120 | e38ad44f | Georgios D. Tsoukalas | user.date_joined) |
121 | e38ad44f | Georgios D. Tsoukalas | |
122 | e38ad44f | Georgios D. Tsoukalas | print "\nExecute? (yes/no): ", |
123 | e38ad44f | Georgios D. Tsoukalas | line = raw_input().rstrip() |
124 | e38ad44f | Georgios D. Tsoukalas | if line != 'yes': |
125 | e38ad44f | Georgios D. Tsoukalas | print "\nCancelled" |
126 | e38ad44f | Georgios D. Tsoukalas | raise SystemExit() |
127 | e38ad44f | Georgios D. Tsoukalas | |
128 | e38ad44f | Georgios D. Tsoukalas | print "\nConfirmed." |
129 | e38ad44f | Georgios D. Tsoukalas | sleep(2) |
130 | e38ad44f | Georgios D. Tsoukalas | for user in users: |
131 | e38ad44f | Georgios D. Tsoukalas | print "deleting %s: %s" % (user.id, user.email) |
132 | e38ad44f | Georgios D. Tsoukalas | user.delete() |
133 | e38ad44f | Georgios D. Tsoukalas | |
134 | e38ad44f | Georgios D. Tsoukalas | EOF |
135 | e38ad44f | Georgios D. Tsoukalas | |
136 | 457fe4fc | Georgios D. Tsoukalas | astakos-host$ python ./delete_astakos_users.py 30 40 |
137 | 7e38e668 | Georgios D. Tsoukalas | |
138 | 457fe4fc | Georgios D. Tsoukalas | .. warning:: |
139 | 7e38e668 | Georgios D. Tsoukalas | |
140 | e38ad44f | Georgios D. Tsoukalas | After deleting users with the ``delete_astakos_users.py`` script, |
141 | e38ad44f | Georgios D. Tsoukalas | check again with ``find_astakos_users_with_conflicting_emails.py`` |
142 | e38ad44f | Georgios D. Tsoukalas | (as in :ref:`find_duplicate_emails`) |
143 | e38ad44f | Georgios D. Tsoukalas | to make sure that no duplicate email conflicts remain. |
144 | 7e38e668 | Georgios D. Tsoukalas | |
145 | 457fe4fc | Georgios D. Tsoukalas | |
146 | 457fe4fc | Georgios D. Tsoukalas | 3. Upgrade Synnefo and configure settings |
147 | 457fe4fc | Georgios D. Tsoukalas | ========================================= |
148 | 7e38e668 | Georgios D. Tsoukalas | |
149 | 7e38e668 | Georgios D. Tsoukalas | 3.1 Install the new versions of packages |
150 | 7e38e668 | Georgios D. Tsoukalas | ---------------------------------------- |
151 | 7e38e668 | Georgios D. Tsoukalas | |
152 | 7e38e668 | Georgios D. Tsoukalas | :: |
153 | 7e38e668 | Georgios D. Tsoukalas | |
154 | 7e38e668 | Georgios D. Tsoukalas | astakos.host$ apt-get install \ |
155 | eeb3f40b | Georgios D. Tsoukalas | kamaki \ |
156 | 7e38e668 | Georgios D. Tsoukalas | snf-common \ |
157 | 7e38e668 | Georgios D. Tsoukalas | snf-webproject \ |
158 | 7e38e668 | Georgios D. Tsoukalas | snf-quotaholder-app \ |
159 | 9a72e003 | Giorgos Korfiatis | snf-astakos-app |
160 | 7e38e668 | Georgios D. Tsoukalas | |
161 | 7e38e668 | Georgios D. Tsoukalas | |
162 | 7e38e668 | Georgios D. Tsoukalas | cyclades.host$ apt-get install \ |
163 | eeb3f40b | Georgios D. Tsoukalas | kamaki \ |
164 | 7e38e668 | Georgios D. Tsoukalas | snf-common \ |
165 | 9a72e003 | Giorgos Korfiatis | snf-webproject \ |
166 | 7e38e668 | Georgios D. Tsoukalas | snf-pithos-backend \ |
167 | 9a72e003 | Giorgos Korfiatis | snf-cyclades-app |
168 | 7e38e668 | Georgios D. Tsoukalas | |
169 | 457fe4fc | Georgios D. Tsoukalas | pithos.host$ apt-get install \ |
170 | eeb3f40b | Georgios D. Tsoukalas | kamaki \ |
171 | 7e38e668 | Georgios D. Tsoukalas | snf-common \ |
172 | 9a72e003 | Giorgos Korfiatis | snf-webproject \ |
173 | 7e38e668 | Georgios D. Tsoukalas | snf-pithos-backend \ |
174 | 7e38e668 | Georgios D. Tsoukalas | snf-pithos-app \ |
175 | 9a72e003 | Giorgos Korfiatis | snf-pithos-webclient |
176 | 7e38e668 | Georgios D. Tsoukalas | |
177 | 3910c691 | Constantinos Venetsanopoulos | |
178 | 3910c691 | Constantinos Venetsanopoulos | ganeti.node$ apt-get install \ |
179 | eeb3f40b | Georgios D. Tsoukalas | kamaki \ |
180 | eeb3f40b | Georgios D. Tsoukalas | snf-common \ |
181 | 3910c691 | Constantinos Venetsanopoulos | snf-cyclades-gtools \ |
182 | 9a72e003 | Giorgos Korfiatis | snf-pithos-backend |
183 | 3910c691 | Constantinos Venetsanopoulos | |
184 | f4c10ea9 | Kostas Papadimitriou | .. note:: |
185 | f4c10ea9 | Kostas Papadimitriou | |
186 | eeb3f40b | Georgios D. Tsoukalas | Installing the packages will cause services to start. Make sure you bring |
187 | eeb3f40b | Georgios D. Tsoukalas | them down again (at least ``gunicorn``, ``snf-dispatcher``) |
188 | f4c10ea9 | Kostas Papadimitriou | |
189 | 457fe4fc | Georgios D. Tsoukalas | 3.2 Sync and migrate Django DB |
190 | 7e38e668 | Georgios D. Tsoukalas | ------------------------------ |
191 | 7e38e668 | Georgios D. Tsoukalas | |
192 | e38ad44f | Georgios D. Tsoukalas | .. note:: |
193 | e38ad44f | Georgios D. Tsoukalas | |
194 | e38ad44f | Georgios D. Tsoukalas | If you are asked about stale content types during the migration process, |
195 | e38ad44f | Georgios D. Tsoukalas | answer 'no' and let the migration finish. |
196 | e38ad44f | Georgios D. Tsoukalas | |
197 | 7e38e668 | Georgios D. Tsoukalas | :: |
198 | 7e38e668 | Georgios D. Tsoukalas | |
199 | 457fe4fc | Georgios D. Tsoukalas | astakos-host$ snf-manage syncdb |
200 | 457fe4fc | Georgios D. Tsoukalas | astakos-host$ snf-manage migrate |
201 | 457fe4fc | Georgios D. Tsoukalas | |
202 | 457fe4fc | Georgios D. Tsoukalas | cyclades-host$ snf-manage syncdb |
203 | 457fe4fc | Georgios D. Tsoukalas | cyclades-host$ snf-manage migrate |
204 | 457fe4fc | Georgios D. Tsoukalas | |
205 | 457fe4fc | Georgios D. Tsoukalas | .. note:: |
206 | 457fe4fc | Georgios D. Tsoukalas | |
207 | 457fe4fc | Georgios D. Tsoukalas | After the migration, Astakos has created uuids for all users, |
208 | 457fe4fc | Georgios D. Tsoukalas | and has set the uuid as the public identifier of a user. |
209 | 457fe4fc | Georgios D. Tsoukalas | This uuid is to be used both at other services (Cyclades, Pithos) |
210 | 457fe4fc | Georgios D. Tsoukalas | and at the clientside (kamaki client settings). |
211 | 457fe4fc | Georgios D. Tsoukalas | |
212 | 457fe4fc | Georgios D. Tsoukalas | Duplicate-email users have been deleted earlier in |
213 | 457fe4fc | Georgios D. Tsoukalas | :ref:`remove_astakos_duplicate` |
214 | 457fe4fc | Georgios D. Tsoukalas | |
215 | 457fe4fc | Georgios D. Tsoukalas | 3.3 Setup quota settings for all services |
216 | 457fe4fc | Georgios D. Tsoukalas | ----------------------------------------- |
217 | 7e38e668 | Georgios D. Tsoukalas | |
218 | 5c088197 | Constantinos Venetsanopoulos | Generally: |
219 | 5c088197 | Constantinos Venetsanopoulos | |
220 | 457fe4fc | Georgios D. Tsoukalas | :: |
221 | 7e38e668 | Georgios D. Tsoukalas | |
222 | 7e38e668 | Georgios D. Tsoukalas | # Service Setting Value |
223 | 7a882417 | Georgios D. Tsoukalas | # quotaholder: QUOTAHOLDER_URL = https://quotaholder.host/quotaholder/v |
224 | 7e38e668 | Georgios D. Tsoukalas | # quotaholder: QUOTAHOLDER_TOKEN = <random string> |
225 | 7e38e668 | Georgios D. Tsoukalas | |
226 | 7e38e668 | Georgios D. Tsoukalas | # astakos: ASTAKOS_QUOTAHOLDER_TOKEN = <the same random string> |
227 | 7e38e668 | Georgios D. Tsoukalas | # astakos: ASTAKOS_QUOTAHOLDER_URL = https://quotaholder.host/quotaholder/v |
228 | 7e38e668 | Georgios D. Tsoukalas | |
229 | 7e38e668 | Georgios D. Tsoukalas | # cyclades: CYCLADES_QUOTAHOLDER_TOKEN = <the same random string> |
230 | eeb3f40b | Georgios D. Tsoukalas | # cyclades: CYCLADES_QUOTAHOLDER_URL = https://quotaholder.host/quotaholder/v |
231 | 7e38e668 | Georgios D. Tsoukalas | # cyclades: CYCLADES_USE_QUOTAHOLDER = True |
232 | 7e38e668 | Georgios D. Tsoukalas | |
233 | 7e38e668 | Georgios D. Tsoukalas | |
234 | 7e38e668 | Georgios D. Tsoukalas | # pithos: PITHOS_QUOTAHOLDER_TOKEN = <the same random string> |
235 | eeb3f40b | Georgios D. Tsoukalas | # pithos: PITHOS_QUOTAHOLDER_URL = https://quotaholder.host/quotaholder/v |
236 | 6c931e8b | Georgios D. Tsoukalas | # pithos: PITHOS_USE_QUOTAHOLDER = True |
237 | 7e38e668 | Georgios D. Tsoukalas | # All services must match the quotaholder token and url configured for quotaholder. |
238 | 7e38e668 | Georgios D. Tsoukalas | |
239 | 5c088197 | Constantinos Venetsanopoulos | Specifically: |
240 | 5c088197 | Constantinos Venetsanopoulos | |
241 | 5c088197 | Constantinos Venetsanopoulos | On the Astakos host, edit ``/etc/synnefo/20-snf-astakos-app-settings.conf``: |
242 | 5c088197 | Constantinos Venetsanopoulos | |
243 | 5c088197 | Constantinos Venetsanopoulos | :: |
244 | 5c088197 | Constantinos Venetsanopoulos | |
245 | 7a882417 | Georgios D. Tsoukalas | QUOTAHOLDER_URL = 'https://accounts.example.synnefo.org/quotaholder/v' |
246 | 5c088197 | Constantinos Venetsanopoulos | QUOTAHOLDER_TOKEN = 'aExampleTokenJbFm12w' |
247 | 5c088197 | Constantinos Venetsanopoulos | ASTAKOS_QUOTAHOLDER_TOKEN = 'aExampleTokenJbFm12w' |
248 | 9521d937 | Georgios D. Tsoukalas | ASTAKOS_QUOTAHOLDER_URL = 'https://accounts.example.synnefo.org/quotaholder/v' |
249 | 5c088197 | Constantinos Venetsanopoulos | |
250 | 5c088197 | Constantinos Venetsanopoulos | On the Cyclades host, edit ``/etc/synnefo/20-snf-cyclades-app-quotas.conf``: |
251 | 5c088197 | Constantinos Venetsanopoulos | |
252 | 5c088197 | Constantinos Venetsanopoulos | :: |
253 | 5c088197 | Constantinos Venetsanopoulos | |
254 | 5c088197 | Constantinos Venetsanopoulos | CYCLADES_USE_QUOTAHOLDER = True |
255 | 9521d937 | Georgios D. Tsoukalas | CYCLADES_QUOTAHOLDER_URL = 'https://accounts.example.synnefo.org/quotaholder/v' |
256 | 5c088197 | Constantinos Venetsanopoulos | CYCLADES_QUOTAHOLDER_TOKEN = 'aExampleTokenJbFm12w' |
257 | 5c088197 | Constantinos Venetsanopoulos | |
258 | 7fa06e70 | Georgios D. Tsoukalas | # Set to False if astakos & cyclades are on the same host |
259 | 7fa06e70 | Georgios D. Tsoukalas | #CYCLADES_PROXY_USER_SERVICES = True |
260 | 7fa06e70 | Georgios D. Tsoukalas | |
261 | 7fa06e70 | Georgios D. Tsoukalas | .. note:: |
262 | 7fa06e70 | Georgios D. Tsoukalas | |
263 | 7fa06e70 | Georgios D. Tsoukalas | If Cylcades and Astakos are installed on the same server, |
264 | 7fa06e70 | Georgios D. Tsoukalas | set ``CYCLADES_PROXY_USER_SERVICES = False`` |
265 | 7fa06e70 | Georgios D. Tsoukalas | |
266 | 7fa06e70 | Georgios D. Tsoukalas | |
267 | 5c088197 | Constantinos Venetsanopoulos | On the Pithos host, edit ``/etc/synnefo/20-snf-pithos-app-settings.conf``: |
268 | 5c088197 | Constantinos Venetsanopoulos | |
269 | 5c088197 | Constantinos Venetsanopoulos | :: |
270 | 5c088197 | Constantinos Venetsanopoulos | |
271 | 9521d937 | Georgios D. Tsoukalas | PITHOS_QUOTAHOLDER_URL = 'https://accounts.example.synnefo.org/quotaholder/v' |
272 | 5c088197 | Constantinos Venetsanopoulos | PITHOS_QUOTAHOLDER_TOKEN = 'aExampleTokenJbFm12w' |
273 | eeb3f40b | Georgios D. Tsoukalas | PITHOS_USE_QUOTAHOLDER = False # will set to True after migration |
274 | eeb3f40b | Georgios D. Tsoukalas | |
275 | eeb3f40b | Georgios D. Tsoukalas | .. note:: |
276 | eeb3f40b | Georgios D. Tsoukalas | |
277 | eeb3f40b | Georgios D. Tsoukalas | During the migration it must be set, ``PITHOS_USE_QUOTAHOLDER = False``. |
278 | eeb3f40b | Georgios D. Tsoukalas | Set to ``True`` once the migration is over. |
279 | 5c088197 | Constantinos Venetsanopoulos | |
280 | 457fe4fc | Georgios D. Tsoukalas | 3.4 Setup astakos |
281 | 457fe4fc | Georgios D. Tsoukalas | ----------------- |
282 | 7e38e668 | Georgios D. Tsoukalas | |
283 | 5c088197 | Constantinos Venetsanopoulos | - **Remove** this redirection from astakos front-end web server :: |
284 | 7e38e668 | Georgios D. Tsoukalas | |
285 | 457fe4fc | Georgios D. Tsoukalas | RewriteRule ^/login(.*) /im/login/redirect$1 [PT,NE] |
286 | 7e38e668 | Georgios D. Tsoukalas | |
287 | 9a72e003 | Giorgos Korfiatis | (see `<http://www.synnefo.org/docs/synnefo/latest/quick-install-admin-guide.html#apache2-setup>`_) |
288 | 7e38e668 | Georgios D. Tsoukalas | |
289 | 52f2c69b | Stavros Sachtouris | - Enable users to change their contact email. Edit ``/etc/synnefo/20-snf-astakos-app-settings.conf`` :: |
290 | 7e38e668 | Georgios D. Tsoukalas | |
291 | 5c088197 | Constantinos Venetsanopoulos | ASTAKOS_EMAILCHANGE_ENABLED = True |
292 | 7e38e668 | Georgios D. Tsoukalas | |
293 | eeb3f40b | Georgios D. Tsoukalas | - Rename the following (Astakos-specific) setting:: |
294 | eeb3f40b | Georgios D. Tsoukalas | |
295 | eeb3f40b | Georgios D. Tsoukalas | ASTAKOS_DEFAULT_FROM_EMAIL |
296 | eeb3f40b | Georgios D. Tsoukalas | |
297 | eeb3f40b | Georgios D. Tsoukalas | to this (Django-specific) name:: |
298 | eeb3f40b | Georgios D. Tsoukalas | |
299 | eeb3f40b | Georgios D. Tsoukalas | SERVER_EMAIL |
300 | eeb3f40b | Georgios D. Tsoukalas | |
301 | eeb3f40b | Georgios D. Tsoukalas | - Instead of using the following (Astakos-specific) setting:: |
302 | eeb3f40b | Georgios D. Tsoukalas | |
303 | eeb3f40b | Georgios D. Tsoukalas | ASTAKOS_DEFAULT_ADMIN_EMAIL |
304 | eeb3f40b | Georgios D. Tsoukalas | |
305 | eeb3f40b | Georgios D. Tsoukalas | include one or more entries in this (Django-specific) setting:: |
306 | eeb3f40b | Georgios D. Tsoukalas | |
307 | eeb3f40b | Georgios D. Tsoukalas | ADMINS = ( |
308 | eeb3f40b | Georgios D. Tsoukalas | ('Joe Doe', 'doe@example.net'), |
309 | eeb3f40b | Georgios D. Tsoukalas | ('Mary Jean', 'mary@example.net'), |
310 | eeb3f40b | Georgios D. Tsoukalas | ) |
311 | eeb3f40b | Georgios D. Tsoukalas | |
312 | eeb3f40b | Georgios D. Tsoukalas | .. note:: |
313 | eeb3f40b | Georgios D. Tsoukalas | |
314 | eeb3f40b | Georgios D. Tsoukalas | The ``SERVER_EMAIL`` and ``ADMINS`` settings are Django-specific. |
315 | 9a72e003 | Giorgos Korfiatis | As such they will be shared among any two (or more) services that happen |
316 | eeb3f40b | Georgios D. Tsoukalas | to be collocated within the same application server (e.g. astakos & |
317 | eeb3f40b | Georgios D. Tsoukalas | cyclades within the same gunicorn) |
318 | eeb3f40b | Georgios D. Tsoukalas | |
319 | 5c088197 | Constantinos Venetsanopoulos | 3.5 Setup Cyclades |
320 | 457fe4fc | Georgios D. Tsoukalas | ------------------ |
321 | 7e38e668 | Georgios D. Tsoukalas | |
322 | 5c088197 | Constantinos Venetsanopoulos | - Run on the Astakos host :: |
323 | 7e38e668 | Georgios D. Tsoukalas | |
324 | 5c088197 | Constantinos Venetsanopoulos | # snf-manage service-list |
325 | 7e38e668 | Georgios D. Tsoukalas | |
326 | eeb3f40b | Georgios D. Tsoukalas | - Set the Cyclades service token in |
327 | eeb3f40b | Georgios D. Tsoukalas | ``/etc/synnefo/20-snf-cyclades-app-api.conf`` :: |
328 | 7e38e668 | Georgios D. Tsoukalas | |
329 | 5c088197 | Constantinos Venetsanopoulos | CYCLADES_ASTAKOS_SERVICE_TOKEN = 'asfasdf_CycladesServiceToken_iknl' |
330 | 61db6580 | Georgios D. Tsoukalas | |
331 | 2291ead5 | Christos Stavrakakis | - Since version 0.13, Synnefo uses **VMAPI** in order to prevent sensitive data |
332 | 2291ead5 | Christos Stavrakakis | needed by 'snf-image' to be stored in Ganeti configuration (e.g. VM |
333 | 2291ead5 | Christos Stavrakakis | password). This is achieved by storing all sensitive information to a CACHE |
334 | 2291ead5 | Christos Stavrakakis | backend and exporting it via VMAPI. The cache entries are invalidated after |
335 | 2291ead5 | Christos Stavrakakis | the first request. Synnefo uses **memcached** as a django cache backend. |
336 | 5c088197 | Constantinos Venetsanopoulos | To install, run on the Cyclades host:: |
337 | 7e38e668 | Georgios D. Tsoukalas | |
338 | 457fe4fc | Georgios D. Tsoukalas | apt-get install memcached |
339 | 457fe4fc | Georgios D. Tsoukalas | apt-get install python-memcache |
340 | 7e38e668 | Georgios D. Tsoukalas | |
341 | 2291ead5 | Christos Stavrakakis | You will also need to configure Cyclades to use the memcached cache backend. |
342 | 2291ead5 | Christos Stavrakakis | Namely, you need to set IP address and port of the memcached daemon, and the |
343 | 5c088197 | Constantinos Venetsanopoulos | default timeout (seconds tha value is stored in the cache). Edit |
344 | 5c088197 | Constantinos Venetsanopoulos | ``/etc/synnefo/20-snf-cyclades-app-vmapi.conf`` :: |
345 | 7e38e668 | Georgios D. Tsoukalas | |
346 | 2291ead5 | Christos Stavrakakis | VMAPI_CACHE_BACKEND = "memcached://127.0.0.1:11211/?timeout=3600" |
347 | 2291ead5 | Christos Stavrakakis | |
348 | 2291ead5 | Christos Stavrakakis | |
349 | 2291ead5 | Christos Stavrakakis | Finally, set the BASE_URL for the VMAPI, which is actually the base URL of |
350 | 3910c691 | Constantinos Venetsanopoulos | Cyclades, again in ``/etc/synnefo/20-snf-cyclades-app-vmapi.conf``. Make sure |
351 | 3910c691 | Constantinos Venetsanopoulos | the domain is exaclty the same, so that no re-directs happen :: |
352 | 7e38e668 | Georgios D. Tsoukalas | |
353 | 9521d937 | Georgios D. Tsoukalas | VMAPI_BASE_URL = "https://cyclades.example.synnefo.org" |
354 | 7e38e668 | Georgios D. Tsoukalas | |
355 | 457fe4fc | Georgios D. Tsoukalas | .. note:: |
356 | 7e38e668 | Georgios D. Tsoukalas | |
357 | 457fe4fc | Georgios D. Tsoukalas | - These settings are needed in all Cyclades workers. |
358 | 7e38e668 | Georgios D. Tsoukalas | |
359 | 2291ead5 | Christos Stavrakakis | - VMAPI_CACHE_BACKEND just overrides django's CACHE_BACKEND setting |
360 | 2291ead5 | Christos Stavrakakis | |
361 | 457fe4fc | Georgios D. Tsoukalas | - memcached must be reachable from all Cyclades workers. |
362 | 457fe4fc | Georgios D. Tsoukalas | |
363 | 457fe4fc | Georgios D. Tsoukalas | - For more information about configuring django to use memcached: |
364 | 457fe4fc | Georgios D. Tsoukalas | https://docs.djangoproject.com/en/1.2/topics/cache |
365 | 7e38e668 | Georgios D. Tsoukalas | |
366 | 457fe4fc | Georgios D. Tsoukalas | 3.6 Setup Pithos |
367 | 457fe4fc | Georgios D. Tsoukalas | ---------------- |
368 | 7e38e668 | Georgios D. Tsoukalas | |
369 | 457fe4fc | Georgios D. Tsoukalas | - Pithos forwards user catalog services to Astakos so that web clients may |
370 | 5c088197 | Constantinos Venetsanopoulos | access them for uuid-displayname translations. Edit on the Pithos host |
371 | 5c088197 | Constantinos Venetsanopoulos | ``/etc/synnefo/20-snf-pithos-app-settings.conf`` :: |
372 | 7e38e668 | Georgios D. Tsoukalas | |
373 | 9521d937 | Georgios D. Tsoukalas | PITHOS_USER_CATALOG_URL = https://accounts.example.synnefo.org/user_catalogs/ |
374 | 9521d937 | Georgios D. Tsoukalas | PITHOS_USER_FEEDBACK_URL = https://accounts.example.synnefo.org/feedback/ |
375 | 9521d937 | Georgios D. Tsoukalas | PITHOS_USER_LOGIN_URL = https://accounts.example.synnefo.org/login/ |
376 | 7e38e668 | Georgios D. Tsoukalas | |
377 | 7fa06e70 | Georgios D. Tsoukalas | # Set to False if astakos & pithos are on the same host |
378 | 7fa06e70 | Georgios D. Tsoukalas | #PITHOS_PROXY_USER_SERVICES = True |
379 | 7fa06e70 | Georgios D. Tsoukalas | |
380 | 7fa06e70 | Georgios D. Tsoukalas | .. note:: |
381 | 7fa06e70 | Georgios D. Tsoukalas | |
382 | 7fa06e70 | Georgios D. Tsoukalas | If Pithos and Astakos are installed on the same server, |
383 | 7fa06e70 | Georgios D. Tsoukalas | set ``PITHOS_PROXY_USER_SERVICES = False`` |
384 | 7e38e668 | Georgios D. Tsoukalas | |
385 | 7e38e668 | Georgios D. Tsoukalas | 4. Start astakos and quota services |
386 | 7e38e668 | Georgios D. Tsoukalas | =================================== |
387 | e38ad44f | Georgios D. Tsoukalas | .. warning:: |
388 | 7e38e668 | Georgios D. Tsoukalas | |
389 | e38ad44f | Georgios D. Tsoukalas | To ensure consistency, prevent public access to astakos during migrations. |
390 | e38ad44f | Georgios D. Tsoukalas | This can be done via firewall or webserver access control. |
391 | 457fe4fc | Georgios D. Tsoukalas | |
392 | eeb3f40b | Georgios D. Tsoukalas | Start (or restart, if running) the webserver and gunicorn on the Astakos host. |
393 | eeb3f40b | Georgios D. Tsoukalas | E.g.:: |
394 | eeb3f40b | Georgios D. Tsoukalas | |
395 | eeb3f40b | Georgios D. Tsoukalas | # service apache2 start |
396 | eeb3f40b | Georgios D. Tsoukalas | # service gunicorn start |
397 | eeb3f40b | Georgios D. Tsoukalas | |
398 | 457fe4fc | Georgios D. Tsoukalas | .. _astakos-load-resources: |
399 | 7e38e668 | Georgios D. Tsoukalas | |
400 | 7e38e668 | Georgios D. Tsoukalas | 5. Load resource definitions into Astakos |
401 | 7e38e668 | Georgios D. Tsoukalas | ========================================= |
402 | 7e38e668 | Georgios D. Tsoukalas | |
403 | 5c088197 | Constantinos Venetsanopoulos | First, set the corresponding values on the following dict in |
404 | 5c088197 | Constantinos Venetsanopoulos | ``/etc/synnefo/20-snf-astakos-app-settings.conf`` :: |
405 | 7e38e668 | Georgios D. Tsoukalas | |
406 | 7e38e668 | Georgios D. Tsoukalas | # Set the cloud service properties |
407 | 7e38e668 | Georgios D. Tsoukalas | ASTAKOS_SERVICES = { |
408 | 7e38e668 | Georgios D. Tsoukalas | 'cyclades': { |
409 | 6c931e8b | Georgios D. Tsoukalas | # # Specifying the key 'url' will overwrite it. |
410 | 6c931e8b | Georgios D. Tsoukalas | # # Use this to (re)set service URL. |
411 | 9521d937 | Georgios D. Tsoukalas | # 'url': 'https://cyclades.example.synnefo.org/ui/', |
412 | 6c931e8b | Georgios D. Tsoukalas | # # order services in listings, cloudbar, etc. |
413 | 6c931e8b | Georgios D. Tsoukalas | # 'order' : 1 |
414 | 7e38e668 | Georgios D. Tsoukalas | 'resources': [{ |
415 | 6c931e8b | Georgios D. Tsoukalas | 'name': 'disk', |
416 | 6c931e8b | Georgios D. Tsoukalas | 'group': 'compute', |
417 | 6c931e8b | Georgios D. Tsoukalas | 'uplimit': 30*1024*1024*1024, |
418 | 6c931e8b | Georgios D. Tsoukalas | 'unit': 'bytes', |
419 | 7e38e668 | Georgios D. Tsoukalas | 'desc': 'Virtual machine disk size' |
420 | 6c931e8b | Georgios D. Tsoukalas | }, { |
421 | 6c931e8b | Georgios D. Tsoukalas | 'name': 'cpu', |
422 | 6c931e8b | Georgios D. Tsoukalas | 'group': 'compute', |
423 | 6c931e8b | Georgios D. Tsoukalas | 'uplimit': 6, |
424 | 7e38e668 | Georgios D. Tsoukalas | 'desc': 'Number of virtual machine processors' |
425 | 6c931e8b | Georgios D. Tsoukalas | }, { |
426 | 6c931e8b | Georgios D. Tsoukalas | 'name': 'ram', |
427 | 6c931e8b | Georgios D. Tsoukalas | 'group': 'compute', |
428 | 6c931e8b | Georgios D. Tsoukalas | 'uplimit': 6*1024*1024*1024, |
429 | 6c931e8b | Georgios D. Tsoukalas | 'unit': 'bytes', |
430 | 7e38e668 | Georgios D. Tsoukalas | 'desc': 'Virtual machines' |
431 | 6c931e8b | Georgios D. Tsoukalas | }, { |
432 | 6c931e8b | Georgios D. Tsoukalas | 'name': 'vm', |
433 | 6c931e8b | Georgios D. Tsoukalas | 'group': 'compute', |
434 | 6c931e8b | Georgios D. Tsoukalas | 'uplimit': 2, |
435 | 7e38e668 | Georgios D. Tsoukalas | 'desc': 'Number of virtual machines' |
436 | 6c931e8b | Georgios D. Tsoukalas | }, { |
437 | 6c931e8b | Georgios D. Tsoukalas | 'name': 'network.private', |
438 | 6c931e8b | Georgios D. Tsoukalas | 'group': 'network', |
439 | 6c931e8b | Georgios D. Tsoukalas | 'uplimit': 1, |
440 | 7e38e668 | Georgios D. Tsoukalas | 'desc': 'Private networks' |
441 | 7e38e668 | Georgios D. Tsoukalas | } |
442 | 7e38e668 | Georgios D. Tsoukalas | ] |
443 | 7e38e668 | Georgios D. Tsoukalas | }, |
444 | 7e38e668 | Georgios D. Tsoukalas | 'pithos+': { |
445 | 6c931e8b | Georgios D. Tsoukalas | # # Use this to (re)set service URL. |
446 | 9521d937 | Georgios D. Tsoukalas | # 'url': 'https://pithos.example.synnefo.org/ui/', |
447 | 6c931e8b | Georgios D. Tsoukalas | # # order services in listings, cloudbar, etc. |
448 | 6c931e8b | Georgios D. Tsoukalas | # 'order' : 2 |
449 | 7e38e668 | Georgios D. Tsoukalas | 'resources':[{ |
450 | 6c931e8b | Georgios D. Tsoukalas | 'name': 'diskspace', |
451 | 6c931e8b | Georgios D. Tsoukalas | 'group': 'storage', |
452 | 6c931e8b | Georgios D. Tsoukalas | 'uplimit': 5*1024*1024*1024, |
453 | 6c931e8b | Georgios D. Tsoukalas | 'unit': 'bytes', |
454 | 7e38e668 | Georgios D. Tsoukalas | 'desc': 'Pithos account diskspace' |
455 | 7e38e668 | Georgios D. Tsoukalas | }] |
456 | 7e38e668 | Georgios D. Tsoukalas | } |
457 | 7e38e668 | Georgios D. Tsoukalas | } |
458 | 7e38e668 | Georgios D. Tsoukalas | |
459 | eeb3f40b | Georgios D. Tsoukalas | .. note:: |
460 | eeb3f40b | Georgios D. Tsoukalas | |
461 | eeb3f40b | Georgios D. Tsoukalas | The name of the Pithos service is ``pithos+``. |
462 | eeb3f40b | Georgios D. Tsoukalas | If you have named your pithos service ``pithos``, without ``+``, |
463 | eeb3f40b | Georgios D. Tsoukalas | then you must rename it:: |
464 | eeb3f40b | Georgios D. Tsoukalas | |
465 | eeb3f40b | Georgios D. Tsoukalas | $ snf-manage service-list | grep pithos # find service id |
466 | eeb3f40b | Georgios D. Tsoukalas | $ snf-manage service-update --name='pithos+' <service id> |
467 | eeb3f40b | Georgios D. Tsoukalas | |
468 | 5c088197 | Constantinos Venetsanopoulos | Then, configure and load the available resources per service |
469 | 5c088197 | Constantinos Venetsanopoulos | and associated default limits into Astakos. On the Astakos host run :: |
470 | 5c088197 | Constantinos Venetsanopoulos | |
471 | 5c088197 | Constantinos Venetsanopoulos | # snf-manage astakos-init --load-service-resources |
472 | 5c088197 | Constantinos Venetsanopoulos | |
473 | 5c088197 | Constantinos Venetsanopoulos | |
474 | 457fe4fc | Georgios D. Tsoukalas | .. note:: |
475 | 457fe4fc | Georgios D. Tsoukalas | |
476 | 457fe4fc | Georgios D. Tsoukalas | Before v0.13, only `cyclades.vm`, `cyclades.network.private`, |
477 | eeb3f40b | Georgios D. Tsoukalas | and `pithos+.diskspace` existed (not with these names, |
478 | eeb3f40b | Georgios D. Tsoukalas | there were per-service settings). |
479 | 457fe4fc | Georgios D. Tsoukalas | However, limits to the new resources must also be set. |
480 | 457fe4fc | Georgios D. Tsoukalas | |
481 | 9a72e003 | Giorgos Korfiatis | If the intention is to keep a resource unlimited, (counting on that VM |
482 | 457fe4fc | Georgios D. Tsoukalas | creation will be limited by other resources' limit) it is best to calculate |
483 | 457fe4fc | Georgios D. Tsoukalas | a value that is too large to be reached because of other limits (and |
484 | 457fe4fc | Georgios D. Tsoukalas | available flavours), but not much larger than needed because this might |
485 | 457fe4fc | Georgios D. Tsoukalas | confuse users who do not readily understand that multiple limits apply and |
486 | 457fe4fc | Georgios D. Tsoukalas | flavors are limited. |
487 | 7e38e668 | Georgios D. Tsoukalas | |
488 | 7e38e668 | Georgios D. Tsoukalas | |
489 | 457fe4fc | Georgios D. Tsoukalas | 6. Migrate Services user names to uuids |
490 | 457fe4fc | Georgios D. Tsoukalas | ======================================= |
491 | 7e38e668 | Georgios D. Tsoukalas | |
492 | 7e38e668 | Georgios D. Tsoukalas | |
493 | 457fe4fc | Georgios D. Tsoukalas | 6.1 Double-check cyclades before user case/uuid migration |
494 | 457fe4fc | Georgios D. Tsoukalas | --------------------------------------------------------- |
495 | 7e38e668 | Georgios D. Tsoukalas | |
496 | 7e38e668 | Georgios D. Tsoukalas | :: |
497 | 7e38e668 | Georgios D. Tsoukalas | |
498 | f4c10ea9 | Kostas Papadimitriou | cyclades.host$ snf-manage cyclades-astakos-migrate-013 --validate |
499 | 7e38e668 | Georgios D. Tsoukalas | |
500 | 7e38e668 | Georgios D. Tsoukalas | Duplicate user found? |
501 | 7e38e668 | Georgios D. Tsoukalas | |
502 | 7e38e668 | Georgios D. Tsoukalas | - either *merge* (merge will merge all resources to one user):: |
503 | 7e38e668 | Georgios D. Tsoukalas | |
504 | f4c10ea9 | Kostas Papadimitriou | cyclades.host$ snf-manage cyclades-astakos-migrate-013 --merge-user=kpap@grnet.gr |
505 | 7e38e668 | Georgios D. Tsoukalas | |
506 | 7e38e668 | Georgios D. Tsoukalas | - or *delete* :: |
507 | 7e38e668 | Georgios D. Tsoukalas | |
508 | f4c10ea9 | Kostas Papadimitriou | cyclades.host$ snf-manage cyclades-astakos-migrate-013 --delete-user=KPap@grnet.gr |
509 | 7e38e668 | Georgios D. Tsoukalas | # (only KPap will be deleted not kpap) |
510 | 7e38e668 | Georgios D. Tsoukalas | |
511 | 1ed4428d | Sofia Papagiannaki | 6.2 Double-check pithos before user case/uuid migration |
512 | 1ed4428d | Sofia Papagiannaki | --------------------------------------------------------- |
513 | 1ed4428d | Sofia Papagiannaki | |
514 | 1ed4428d | Sofia Papagiannaki | :: |
515 | 1ed4428d | Sofia Papagiannaki | |
516 | 1ed4428d | Sofia Papagiannaki | pithos.host$ snf-manage pithos-manage-accounts --list-duplicate |
517 | 1ed4428d | Sofia Papagiannaki | |
518 | 1ed4428d | Sofia Papagiannaki | Duplicate user found? |
519 | 1ed4428d | Sofia Papagiannaki | |
520 | 5c088197 | Constantinos Venetsanopoulos | If you want to migrate files first: |
521 | 5c088197 | Constantinos Venetsanopoulos | |
522 | 5c088197 | Constantinos Venetsanopoulos | - *merge* (merge will merge all resources to one user):: |
523 | 1ed4428d | Sofia Papagiannaki | |
524 | 1ed4428d | Sofia Papagiannaki | pithos.host$ snf-manage pithos-manage-accounts --merge-accounts --src-account=SPapagian@grnet.gr --dest-account=spapagian@grnet.gr |
525 | 1ed4428d | Sofia Papagiannaki | # (SPapagian@grnet.gr's contents will be merged into spapagian@grnet.gr, but SPapagian@grnet.gr account will still exist) |
526 | 1ed4428d | Sofia Papagiannaki | |
527 | 5c088197 | Constantinos Venetsanopoulos | - and then *delete* :: |
528 | 1ed4428d | Sofia Papagiannaki | |
529 | 1ed4428d | Sofia Papagiannaki | pithos.host$ snf-manage pithos-manage-accounts --delete-account=SPapagian@grnet.gr |
530 | 1ed4428d | Sofia Papagiannaki | # (only SPapagian@grnet.gr will be deleted not spapagian@grnet.gr) |
531 | 1ed4428d | Sofia Papagiannaki | |
532 | 5c088197 | Constantinos Venetsanopoulos | If you do *NOT* want to migrate files just run the second step and delete |
533 | 5c088197 | Constantinos Venetsanopoulos | the duplicate account. |
534 | 5c088197 | Constantinos Venetsanopoulos | |
535 | 5c088197 | Constantinos Venetsanopoulos | 6.3 Migrate Cyclades users (email case/uuid) |
536 | 7e38e668 | Georgios D. Tsoukalas | -------------------------------------------- |
537 | 7e38e668 | Georgios D. Tsoukalas | |
538 | 7e38e668 | Georgios D. Tsoukalas | :: |
539 | 7e38e668 | Georgios D. Tsoukalas | |
540 | f4c10ea9 | Kostas Papadimitriou | cyclades.host$ snf-manage cyclades-astakos-migrate-013 --migrate-users |
541 | 7e38e668 | Georgios D. Tsoukalas | |
542 | 7e38e668 | Georgios D. Tsoukalas | - if invalid usernames are found, verify that they do not exist in astakos:: |
543 | 7e38e668 | Georgios D. Tsoukalas | |
544 | 7e38e668 | Georgios D. Tsoukalas | astakos.host$ snf-manage user-list |
545 | 7e38e668 | Georgios D. Tsoukalas | |
546 | 7e38e668 | Georgios D. Tsoukalas | - if no user exists:: |
547 | 7e38e668 | Georgios D. Tsoukalas | |
548 | f4c10ea9 | Kostas Papadimitriou | cyclades.host$ snf-manage cyclades-astakos-migrate-013 --delete-user=<userid> |
549 | 7e38e668 | Georgios D. Tsoukalas | |
550 | 5c088197 | Constantinos Venetsanopoulos | Finally, if you have set manually quotas for specific users inside |
551 | 5c088197 | Constantinos Venetsanopoulos | ``/etc/synnefo/20-snf-cyclades-app-api.conf`` (in ``VMS_USER_QUOTA``, |
552 | 5c088197 | Constantinos Venetsanopoulos | ``NETWORKS_USER_QUOTA`` make sure to update them so that: |
553 | 5c088197 | Constantinos Venetsanopoulos | |
554 | 5c088197 | Constantinos Venetsanopoulos | 1. There are no double entries wrt case sensitivity |
555 | 5c088197 | Constantinos Venetsanopoulos | 2. Replace all user email addresses with the corresponding UUIDs |
556 | 5c088197 | Constantinos Venetsanopoulos | |
557 | 5c088197 | Constantinos Venetsanopoulos | To find the UUIDs for step 2 run on the Astakos host :: |
558 | 5c088197 | Constantinos Venetsanopoulos | |
559 | 5c088197 | Constantinos Venetsanopoulos | # snf-manage user-list |
560 | 5c088197 | Constantinos Venetsanopoulos | |
561 | 5c088197 | Constantinos Venetsanopoulos | 6.4 Migrate Pithos user names |
562 | 7e38e668 | Georgios D. Tsoukalas | ----------------------------- |
563 | 7e38e668 | Georgios D. Tsoukalas | |
564 | 7e38e668 | Georgios D. Tsoukalas | Check if alembic has not been initialized :: |
565 | 7e38e668 | Georgios D. Tsoukalas | |
566 | e6df0da5 | Kostas Papadimitriou | pithos.host$ pithos-migrate current |
567 | 7e38e668 | Georgios D. Tsoukalas | |
568 | 7e38e668 | Georgios D. Tsoukalas | - If alembic current is None (e.g. okeanos.io) :: |
569 | 7e38e668 | Georgios D. Tsoukalas | |
570 | e6df0da5 | Kostas Papadimitriou | pithos.host$ pithos-migrate stamp 3dd56e750a3 |
571 | 7e38e668 | Georgios D. Tsoukalas | |
572 | eeb3f40b | Georgios D. Tsoukalas | Then, migrate pithos account name to uuid:: |
573 | 7e38e668 | Georgios D. Tsoukalas | |
574 | e6df0da5 | Kostas Papadimitriou | pithos.host$ pithos-migrate upgrade head |
575 | 7e38e668 | Georgios D. Tsoukalas | |
576 | eeb3f40b | Georgios D. Tsoukalas | Finally, set this setting to ``True``:: |
577 | eeb3f40b | Georgios D. Tsoukalas | |
578 | eeb3f40b | Georgios D. Tsoukalas | PITHOS_USE_QUOTAHOLDER = True |
579 | eeb3f40b | Georgios D. Tsoukalas | |
580 | eeb3f40b | Georgios D. Tsoukalas | |
581 | 7e38e668 | Georgios D. Tsoukalas | 7. Migrate old quota limits |
582 | 7e38e668 | Georgios D. Tsoukalas | =========================== |
583 | 7e38e668 | Georgios D. Tsoukalas | |
584 | 7e38e668 | Georgios D. Tsoukalas | 7.1 Migrate Pithos quota limits to Astakos |
585 | 7e38e668 | Georgios D. Tsoukalas | ------------------------------------------ |
586 | 7e38e668 | Georgios D. Tsoukalas | |
587 | 7e38e668 | Georgios D. Tsoukalas | Migrate from pithos native to astakos/quotaholder. |
588 | 7e38e668 | Georgios D. Tsoukalas | This requires a file to be transfered from Cyclades to Astakos:: |
589 | 7e38e668 | Georgios D. Tsoukalas | |
590 | 457fe4fc | Georgios D. Tsoukalas | pithos.host$ snf-manage pithos-export-quota --location=pithos-quota.txt |
591 | eeb3f40b | Georgios D. Tsoukalas | pithos.host$ scp pithos-quota.txt astakos.host: |
592 | 457fe4fc | Georgios D. Tsoukalas | astakos.host$ snf-manage user-set-initial-quota pithos-quota.txt |
593 | 457fe4fc | Georgios D. Tsoukalas | |
594 | 457fe4fc | Georgios D. Tsoukalas | .. _export-quota-note: |
595 | 457fe4fc | Georgios D. Tsoukalas | |
596 | 457fe4fc | Georgios D. Tsoukalas | .. note:: |
597 | 7e38e668 | Georgios D. Tsoukalas | |
598 | 457fe4fc | Georgios D. Tsoukalas | `pithos-export-quota` will only export quotas that are not equal to the |
599 | 457fe4fc | Georgios D. Tsoukalas | defaults in Pithos. Therefore, it is possible to both change or maintain |
600 | 457fe4fc | Georgios D. Tsoukalas | the default quotas across the migration. To maintain quotas the new default |
601 | 457fe4fc | Georgios D. Tsoukalas | pithos+.diskpace limit in Astakos must be equal to the (old) default quota |
602 | 457fe4fc | Georgios D. Tsoukalas | limit in Pithos. Change either one of them make them equal. |
603 | 7e38e668 | Georgios D. Tsoukalas | |
604 | 457fe4fc | Georgios D. Tsoukalas | see :ref:`astakos-load-resources` on how to set the (new) default quotas in Astakos. |
605 | 7e38e668 | Georgios D. Tsoukalas | |
606 | 7e38e668 | Georgios D. Tsoukalas | 7.2 Migrate Cyclades quota limits to Astakos |
607 | 7e38e668 | Georgios D. Tsoukalas | -------------------------------------------- |
608 | 7e38e668 | Georgios D. Tsoukalas | |
609 | 7e38e668 | Georgios D. Tsoukalas | :: |
610 | 7e38e668 | Georgios D. Tsoukalas | |
611 | 457fe4fc | Georgios D. Tsoukalas | cyclades.host$ snf-manage cyclades-export-quota --location=cyclades-quota.txt |
612 | eeb3f40b | Georgios D. Tsoukalas | cyclades.host$ scp cyclades-quota.txt astakos.host: |
613 | 457fe4fc | Georgios D. Tsoukalas | astakos.host$ snf-manage user-set-initial-quota cyclades-quota.txt |
614 | 457fe4fc | Georgios D. Tsoukalas | |
615 | 457fe4fc | Georgios D. Tsoukalas | `cyclades-export-quota` will only export quotas that are not equal to the defaults. |
616 | 457fe4fc | Georgios D. Tsoukalas | See :ref:`note above <export-quota-note>`. |
617 | 7e38e668 | Georgios D. Tsoukalas | |
618 | 7e38e668 | Georgios D. Tsoukalas | 8. Enforce the new quota limits migrated to Astakos |
619 | 7e38e668 | Georgios D. Tsoukalas | =================================================== |
620 | 7e38e668 | Georgios D. Tsoukalas | The following should report all users not having quota limits set |
621 | 457fe4fc | Georgios D. Tsoukalas | because the effective quota database has not been initialized yet. :: |
622 | 7e38e668 | Georgios D. Tsoukalas | |
623 | 7e38e668 | Georgios D. Tsoukalas | astakos.host$ snf-manage astakos-quota --verify |
624 | 7e38e668 | Georgios D. Tsoukalas | |
625 | 7e38e668 | Georgios D. Tsoukalas | Initialize the effective quota database:: |
626 | 7e38e668 | Georgios D. Tsoukalas | |
627 | 7e38e668 | Georgios D. Tsoukalas | astakos.host$ snf-manage astakos-quota --sync |
628 | 7e38e668 | Georgios D. Tsoukalas | |
629 | 7e38e668 | Georgios D. Tsoukalas | This procedure may be used to verify and re-synchronize the effective quota |
630 | 7e38e668 | Georgios D. Tsoukalas | database with the quota limits that are derived from policies in Astakos |
631 | 7e38e668 | Georgios D. Tsoukalas | (initial quotas, project memberships, etc.) |
632 | 7e38e668 | Georgios D. Tsoukalas | |
633 | 7e38e668 | Georgios D. Tsoukalas | 9. Initialize resource usage |
634 | 7e38e668 | Georgios D. Tsoukalas | ============================ |
635 | 7e38e668 | Georgios D. Tsoukalas | |
636 | 457fe4fc | Georgios D. Tsoukalas | The effective quota database (quotaholder) has just been initialized and knows |
637 | 457fe4fc | Georgios D. Tsoukalas | nothing of the current resource usage. Therefore, each service must send it in. |
638 | 7e38e668 | Georgios D. Tsoukalas | |
639 | 457fe4fc | Georgios D. Tsoukalas | 9.1 Initialize Pithos resource usage |
640 | 457fe4fc | Georgios D. Tsoukalas | ------------------------------------ |
641 | 7e38e668 | Georgios D. Tsoukalas | |
642 | 7e38e668 | Georgios D. Tsoukalas | :: |
643 | 7e38e668 | Georgios D. Tsoukalas | |
644 | 9a72e003 | Giorgos Korfiatis | pithos.host$ snf-manage pithos-usage --reset |
645 | 7e38e668 | Georgios D. Tsoukalas | |
646 | 457fe4fc | Georgios D. Tsoukalas | 9.2 Initialize Cyclades resource usage |
647 | 457fe4fc | Georgios D. Tsoukalas | -------------------------------------- |
648 | 7e38e668 | Georgios D. Tsoukalas | |
649 | 7e38e668 | Georgios D. Tsoukalas | :: |
650 | 7e38e668 | Georgios D. Tsoukalas | |
651 | 457fe4fc | Georgios D. Tsoukalas | cyclades.host$ snf-manage cyclades-reset-usage |
652 | 7e38e668 | Georgios D. Tsoukalas | |
653 | 7e38e668 | Georgios D. Tsoukalas | 10. Install periodic project maintainance checks |
654 | 7e38e668 | Georgios D. Tsoukalas | ================================================ |
655 | 7e38e668 | Georgios D. Tsoukalas | In order to detect and effect project expiration, |
656 | 7e38e668 | Georgios D. Tsoukalas | a management command has to be run periodically |
657 | 7e38e668 | Georgios D. Tsoukalas | (depending on the required granularity, e.g. once a day/hour):: |
658 | 7e38e668 | Georgios D. Tsoukalas | |
659 | 7e38e668 | Georgios D. Tsoukalas | astakos.host$ snf-manage project-control --terminate-expired |
660 | 7e38e668 | Georgios D. Tsoukalas | |
661 | 457fe4fc | Georgios D. Tsoukalas | A list of expired projects can be extracted with:: |
662 | 7e38e668 | Georgios D. Tsoukalas | |
663 | 9a72e003 | Giorgos Korfiatis | astakos.host$ snf-manage project-control --check-expired |
664 | 7e38e668 | Georgios D. Tsoukalas | |
665 | eeb3f40b | Georgios D. Tsoukalas | |
666 | eeb3f40b | Georgios D. Tsoukalas | 11. Restart all services |
667 | eeb3f40b | Georgios D. Tsoukalas | ======================== |
668 | eeb3f40b | Georgios D. Tsoukalas | |
669 | eeb3f40b | Georgios D. Tsoukalas | Start (or restart, if running) all Synnefo services on all hosts. |
670 | eeb3f40b | Georgios D. Tsoukalas | |
671 | eeb3f40b | Georgios D. Tsoukalas | :: |
672 | eeb3f40b | Georgios D. Tsoukalas | |
673 | eeb3f40b | Georgios D. Tsoukalas | # service gunicorn restart |
674 | eeb3f40b | Georgios D. Tsoukalas | # service snf-dispatcher restart |
675 | eeb3f40b | Georgios D. Tsoukalas | # etc. |