root / docs / upgrade-0.13.rst @ 3910c691
History | View | Annotate | Download (18.7 kB)
1 |
Upgrade to Synnefo v0.13 |
---|---|
2 |
^^^^^^^^^^^^^^^^^^^^^^^^ |
3 |
|
4 |
The bulk of the upgrade to v0.13 is about user and quota migrations. |
5 |
In summary, the migration process has 3 steps: |
6 |
|
7 |
1. Run some commands and scripts to diagnose and extract some migration data |
8 |
while the OLD code is running, and BEFORE any changes are made. |
9 |
|
10 |
2. Bring down services, upgrade packages, configure services, and perform |
11 |
django database migrations. These migrations do not need any interaction |
12 |
between services. |
13 |
|
14 |
3. Initialize the Astakos quota system and bring the Astakos service up, since |
15 |
it will be needed during a second-phase of UUID and quota migrations, that |
16 |
also uses data extracted from step 1. |
17 |
|
18 |
|
19 |
.. warning:: |
20 |
|
21 |
It is strongly suggested that you keep separate database backups |
22 |
for each service after the completion of each of step. |
23 |
|
24 |
1. Bring web services down, backup databases |
25 |
============================================ |
26 |
|
27 |
1. All web services must be brought down so that the database maintains a |
28 |
predictable and consistent state during the migration process. |
29 |
|
30 |
2. Backup databases for recovery to a pre-migration state. |
31 |
|
32 |
3. Keep the database servers running during the migration process |
33 |
|
34 |
|
35 |
2. Prepare astakos user migration to case insensitive emails |
36 |
============================================================ |
37 |
|
38 |
It is possible that two or more users have been registered with emails that |
39 |
differ only in the case of its letters. There can only be one of those |
40 |
accounts after the migration, so the rest must be deleted. |
41 |
|
42 |
Note that even if the users are deleted in Astakos, there still are duplicate |
43 |
entries in Cyclades and Pithos. For each service we need to reduce those |
44 |
multiple accounts into one, either merging them together, or deleting and |
45 |
discarding data from all but one. |
46 |
|
47 |
.. _find_duplicate_emails: |
48 |
|
49 |
2.1 Find duplicate email entries in Astakos |
50 |
------------------------------------------- |
51 |
(script: ``find_astakos_users_with_conflicting_emails.py``):: |
52 |
|
53 |
astakos-host$ cat << EOF > find_astakos_users_with_conflicting_emails.py |
54 |
#!/usr/bin/env python |
55 |
import os |
56 |
import sys |
57 |
|
58 |
os.environ['DJANGO_SETTINGS_MODULE'] = 'synnefo.settings' |
59 |
|
60 |
import astakos |
61 |
from astakos.im.models import AstakosUser as A |
62 |
|
63 |
|
64 |
def user_filter(user): |
65 |
return A.objects.filter(email__iexact=user.email).count() > 1 |
66 |
|
67 |
all_users = list(A.objects.all()) |
68 |
userlist = [(str(u.pk) + ': ' + str(u.email) + ' (' + str(u.is_active) + ', ' + |
69 |
str(u.date_joined) + ')') for u in filter(user_filter, all_users)] |
70 |
|
71 |
sys.stderr.write("id email (is_active, creation date)\n") |
72 |
print "\n".join(userlist) |
73 |
EOF |
74 |
|
75 |
astakos-host$ python ./find_astakos_users_with_conflicting_emails.py |
76 |
|
77 |
.. _remove_astakos_duplicate: |
78 |
|
79 |
2.1 Remove duplicate users in Astakos by their id |
80 |
------------------------------------------------- |
81 |
(script: ``delete_astakos_users.py``):: |
82 |
|
83 |
astakos-host$ cat << EOF > delete_astakos_users.py |
84 |
#!/usr/bin/env python |
85 |
|
86 |
import os |
87 |
import sys |
88 |
from time import sleep |
89 |
|
90 |
os.environ['DJANGO_SETTINGS_MODULE'] = 'synnefo.settings' |
91 |
|
92 |
import astakos |
93 |
from astakos.im.models import AstakosUser as A |
94 |
|
95 |
|
96 |
def user_filter(user): |
97 |
return A.objects.filter(email__iexact=user.email).count() > 1 |
98 |
|
99 |
argv = sys.argv |
100 |
argc = len(sys.argv) |
101 |
|
102 |
if argc < 2: |
103 |
print "Usage: ./delete_astakos_users.py <id>..." |
104 |
raise SystemExit() |
105 |
|
106 |
id_list = [int(x) for x in argv[1:]] |
107 |
|
108 |
print "" |
109 |
print "This will permanently delete the following users:\n" |
110 |
print "id email (is_active, creation date)" |
111 |
print "-- --------------------------------" |
112 |
|
113 |
users = A.objects.filter(id__in=id_list) |
114 |
for user in users: |
115 |
print "%s: %s (%s, %s)" % (user.id, user.email, user.is_active, |
116 |
user.date_joined) |
117 |
|
118 |
print "\nExecute? (yes/no): ", |
119 |
line = raw_input().rstrip() |
120 |
if line != 'yes': |
121 |
print "\nCancelled" |
122 |
raise SystemExit() |
123 |
|
124 |
print "\nConfirmed." |
125 |
sleep(2) |
126 |
for user in users: |
127 |
print "deleting %s: %s" % (user.id, user.email) |
128 |
user.delete() |
129 |
|
130 |
EOF |
131 |
|
132 |
astakos-host$ python ./delete_astakos_users.py 30 40 |
133 |
|
134 |
.. warning:: |
135 |
|
136 |
After deleting users with the ``delete_astakos_users.py`` script, |
137 |
check again with ``find_astakos_users_with_conflicting_emails.py`` |
138 |
(as in :ref:`find_duplicate_emails`) |
139 |
to make sure that no duplicate email conflicts remain. |
140 |
|
141 |
|
142 |
3. Upgrade Synnefo and configure settings |
143 |
========================================= |
144 |
|
145 |
3.1 Install the new versions of packages |
146 |
---------------------------------------- |
147 |
|
148 |
:: |
149 |
|
150 |
astakos.host$ apt-get install \ |
151 |
snf-common \ |
152 |
snf-webproject \ |
153 |
snf-quotaholder-app \ |
154 |
snf-astakos-app \ |
155 |
kamaki \ |
156 |
|
157 |
|
158 |
cyclades.host$ apt-get install \ |
159 |
snf-common \ |
160 |
snf-webproject |
161 |
snf-pithos-backend \ |
162 |
snf-cyclades-app \ |
163 |
kamaki \ |
164 |
|
165 |
|
166 |
pithos.host$ apt-get install \ |
167 |
snf-common \ |
168 |
snf-webproject |
169 |
snf-pithos-backend \ |
170 |
snf-pithos-app \ |
171 |
snf-pithos-webclient \ |
172 |
kamaki \ |
173 |
|
174 |
|
175 |
ganeti.node$ apt-get install \ |
176 |
snf-cyclades-gtools \ |
177 |
snf-pithos-backend \ |
178 |
kamaki \ |
179 |
|
180 |
.. note:: |
181 |
|
182 |
If you get questioned about stale content types during the |
183 |
migration process, answer ``no`` and let the migration finish. |
184 |
|
185 |
|
186 |
3.2 Sync and migrate Django DB |
187 |
------------------------------ |
188 |
|
189 |
.. note:: |
190 |
|
191 |
If you are asked about stale content types during the migration process, |
192 |
answer 'no' and let the migration finish. |
193 |
|
194 |
:: |
195 |
|
196 |
astakos-host$ snf-manage syncdb |
197 |
astakos-host$ snf-manage migrate |
198 |
|
199 |
cyclades-host$ snf-manage syncdb |
200 |
cyclades-host$ snf-manage migrate |
201 |
|
202 |
.. note:: |
203 |
|
204 |
After the migration, Astakos has created uuids for all users, |
205 |
and has set the uuid as the public identifier of a user. |
206 |
This uuid is to be used both at other services (Cyclades, Pithos) |
207 |
and at the clientside (kamaki client settings). |
208 |
|
209 |
Duplicate-email users have been deleted earlier in |
210 |
:ref:`remove_astakos_duplicate` |
211 |
|
212 |
3.3 Setup quota settings for all services |
213 |
----------------------------------------- |
214 |
|
215 |
Generally: |
216 |
|
217 |
:: |
218 |
|
219 |
# Service Setting Value |
220 |
# quotaholder: QUOTAHOLDER_TOKEN = <random string> |
221 |
|
222 |
# astakos: ASTAKOS_QUOTAHOLDER_TOKEN = <the same random string> |
223 |
# astakos: ASTAKOS_QUOTAHOLDER_URL = https://quotaholder.host/quotaholder/v |
224 |
|
225 |
# cyclades: CYCLADES_QUOTAHOLDER_TOKEN = <the same random string> |
226 |
# cyclades: CYCLADES_QUOTAHOLDER_URL = http://quotaholder.host/quotaholder/v |
227 |
# cyclades: CYCLADES_USE_QUOTAHOLDER = True |
228 |
|
229 |
|
230 |
# pithos: PITHOS_QUOTAHOLDER_TOKEN = <the same random string> |
231 |
# pithos: PITHOS_QUOTAHOLDER_URL = http://quotaholder.host/quotaholder/v |
232 |
# All services must match the quotaholder token and url configured for quotaholder. |
233 |
|
234 |
Specifically: |
235 |
|
236 |
On the Astakos host, edit ``/etc/synnefo/20-snf-astakos-app-settings.conf``: |
237 |
|
238 |
:: |
239 |
|
240 |
QUOTAHOLDER_TOKEN = 'aExampleTokenJbFm12w' |
241 |
ASTAKOS_QUOTAHOLDER_TOKEN = 'aExampleTokenJbFm12w' |
242 |
ASTAKOS_QUOTAHOLDER_URL = 'https://accounts.synnefo.local/quotaholder/v' |
243 |
|
244 |
On the Cyclades host, edit ``/etc/synnefo/20-snf-cyclades-app-quotas.conf``: |
245 |
|
246 |
:: |
247 |
|
248 |
CYCLADES_USE_QUOTAHOLDER = True |
249 |
CYCLADES_QUOTAHOLDER_URL = 'https://accounts.synnefo.local/quotaholder/v' |
250 |
CYCLADES_QUOTAHOLDER_TOKEN = 'aExampleTokenJbFm12w' |
251 |
|
252 |
On the Pithos host, edit ``/etc/synnefo/20-snf-pithos-app-settings.conf``: |
253 |
|
254 |
:: |
255 |
|
256 |
PITHOS_QUOTAHOLDER_URL = 'https://accounts.synnefo.local/quotaholder/v' |
257 |
PITHOS_QUOTAHOLDER_TOKEN = 'aExampleTokenJbFm12w' |
258 |
|
259 |
3.4 Setup astakos |
260 |
----------------- |
261 |
|
262 |
- **Remove** this redirection from astakos front-end web server :: |
263 |
|
264 |
RewriteRule ^/login(.*) /im/login/redirect$1 [PT,NE] |
265 |
|
266 |
(see `<http://docs.dev.grnet.gr/synnefo/latest/quick-install-admin-guide.html#apache2-setup>`_) |
267 |
|
268 |
- Enable users to change their contact email. Edit |
269 |
``/etc/synnefo/20-snf-astakos-app-settings.conf`` :: |
270 |
|
271 |
ASTAKOS_EMAILCHANGE_ENABLED = True |
272 |
|
273 |
3.5 Setup Cyclades |
274 |
------------------ |
275 |
|
276 |
- Run on the Astakos host :: |
277 |
|
278 |
# snf-manage service-list |
279 |
|
280 |
- Set the Cyclades service token in ``/etc/synnefo/20-snf-cyclades-app-api.conf`` :: |
281 |
|
282 |
CYCLADES_ASTAKOS_SERVICE_TOKEN = 'asfasdf_CycladesServiceToken_iknl' |
283 |
|
284 |
- Since version 0.13, Synnefo uses **VMAPI** in order to prevent sensitive data |
285 |
needed by 'snf-image' to be stored in Ganeti configuration (e.g. VM |
286 |
password). This is achieved by storing all sensitive information to a CACHE |
287 |
backend and exporting it via VMAPI. The cache entries are invalidated after |
288 |
the first request. Synnefo uses **memcached** as a django cache backend. |
289 |
To install, run on the Cyclades host:: |
290 |
|
291 |
apt-get install memcached |
292 |
apt-get install python-memcache |
293 |
|
294 |
You will also need to configure Cyclades to use the memcached cache backend. |
295 |
Namely, you need to set IP address and port of the memcached daemon, and the |
296 |
default timeout (seconds tha value is stored in the cache). Edit |
297 |
``/etc/synnefo/20-snf-cyclades-app-vmapi.conf`` :: |
298 |
|
299 |
VMAPI_CACHE_BACKEND = "memcached://127.0.0.1:11211/?timeout=3600" |
300 |
|
301 |
|
302 |
Finally, set the BASE_URL for the VMAPI, which is actually the base URL of |
303 |
Cyclades, again in ``/etc/synnefo/20-snf-cyclades-app-vmapi.conf``. Make sure |
304 |
the domain is exaclty the same, so that no re-directs happen :: |
305 |
|
306 |
VMAPI_BASE_URL = "https://cyclades.synnefo.local" |
307 |
|
308 |
.. note:: |
309 |
|
310 |
- These settings are needed in all Cyclades workers. |
311 |
|
312 |
- VMAPI_CACHE_BACKEND just overrides django's CACHE_BACKEND setting |
313 |
|
314 |
- memcached must be reachable from all Cyclades workers. |
315 |
|
316 |
- For more information about configuring django to use memcached: |
317 |
https://docs.djangoproject.com/en/1.2/topics/cache |
318 |
|
319 |
3.6 Setup Pithos |
320 |
---------------- |
321 |
|
322 |
- Pithos forwards user catalog services to Astakos so that web clients may |
323 |
access them for uuid-displayname translations. Edit on the Pithos host |
324 |
``/etc/synnefo/20-snf-pithos-app-settings.conf`` :: |
325 |
|
326 |
PITHOS_USER_CATALOG_URL = https://accounts.synnefo.local/user_catalogs/ |
327 |
PITHOS_USER_FEEDBACK_URL = https://accounts.synnefo.local/feedback/ |
328 |
PITHOS_USER_LOGIN_URL = https://accounts.synnefo.local/login/ |
329 |
#PITHOS_PROXY_USER_SERVICES = True # Set False if astakos & pithos are on the same host |
330 |
|
331 |
|
332 |
4. Start astakos and quota services |
333 |
=================================== |
334 |
Start the webserver and gunicorn on the Astakos host. E.g.:: |
335 |
|
336 |
# service apache2 start |
337 |
# service gunicorn start |
338 |
|
339 |
.. warning:: |
340 |
|
341 |
To ensure consistency, prevent public access to astakos during migrations. |
342 |
This can be done via firewall or webserver access control. |
343 |
|
344 |
.. _astakos-load-resources: |
345 |
|
346 |
5. Load resource definitions into Astakos |
347 |
========================================= |
348 |
|
349 |
First, set the corresponding values on the following dict in |
350 |
``/etc/synnefo/20-snf-astakos-app-settings.conf`` :: |
351 |
|
352 |
# Set the cloud service properties |
353 |
ASTAKOS_SERVICES = { |
354 |
'cyclades': { |
355 |
#This can also be set from a management command |
356 |
'url': 'https://cyclades.host/ui/', |
357 |
'order': 0, |
358 |
'resources': [{ |
359 |
'name':'disk', |
360 |
'group':'compute', |
361 |
'uplimit':300*1024*1024*1024, |
362 |
'unit':'bytes', |
363 |
'desc': 'Virtual machine disk size' |
364 |
},{ |
365 |
'name':'cpu', |
366 |
'group':'compute', |
367 |
'uplimit':24, |
368 |
'desc': 'Number of virtual machine processors' |
369 |
},{ |
370 |
'name':'ram', |
371 |
'group':'compute', |
372 |
'uplimit':40*1024*1024*1024, |
373 |
'unit':'bytes', |
374 |
'desc': 'Virtual machines' |
375 |
},{ |
376 |
'name':'vm', |
377 |
'group':'compute', |
378 |
'uplimit':5, |
379 |
'desc': 'Number of virtual machines' |
380 |
},{ |
381 |
'name':'network.private', |
382 |
'group':'network', |
383 |
'uplimit':5, |
384 |
'desc': 'Private networks' |
385 |
} |
386 |
] |
387 |
}, |
388 |
'pithos+': { |
389 |
'url': 'https://pithos.host/ui/', |
390 |
'order': 1, |
391 |
'resources':[{ |
392 |
'name':'diskspace', |
393 |
'group':'storage', |
394 |
'uplimit':20 * 1024 * 1024 * 1024, |
395 |
'unit':'bytes', |
396 |
'desc': 'Pithos account diskspace' |
397 |
}] |
398 |
} |
399 |
} |
400 |
|
401 |
Then, configure and load the available resources per service |
402 |
and associated default limits into Astakos. On the Astakos host run :: |
403 |
|
404 |
# snf-manage astakos-init --load-service-resources |
405 |
|
406 |
|
407 |
.. note:: |
408 |
|
409 |
Before v0.13, only `cyclades.vm`, `cyclades.network.private`, |
410 |
and `pithos+.diskspace` existed (not with this names, of course). |
411 |
However, limits to the new resources must also be set. |
412 |
|
413 |
If the intetion is to keep a resource unlimited, (counting on that VM |
414 |
creation will be limited by other resources' limit) it is best to calculate |
415 |
a value that is too large to be reached because of other limits (and |
416 |
available flavours), but not much larger than needed because this might |
417 |
confuse users who do not readily understand that multiple limits apply and |
418 |
flavors are limited. |
419 |
|
420 |
|
421 |
6. Migrate Services user names to uuids |
422 |
======================================= |
423 |
|
424 |
|
425 |
6.1 Double-check cyclades before user case/uuid migration |
426 |
--------------------------------------------------------- |
427 |
|
428 |
:: |
429 |
|
430 |
cyclades.host$ snf-manage cyclades-astakos-migrate-013 --validate |
431 |
|
432 |
Duplicate user found? |
433 |
|
434 |
- either *merge* (merge will merge all resources to one user):: |
435 |
|
436 |
cyclades.host$ snf-manage cyclades-astakos-migrate-013 --merge-user=kpap@grnet.gr |
437 |
|
438 |
- or *delete* :: |
439 |
|
440 |
cyclades.host$ snf-manage cyclades-astakos-migrate-013 --delete-user=KPap@grnet.gr |
441 |
# (only KPap will be deleted not kpap) |
442 |
|
443 |
6.2 Double-check pithos before user case/uuid migration |
444 |
--------------------------------------------------------- |
445 |
|
446 |
:: |
447 |
|
448 |
pithos.host$ snf-manage pithos-manage-accounts --list-duplicate |
449 |
|
450 |
Duplicate user found? |
451 |
|
452 |
If you want to migrate files first: |
453 |
|
454 |
- *merge* (merge will merge all resources to one user):: |
455 |
|
456 |
pithos.host$ snf-manage pithos-manage-accounts --merge-accounts --src-account=SPapagian@grnet.gr --dest-account=spapagian@grnet.gr |
457 |
# (SPapagian@grnet.gr's contents will be merged into spapagian@grnet.gr, but SPapagian@grnet.gr account will still exist) |
458 |
|
459 |
- and then *delete* :: |
460 |
|
461 |
pithos.host$ snf-manage pithos-manage-accounts --delete-account=SPapagian@grnet.gr |
462 |
# (only SPapagian@grnet.gr will be deleted not spapagian@grnet.gr) |
463 |
|
464 |
If you do *NOT* want to migrate files just run the second step and delete |
465 |
the duplicate account. |
466 |
|
467 |
6.3 Migrate Cyclades users (email case/uuid) |
468 |
-------------------------------------------- |
469 |
|
470 |
:: |
471 |
|
472 |
cyclades.host$ snf-manage cyclades-astakos-migrate-013 --migrate-users |
473 |
|
474 |
- if invalid usernames are found, verify that they do not exist in astakos:: |
475 |
|
476 |
astakos.host$ snf-manage user-list |
477 |
|
478 |
- if no user exists:: |
479 |
|
480 |
cyclades.host$ snf-manage cyclades-astakos-migrate-013 --delete-user=<userid> |
481 |
|
482 |
Finally, if you have set manually quotas for specific users inside |
483 |
``/etc/synnefo/20-snf-cyclades-app-api.conf`` (in ``VMS_USER_QUOTA``, |
484 |
``NETWORKS_USER_QUOTA`` make sure to update them so that: |
485 |
|
486 |
1. There are no double entries wrt case sensitivity |
487 |
2. Replace all user email addresses with the corresponding UUIDs |
488 |
|
489 |
To find the UUIDs for step 2 run on the Astakos host :: |
490 |
|
491 |
# snf-manage user-list |
492 |
|
493 |
6.4 Migrate Pithos user names |
494 |
----------------------------- |
495 |
|
496 |
Check if alembic has not been initialized :: |
497 |
|
498 |
pithos.host$ pithos-migrate current |
499 |
|
500 |
- If alembic current is None (e.g. okeanos.io) :: |
501 |
|
502 |
pithos.host$ pithos-migrate stamp 3dd56e750a3 |
503 |
|
504 |
Finally, migrate pithos account name to uuid:: |
505 |
|
506 |
pithos.host$ pithos-migrate upgrade head |
507 |
|
508 |
7. Migrate old quota limits |
509 |
=========================== |
510 |
|
511 |
7.1 Migrate Pithos quota limits to Astakos |
512 |
------------------------------------------ |
513 |
|
514 |
Migrate from pithos native to astakos/quotaholder. |
515 |
This requires a file to be transfered from Cyclades to Astakos:: |
516 |
|
517 |
pithos.host$ snf-manage pithos-export-quota --location=pithos-quota.txt |
518 |
pithos.host$ rsync -avP pithos-quota.txt astakos.host: |
519 |
astakos.host$ snf-manage user-set-initial-quota pithos-quota.txt |
520 |
|
521 |
.. _export-quota-note: |
522 |
|
523 |
.. note:: |
524 |
|
525 |
`pithos-export-quota` will only export quotas that are not equal to the |
526 |
defaults in Pithos. Therefore, it is possible to both change or maintain |
527 |
the default quotas across the migration. To maintain quotas the new default |
528 |
pithos+.diskpace limit in Astakos must be equal to the (old) default quota |
529 |
limit in Pithos. Change either one of them make them equal. |
530 |
|
531 |
see :ref:`astakos-load-resources` on how to set the (new) default quotas in Astakos. |
532 |
|
533 |
7.2 Migrate Cyclades quota limits to Astakos |
534 |
-------------------------------------------- |
535 |
|
536 |
:: |
537 |
|
538 |
cyclades.host$ snf-manage cyclades-export-quota --location=cyclades-quota.txt |
539 |
cyclades.host$ rsync -avP cyclades-quota.txt astakos.host: |
540 |
astakos.host$ snf-manage user-set-initial-quota cyclades-quota.txt |
541 |
|
542 |
`cyclades-export-quota` will only export quotas that are not equal to the defaults. |
543 |
See :ref:`note above <export-quota-note>`. |
544 |
|
545 |
8. Enforce the new quota limits migrated to Astakos |
546 |
=================================================== |
547 |
The following should report all users not having quota limits set |
548 |
because the effective quota database has not been initialized yet. :: |
549 |
|
550 |
astakos.host$ snf-manage astakos-quota --verify |
551 |
|
552 |
Initialize the effective quota database:: |
553 |
|
554 |
astakos.host$ snf-manage astakos-quota --sync |
555 |
|
556 |
This procedure may be used to verify and re-synchronize the effective quota |
557 |
database with the quota limits that are derived from policies in Astakos |
558 |
(initial quotas, project memberships, etc.) |
559 |
|
560 |
9. Initialize resource usage |
561 |
============================ |
562 |
|
563 |
The effective quota database (quotaholder) has just been initialized and knows |
564 |
nothing of the current resource usage. Therefore, each service must send it in. |
565 |
|
566 |
9.1 Initialize Pithos resource usage |
567 |
------------------------------------ |
568 |
|
569 |
:: |
570 |
|
571 |
pithos.host$ snf-manage pithos-reset-usage |
572 |
|
573 |
9.2 Initialize Cyclades resource usage |
574 |
-------------------------------------- |
575 |
|
576 |
:: |
577 |
|
578 |
cyclades.host$ snf-manage cyclades-reset-usage |
579 |
|
580 |
10. Install periodic project maintainance checks |
581 |
================================================ |
582 |
In order to detect and effect project expiration, |
583 |
a management command has to be run periodically |
584 |
(depending on the required granularity, e.g. once a day/hour):: |
585 |
|
586 |
astakos.host$ snf-manage project-control --terminate-expired |
587 |
|
588 |
A list of expired projects can be extracted with:: |
589 |
|
590 |
astakos.host$ snf-manage project-control --list-expired |
591 |
|