Statistics
| Branch: | Tag: | Revision:

root / docs / design / resource-pool-projects.rst @ 4398adc9

History | View | Annotate | Download (19.6 kB)

1
Resource-pool projects
2
^^^^^^^^^^^^^^^^^^^^^^
3

    
4
This document describes the current state of the quota and projects system,
5
and proposes a new design for projects that would function as resource
6
pools. It sketches implementation details and migration concerns.
7

    
8
Current state and shortcomings
9
==============================
10

    
11
Each Synnefo user is granted quota for several resources. These quota
12
originate from two different sources: the system and projects. By default
13
a user holds so-called base quota granted by the system upon activation;
14
base quota can be customized per user. When a user joins a project,
15
resources offered by the project add up to the existing quota, increasing
16
the total amount of resources one can reserve.
17

    
18
This design fails to associate an actual (reserved) resource (e.g. VM) with
19
a particular project. There is no way to tell which project a resource
20
originates from and is thus not possible to employ any targeted policy when
21
a user leaves a project, such as reclaiming the granted resource. It is also
22
not possible to employ more advanced access control on resources, such as
23
sharing VMs among members of a project.
24

    
25
Proposed changes
26
================
27

    
28
We will alter project semantics so that a project is viewed as a pool of
29
finite resources. Each project member can reserve a portion of these
30
resources up to a specified limit. Each actual resource (e.g. VM) is
31
associated with a particular project. Admission of a user to a project will
32
no more result in increasing the user's existing overall quota, but in
33
defining new project-specific quota for the user.
34

    
35
A project defines a pair of limits for each resource that it grants (e.g.
36
cyclades.vm): project-level limit and member-level limit; The former is the
37
total amount of a resource that this project can grant; the latter is the
38
maximum amount that an individual user (project member) can reserve and
39
cannot exceed the former. A limit on the number of members allowed is still
40
enforced.
41

    
42
Projects will be the sole source of resources. Current base quota offered to
43
users by the system will be expressed in terms of special-purpose *base*
44
projects. Due to the central role that projects now acquire, we will alter
45
the project schema to facilitate project creation and modification without
46
the extra overhead of submitting and approving applications.
47

    
48
Implementation details
49
======================
50

    
51
Project-related quota holdings
52
------------------------------
53

    
54
The Quotaholder is responsible to record all resource allocations and
55
deallocations, and enforce the limits. It keeps counters of the following
56
structure:
57
 * resource: the resource name (e.g. cyclades.vm)
58
 * holder: the entity holding the resource (user or project)
59
 * source: the origin of the resource; a user-holder reserves from a
60
   project, a project is a top-level entity and reserves from nowhere (None)
61
 * limit: maximum allowed allocation (an integer)
62
 * usage: current allocation (an integer)
63

    
64
[Due to the transactional nature of the mechanism, there are actually two
65
usage fields (usage_min and usage_max). Details are beyond the scope of
66
this document.]
67

    
68
Creation of a new project triggers the creation of counters like::
69

    
70
  resource      holder              source   limit   usage
71
  ------------|-------------------|--------|-------|------
72
  cyclades.vm   project:projectID   None     50      0
73

    
74
When a user is admitted in a project, counters are created like::
75

    
76
  resource      holder          source              limit   usage
77
  ------------|---------------|-------------------|-------|------
78
  cyclades.vm   user:userUUID   project:ProjectID   5       0
79

    
80
Note that the two types of holders (and sources) are made distinguishable with
81
a prefix: ``user:`` or ``project:``.
82

    
83
When a user leaves a project, the latter limit is set to zero. This results
84
in the project-specific user quota being over limit and prohibits any
85
further allocation that would increase this counter. When a project
86
is deactivated, the limit of both types of counters is set to zero.
87
No user can perform any allocation related to this project. However, the
88
holdings cannot be deleted as long as a non-zero usage is recorded.
89
Deallocation is always allowed as long as usage does not fall below zero.
90
Counters with zero usage and limit could by garbage collected by Astakos, if
91
needed.
92

    
93
Base projects
94
-------------
95

    
96
For reasons of uniformity, we replace the base quota mechanism with projects.
97
In a similar vein to OpenStack tenants, we define new user-specific *base*
98
projects to account for the base quota for each user. These projects should
99
be clearly associated with a single user, restrict join/leave actions and
100
specify the quota granted by the system. When a new user is created,
101
their base project will be automatically created and linked back to the user.
102
User activation will trigger project activation, granting the default resource
103
quota. Base projects will have no owner, marked thusly as `system' projects.
104
The administrator can, following the usual project logic, alter quota by
105
modifying the project. Users cannot apply for modification of their base
106
projects.
107

    
108
Projects will, from now on, be identified by a UUID. Base projects will
109
receive the same UUID as the user itself. ProjectID, which appears above in
110
the Quotaholder entries, refers to the project UUID.
111

    
112
Base quota will be expressed both in terms of a project-level and a
113
member-level limit. This will result in two operationally equivalent
114
Quotaholder counters, as in the following example. In the future, we could
115
admit third-party users to a user's base project; in that case, those
116
counters would differ.
117

    
118
::
119

    
120
  resource      holder         source         limit   usage
121
  ------------|--------------|--------------|-------|------
122
  cyclades.vm   project:uuid   None           5       1
123
  cyclades.vm   user:uuid      project:uuid   5       1
124

    
125
Private projects
126
----------------
127

    
128
Since the introduction of base projects will explode the number of total
129
projects, we will need to control their visibility. We add a new flag
130
*private* in project definitions. A private project can only be accessed by
131
its owner and members and not be advertized in the UI. Base projects are
132
marked as private.
133

    
134
Decouple projects from applications
135
-----------------------------------
136

    
137
Base projects do not fit well in the current project/application scheme,
138
because no user has applied for them. Moveover, we would like to easily
139
modify project properties, particularly quota limits, without the need to
140
apply for an application for each project and then approve it.
141

    
142
We will decouple projects from applications by incorporating the project
143
definition into the project object rather than relying on an application.
144
The system will directly make a new (base) project upon user creation and a
145
privileged user will be able to modify an existing project by directly
146
modifying it. An unprivileged user will still need to make an application.
147

    
148
The project model is adapted to reference the *last* application that is
149
related to the project, if any---projects automatically created by the
150
system reference no application. For an uninitialized project, this
151
denotes the original application through which the project was made. If
152
the application is denied or cancelled, the whole project is considered
153
deleted.
154

    
155
Applications as modifications
156
`````````````````````````````
157

    
158
Application for a new project is created in state ``pending`` and its
159
properties are copied into a new project object, which is in state
160
``uninitialized``. To preserve this equality, we disallow modifications of
161
uninitialized projects, either in-place or through an application. An
162
already activated project can be modified by submitting an application
163
containing just the desired changes. An application object stores the
164
specified changes and should remain read-only.
165

    
166
System default quota and resource registration
167
----------------------------------------------
168

    
169
Each resource registered in the system is assigned a default quota limit.
170
A newly-activated user is given these limits as their base quota. This is
171
till now done by copying the default limits as user's entries in
172
AstakosUserQuota. Default limits will from now on be copied into the base
173
project's resource definitions.
174

    
175
Conventional projects are created through a project application, which
176
may not specify limits for all resources registered in the system. In
177
fact, it may even be impossible to specify a resource, if it is set
178
``api_visible=False``. We have to somehow specify these limits. Defaulting
179
to zero is not appropriate: if we don't want to control a resource, we
180
would like it set to infinite. We thus need an extra skeleton, like the
181
one specifying the default base quota, in order to fill in missing limits
182
for conventional projects. It will be controled by a new option
183
``--project-default`` of command ``resource-modify``.
184

    
185
When a project is activated, either directly in the case of base projects
186
or through the approval of a project application, limits for resources not
187
specified are automatically completed by consulting the appropriate
188
skeleton.
189

    
190
Allocation of a new resource
191
----------------------------
192

    
193
When a service allocates a new resource, it should associate it both with a
194
user and a project. The commission issued to the Quotaholder should attempt
195
to update all related counters. For example, it should include the following
196
provisions::
197

    
198
  "provisions": [
199
          {
200
              "holder": "user:user-uuid",
201
              "source": "project:project-uuid",
202
              "resource": "cyclades.vm",
203
              "quantity": 1
204
          },
205
          {
206
              "holder": "project:project-uuid",
207
              "source": None,
208
              "resource": "cyclades.vm",
209
              "quantity": 1
210
          },
211
          {
212
              "holder": "user:user-uuid",
213
              "source": "project:project-uuid",
214
              "resource": "cyclades.cpu",
215
              "quantity": 2
216
          },
217
          {
218
              "holder": "project:project-uuid",
219
              "source": None,
220
              "resource": "cyclades.cpu",
221
              "quantity": 2
222
          }
223
  ]
224

    
225
If any of these provisions fails, i.e. either on the project-level limits or
226
the user-level ones, the whole commission fails.
227

    
228
The astakosclient call ``issue_one_commission`` will be adapted to abstract
229
away the need to write both the user-level and the project-level provisions.
230
The previous commission will be issued with::
231

    
232
  issue_one_commission(holder="user-uuid", source="project-uuid",
233
                       provisions={"cyclades.vm": 1, "cyclades.cpu": 2})
234

    
235
The service is responsible to record this resource-to-project association.
236
In Cyclades, each VM, floating IP, or other distinct resource should be
237
linked to a project. Pithos should link containers to projects.
238

    
239
Astakos will handle its own resource ``astakos.pending_app`` in a special
240
way: it will always be charged at the user's base project.
241

    
242
Resource reassignment
243
---------------------
244

    
245
The system will support reassigning a resource to a new project. One needs
246
to specify all related resource values. Astakosclient will provide a
247
convenience function ``issue_resource_reassignment`` to construct all needed
248
provisions. For instance, reassigning a VM with two CPUs can be done with::
249

    
250
  issue_resource_reassignment(holder="user-uuid",
251
                              from_source="from-uuid", to_source="to-uuid",
252
                              provisions={"cyclades.vm": 1, "cyclades.cpu": 2})
253

    
254
This will issue the following provisions to the Quotaholder::
255

    
256
  "provisions": [
257
          {
258
              "holder": "user:user-uuid",
259
              "source": "project:from-uuid",
260
              "resource": "cyclades.vm",
261
              "quantity": -1
262
          },
263
          {
264
              "holder": "project:from-uuid",
265
              "source": None,
266
              "resource": "cyclades.vm",
267
              "quantity": -1
268
          },
269
          {
270
              "holder": "user:user-uuid",
271
              "source": "project:from-uuid",
272
              "resource": "cyclades.cpu",
273
              "quantity": -2
274
          },
275
          {
276
              "holder": "project:from-uuid",
277
              "source": None,
278
              "resource": "cyclades.cpu",
279
              "quantity": -2
280
          },
281
          {
282
              "holder": "user:user-uuid",
283
              "source": "project:to-uuid",
284
              "resource": "cyclades.vm",
285
              "quantity": 1
286
          },
287
          {
288
              "holder": "project:to-uuid",
289
              "source": None,
290
              "resource": "cyclades.vm",
291
              "quantity": 1
292
          }
293
          {
294
              "holder": "user:user-uuid",
295
              "source": "project:to-uuid",
296
              "resource": "cyclades.cpu",
297
              "quantity": 2
298
          },
299
          {
300
              "holder": "project:to-uuid",
301
              "source": None,
302
              "resource": "cyclades.cpu",
303
              "quantity": 2
304
          }
305
  ]
306

    
307
API changes
308
-----------
309

    
310
API call ``GET /quotas`` is extended to incorporate project-level quota. The
311
response contains entries for all projects for which a user/project pair
312
exists in the quotaholder::
313

    
314
  {
315
      "project1-uuid": {
316
          "cyclades.ram": {
317
              "usage": 2147483648,
318
              "limit": 2147483648,
319
              "pending": 0,
320
              "project_usage": ...,
321
              "project_limit": ...,
322
              "project_pending": ...
323
          },
324
          "cyclades.vm": {
325
              ...
326
          }
327
      }
328
      "project2-uuid": {
329
          ...
330
      }
331
  }
332

    
333
An extra or differentiated call may be needed to retrieve the project quota
334
regardless of user::
335

    
336
  GET /quotas?mode=projects
337

    
338
  {
339
      "project-uuid": {
340
          "cyclades.ram": {
341
              "project_usage": 2147483648,
342
              "project_limit": 2147483648,
343
              "project_pending": 0
344
          }
345
          "cyclades.vm": {
346
              ...
347
          }
348
      }
349
  }
350

    
351
``GET /service_project_quotas`` will be used in a similar way as ``GET
352
/service_quotas`` to get the project-level quotas for resources associated
353
with the Synnefo component that makes the request.
354

    
355
All service API calls that create resources can specify the project where
356
they will be attributed.
357

    
358
In cyclades, ``POST /servers`` (likewise for networks and floating IPs) will
359
receive an extra argument ``project``. If it is missing, the user's base
360
project will be assumed. In calls detailing a resource (e.g., ``GET
361
/servers/<server_id>``), the field ``tenant_id`` will contain the
362
project id.
363

    
364
Moreover, extra calls will be needed for resource reassignment,
365
e.g::
366

    
367
  POST /servers/<server-id>/action
368

    
369
  {
370
      "reassign": {"project": <project-id>}
371
  }
372

    
373
In pithos, ``PUT`` and ``POST`` calls at the container level will accept an
374
extra optional policy ``project``. The former call assigns a newly created
375
container to a given project, the latter reassigns an existing container.
376
Field ``x-container-policy-project`` will be retrieved by a ``HEAD`` call at
377
the container level.
378

    
379
Changes in the projects API
380
```````````````````````````
381

    
382
``PUT /projects`` will be used to make a new project replacing ``POST``.
383

    
384
``POST /projects/<proj_id>`` now expects a dictionary with just the desired
385
changes, not a complete project definition. It is only allowed if the
386
project is already activated.
387

    
388
``GET /projects/<proj_id>`` changes to include a ``last_application`` field,
389
if applicable.
390

    
391
Application actions (approve, deny, dismiss, cancel) are integrated into
392
project actions and expect an extra ``app_id`` argument to specify the
393
application. Actions are allowed only on a project's last application;
394
the application id is required in order to avoid races.
395

    
396
The applications API is removed, incorporated into the projects API.
397

    
398
User interface
399
--------------
400

    
401
User quota will be presented per project, including the aggregate activity
402
of other project members: the Resource Usage page will include a drop-down
403
menu with all relevant projects. By default, user's base project will
404
be assumed. When choosing a project, usage for all resources will be
405
presented for the given project in the following style::
406

    
407
                        limit
408
    used                ^                    taken by others
409
  |::::::|..............|...........|::::::::::::::::::::::::::::::::::|
410
         ^              ^                                              ^
411
         usage          effective                                      project
412
                        limit                                          limit
413

    
414

    
415
                        limit
416
    used                ^          taken by others
417
  |::::::|........|:::::|::::::::::::::::::::::::::::::::::::::::::::::|
418
         ^        ^                                                    ^
419
         usage    effective                                            project
420
                  limit                                                limit
421

    
422
Text accompanying the bar could mention usage based on the effective limit,
423
e.g.: `usage` out of `effective limit` Virtual Machines. Likewise the shaded
424
`used` part of the bar could express the same ratio in percentage terms.
425

    
426
Given the above-mentioned response of the ``/quotas`` call, the effective
427
limit can be computed by::
428

    
429
  taken_by_others = project_usage - usage
430
  effective_limit = min(limit, project_limit - taken_by_others)
431

    
432
Projects show up in a number of service-specific user interactions, too.
433
When creating a Cyclades VM, the flavor-choosing window should first ask
434
for the project where the VM will be charged before showing the
435
available resource combinations. Likewise, creating a new container in
436
Pithos will prompt for picking a project to associate with.
437

    
438
Resource presentation (e.g. Cyclades VMs) will also mention the associated
439
project and provide an action to reassign the resource to a different
440
project.
441

    
442
Command-line interface
443
----------------------
444

    
445
Quota can be queried per user or project::
446

    
447
  # snf-manage user-show <id> --quota
448

    
449
  project  resource    limit  effective_limit usage
450
  -------------------------------------------------
451
  uuid     cyclades.vm 10     9               5
452

    
453
  # snf-manage project-show <id> --quota
454

    
455
  resource    limit  usage
456
  ------------------------
457
  cyclades.vm 100    50
458

    
459
A new command ``snf-manage project-modify`` will enable in-place
460
modification of project properties, such as their quota limits.
461

    
462
Currently, the administrator can change the user base quota with:
463
``snf-manage user-modify <id> --base-quota <resource> <capacity>``.
464
This will be removed in favor of the ``project-modify`` command, so that all
465
quota are handled in a uniform way. Similar to ``user-modify --all``,
466
``project-modify`` will get options ``--all-base-projects`` to
467
allow updating base quota in bulk.
468

    
469
Migration steps
470
===============
471

    
472
Project conversion
473
------------------
474

    
475
Existing projects need to be converted to resource-pool ones. The following
476
steps must be taken in Astakos:
477
  * compute project-level limits for each resource as
478
    max_members * member-level limit
479
  * create base projects based on base quota for each user
480
  * make Quotaholder entries for projects and user/project pairs
481
  * assign all current usage to the base projects (both project
482
    and user/project entries)
483
  * set usage for all other entries to zero
484

    
485
Cyclades and Pithos should initialize their project attribute on each resource
486
with the user's base project, that is, the same UUID as the resource owner.
487

    
488
Initial resource reassignment
489
-----------------------------
490

    
491
Once migration has finished, users will be off-quota on their base project,
492
if they had used additional quota from projects. To alleviate this
493
situation, each service can attempt to reassign resources to other projects,
494
following this strategy:
495
  * consult Astakos for projects and quota for a given user
496
  * select resources that can fit in another project
497
  * issue a commission to decrease usage of the base project and likewise
498
    increase usage of the available project
499
  * record the new ProjectUUID for the reassigned resources