root / docs / design / resource-pool-projects.rst @ 111adca0
History | View | Annotate | Download (16.9 kB)
1 |
Resource-pool projects |
---|---|
2 |
^^^^^^^^^^^^^^^^^^^^^^ |
3 |
|
4 |
This document describes the current state of the quota and projects system, |
5 |
and proposes a new design for projects that would function as resource |
6 |
pools. It sketches implementation details and migration concerns. |
7 |
|
8 |
Current state and shortcomings |
9 |
============================== |
10 |
|
11 |
Each Synnefo user is granted quota for several resources. These quota |
12 |
originate from two different sources: the system and projects. By default |
13 |
a user holds so-called base quota granted by the system upon activation; |
14 |
base quota can be customized per user. When a user joins a project, |
15 |
resources offered by the project add up to the existing quota, increasing |
16 |
the total amount of resources one can reserve. |
17 |
|
18 |
This design fails to associate an actual (reserved) resource (e.g. VM) with |
19 |
a particular project. There is no way to tell which project a resource |
20 |
originates from and is thus not possible to employ any targeted policy when |
21 |
a user leaves a project, such as reclaiming the granted resource. It is also |
22 |
not possible to employ more advanced access control on resources, such as |
23 |
sharing VMs among members of a project. |
24 |
|
25 |
Proposed changes |
26 |
================ |
27 |
|
28 |
We will alter project semantics so that a project is viewed as a pool of |
29 |
finite resources. Each project member can reserve a portion of these |
30 |
resources up to a specified limit. Each actual resource (e.g. VM) is |
31 |
associated with a particular project. Admission of a user to a project will |
32 |
no more result in increasing the user's existing overall quota, but in |
33 |
defining new project-specific quota for the user. |
34 |
|
35 |
A project defines a pair of limits for each resource that it grants (e.g. |
36 |
cyclades.vm): project-level limit and member-level limit; The former is the |
37 |
total amount of a resource that this project can grant; the latter is the |
38 |
maximum amount that an individual user (project member) can reserve and |
39 |
cannot exceed the former. A limit on the number of members allowed is still |
40 |
enforced. |
41 |
|
42 |
Projects will be the sole source of resources. Current base quota offered to |
43 |
users by the system will be expressed in terms of special-purpose *base* |
44 |
projects. |
45 |
|
46 |
Implementation details |
47 |
====================== |
48 |
|
49 |
Project-related quota holdings |
50 |
------------------------------ |
51 |
|
52 |
The Quotaholder is responsible to record all resource allocations and |
53 |
deallocations, and enforce the limits. It keeps counters of the following |
54 |
structure: |
55 |
* resource: the resource name (e.g. cyclades.vm) |
56 |
* holder: the entity holding the resource (user or project) |
57 |
* source: the origin of the resource; a user-holder reserves from a |
58 |
project, a project is a top-level entity and reserves from nowhere (None) |
59 |
* limit: maximum allowed allocation (an integer) |
60 |
* usage: current allocation (an integer) |
61 |
|
62 |
[Due to the transactional nature of the mechanism, there are actually two |
63 |
usage fields (usage_min and usage_max). Details are beyond the scope of |
64 |
this document.] |
65 |
|
66 |
Creation of a new project triggers the creation of counters like:: |
67 |
|
68 |
resource holder source limit usage |
69 |
------------|-------------------|--------|-------|------ |
70 |
cyclades.vm project:projectID None 50 0 |
71 |
|
72 |
When a user is admitted in a project, counters are created like:: |
73 |
|
74 |
resource holder source limit usage |
75 |
------------|---------------|-------------------|-------|------ |
76 |
cyclades.vm user:userUUID project:ProjectID 5 0 |
77 |
|
78 |
Note that the two types of holders (and sources) are made distinguishable with |
79 |
a prefix: ``user:`` or ``project:``. |
80 |
|
81 |
When a user leaves a project, the latter limit is set to zero. This results |
82 |
in the project-specific user quota being over limit and prohibits any |
83 |
further allocation that would increase this counter. When a project |
84 |
is deactivated, the limit of both types of counters is set to zero. |
85 |
No user can perform any allocation related to this project. However, the |
86 |
holdings cannot be deleted as long as a non-zero usage is recorded. |
87 |
Deallocation is always allowed as long as usage does not fall below zero. |
88 |
Counters with zero usage and limit could by garbage collected by Astakos, if |
89 |
needed. |
90 |
|
91 |
Base projects |
92 |
------------- |
93 |
|
94 |
For reasons of uniformity, we replace the base quota mechanism with projects. |
95 |
In a similar vein to OpenStack tenants, we define new user-specific *base* |
96 |
projects to account for the base quota for each user. These projects should |
97 |
be clearly associated with a single user, restrict join/leave actions and |
98 |
specify the quota granted by the system. When a new user is created, |
99 |
their base project will be automatically created and linked back to the user. |
100 |
User activation will trigger project activation, granting the default resource |
101 |
quota. Base projects will have no owner, marked thusly as `system' projects. |
102 |
The administrator can, following the usual project logic, alter quota by |
103 |
modifying the project. Users cannot apply for modification of their base |
104 |
projects. |
105 |
|
106 |
Projects will, from now on, be identified by a UUID. Base projects will |
107 |
receive the same UUID as the user itself. ProjectID, which appears above in |
108 |
the Quotaholder entries, refers to the project UUID. |
109 |
|
110 |
Base quota will be expressed both in terms of a project-level and a |
111 |
member-level limit. This will result in two operationally equivalent |
112 |
Quotaholder counters, as in the following example. In the future, we could |
113 |
admit third-party users to a user's base project; in that case, those |
114 |
counters would differ. |
115 |
|
116 |
:: |
117 |
|
118 |
resource holder source limit usage |
119 |
------------|--------------|--------------|-------|------ |
120 |
cyclades.vm project:uuid None 5 1 |
121 |
cyclades.vm user:uuid project:uuid 5 1 |
122 |
|
123 |
System default quota |
124 |
-------------------- |
125 |
|
126 |
Each resource registered in the system is assigned a default quota limit. |
127 |
A newly-activated user is given these limits as their base quota. This is |
128 |
till now done by copying the default limits as user's entries in |
129 |
AstakosUserQuota. Default limits will from now on be copied into the base |
130 |
project's resource definitions. |
131 |
|
132 |
Conventional projects are created through a project application, which |
133 |
may not specify limits for all resources registered in the system. In |
134 |
fact, it may even be impossible to specify a resource, if it is set |
135 |
``api_visible=False``. We have to somehow specify these limits. Defaulting |
136 |
to zero is not appropriate: if we don't want to control a resource, we |
137 |
would like it set to infinite. We thus need an extra skeleton, like the |
138 |
one specifying the default base quota, in order to fill in missing limits |
139 |
for conventional projects. It will be controled by a new option |
140 |
``--project-default`` of command ``resource-modify``. |
141 |
|
142 |
Private projects |
143 |
---------------- |
144 |
|
145 |
Since the introduction of base projects will explode the number of total |
146 |
projects, we will need to control their visibility. We add a new flag |
147 |
*private* in project definitions. A private project can only be accessed by |
148 |
its owner and members and not be advertised in the UI. Base projects are |
149 |
marked as private. |
150 |
|
151 |
Allocation of a new resource |
152 |
---------------------------- |
153 |
|
154 |
When a service allocates a new resource, it should associate it both with a |
155 |
user and a project. The commission issued to the Quotaholder should attempt |
156 |
to update all related counters. For example, it should include the following |
157 |
provisions:: |
158 |
|
159 |
"provisions": [ |
160 |
{ |
161 |
"holder": "user:user-uuid", |
162 |
"source": "project:project-uuid", |
163 |
"resource": "cyclades.vm", |
164 |
"quantity": 1 |
165 |
}, |
166 |
{ |
167 |
"holder": "project:project-uuid", |
168 |
"source": None, |
169 |
"resource": "cyclades.vm", |
170 |
"quantity": 1 |
171 |
}, |
172 |
{ |
173 |
"holder": "user:user-uuid", |
174 |
"source": "project:project-uuid", |
175 |
"resource": "cyclades.cpu", |
176 |
"quantity": 2 |
177 |
}, |
178 |
{ |
179 |
"holder": "project:project-uuid", |
180 |
"source": None, |
181 |
"resource": "cyclades.cpu", |
182 |
"quantity": 2 |
183 |
} |
184 |
] |
185 |
|
186 |
If any of these provisions fails, i.e. either on the project-level limits or |
187 |
the user-level ones, the whole commission fails. |
188 |
|
189 |
The astakosclient call ``issue_one_commission`` will be adapted to abstract |
190 |
away the need to write both the user-level and the project-level provisions. |
191 |
The previous commission will be issued with:: |
192 |
|
193 |
issue_one_commission(holder="user-uuid", source="project-uuid", |
194 |
provisions={"cyclades.vm": 1, "cyclades.cpu": 2}) |
195 |
|
196 |
The service is responsible to record this resource-to-project association. |
197 |
In Cyclades, each VM, floating IP, or other distinct resource should be |
198 |
linked to a project. Pithos should link containers to projects. |
199 |
|
200 |
Astakos will handle its own resource ``astakos.pending_app`` in a special |
201 |
way: it will always be charged at the user's base project. This resource |
202 |
is marked with ``allow_in_projects = False`` in its definition. Since quota |
203 |
is now project-based, this flag will now be interpreted as forbidding usage |
204 |
in non-base projects. |
205 |
|
206 |
Resource reassignment |
207 |
--------------------- |
208 |
|
209 |
The system will support reassigning a resource to a new project. One needs |
210 |
to specify all related resource values. Astakosclient will provide a |
211 |
convenience function ``issue_resource_reassignment`` to construct all needed |
212 |
provisions. For instance, reassigning a VM with two CPUs can be done with:: |
213 |
|
214 |
issue_resource_reassignment(holder="user-uuid", |
215 |
from_source="from-uuid", to_source="to-uuid", |
216 |
provisions={"cyclades.vm": 1, "cyclades.cpu": 2}) |
217 |
|
218 |
This will issue the following provisions to the Quotaholder:: |
219 |
|
220 |
"provisions": [ |
221 |
{ |
222 |
"holder": "user:user-uuid", |
223 |
"source": "project:from-uuid", |
224 |
"resource": "cyclades.vm", |
225 |
"quantity": -1 |
226 |
}, |
227 |
{ |
228 |
"holder": "project:from-uuid", |
229 |
"source": None, |
230 |
"resource": "cyclades.vm", |
231 |
"quantity": -1 |
232 |
}, |
233 |
{ |
234 |
"holder": "user:user-uuid", |
235 |
"source": "project:from-uuid", |
236 |
"resource": "cyclades.cpu", |
237 |
"quantity": -2 |
238 |
}, |
239 |
{ |
240 |
"holder": "project:from-uuid", |
241 |
"source": None, |
242 |
"resource": "cyclades.cpu", |
243 |
"quantity": -2 |
244 |
}, |
245 |
{ |
246 |
"holder": "user:user-uuid", |
247 |
"source": "project:to-uuid", |
248 |
"resource": "cyclades.vm", |
249 |
"quantity": 1 |
250 |
}, |
251 |
{ |
252 |
"holder": "project:to-uuid", |
253 |
"source": None, |
254 |
"resource": "cyclades.vm", |
255 |
"quantity": 1 |
256 |
} |
257 |
{ |
258 |
"holder": "user:user-uuid", |
259 |
"source": "project:to-uuid", |
260 |
"resource": "cyclades.cpu", |
261 |
"quantity": 2 |
262 |
}, |
263 |
{ |
264 |
"holder": "project:to-uuid", |
265 |
"source": None, |
266 |
"resource": "cyclades.cpu", |
267 |
"quantity": 2 |
268 |
} |
269 |
] |
270 |
|
271 |
API extensions |
272 |
-------------- |
273 |
|
274 |
API call ``GET /quotas`` is extended to incorporate project-level quota. The |
275 |
response contains entries for all projects for which a user/project pair |
276 |
exists in the quotaholder:: |
277 |
|
278 |
{ |
279 |
"project1-uuid": { |
280 |
"cyclades.ram": { |
281 |
"usage": 2147483648, |
282 |
"limit": 2147483648, |
283 |
"pending": 0, |
284 |
"project_usage": ..., |
285 |
"project_limit": ..., |
286 |
"project_pending": ... |
287 |
}, |
288 |
"cyclades.vm": { |
289 |
... |
290 |
} |
291 |
} |
292 |
"project2-uuid": { |
293 |
... |
294 |
} |
295 |
} |
296 |
|
297 |
An extra or differentiated call may be needed to retrieve the project quota |
298 |
regardless of user:: |
299 |
|
300 |
GET /quotas?mode=projects |
301 |
|
302 |
{ |
303 |
"project-uuid": { |
304 |
"cyclades.ram": { |
305 |
"project_usage": 2147483648, |
306 |
"project_limit": 2147483648, |
307 |
"project_pending": 0 |
308 |
} |
309 |
"cyclades.vm": { |
310 |
... |
311 |
} |
312 |
} |
313 |
} |
314 |
|
315 |
All service API calls that create resources can specify the project where |
316 |
they will be attributed. |
317 |
|
318 |
In cyclades, ``POST /servers`` (likewise for networks and floating IPs) will |
319 |
receive an extra argument ``project``. If it is missing, the user's base |
320 |
project will be assumed. In calls detailing a resource (e.g., ``GET |
321 |
/servers/<server_id>``), the field ``tenant_id`` will contain the |
322 |
project id. |
323 |
|
324 |
Moreover, extra calls will be needed for resource reassignment, |
325 |
e.g:: |
326 |
|
327 |
POST /servers/<server-id>/action |
328 |
|
329 |
{ |
330 |
"reassign": {"project": <project-id>} |
331 |
} |
332 |
|
333 |
In pithos, ``PUT`` and ``POST`` calls at the container level will accept an |
334 |
extra optional policy ``project``. The former call assigns a newly created |
335 |
container to a given project, the latter reassigns an existing container. |
336 |
Field ``x-container-policy-project`` will be retrieved by a ``HEAD`` call at |
337 |
the container level. |
338 |
|
339 |
User interface |
340 |
-------------- |
341 |
|
342 |
User quota will be presented per project, including the aggregate activity |
343 |
of other project members: the Resource Usage page will include a drop-down |
344 |
menu with all relevant projects. By default, user's base project will |
345 |
be assumed. When choosing a project, usage for all resources will be |
346 |
presented for the given project in the following style:: |
347 |
|
348 |
limit |
349 |
used ^ taken by others |
350 |
|::::::|..............|...........|::::::::::::::::::::::::::::::::::| |
351 |
^ ^ ^ |
352 |
usage effective project |
353 |
limit limit |
354 |
|
355 |
|
356 |
limit |
357 |
used ^ taken by others |
358 |
|::::::|........|:::::|::::::::::::::::::::::::::::::::::::::::::::::| |
359 |
^ ^ ^ |
360 |
usage effective project |
361 |
limit limit |
362 |
|
363 |
Text accompanying the bar could mention usage based on the effective limit, |
364 |
e.g.: `usage` out of `effective limit` Virtual Machines. Likewise the shaded |
365 |
`used` part of the bar could express the same ratio in percentage terms. |
366 |
|
367 |
Given the above-mentioned response of the ``/quotas`` call, the effective |
368 |
limit can be computed by:: |
369 |
|
370 |
taken_by_others = project_usage - usage |
371 |
effective_limit = min(limit, project_limit - taken_by_others) |
372 |
|
373 |
Projects show up in a number of service-specific user interactions, too. |
374 |
When creating a Cyclades VM, the flavor-choosing window should first ask |
375 |
for the project where the VM will be charged before showing the |
376 |
available resource combinations. Likewise, creating a new container in |
377 |
Pithos will prompt for picking a project to associate with. |
378 |
|
379 |
Resource presentation (e.g. Cyclades VMs) will also mention the associated |
380 |
project and provide an action to reassign the resource to a different |
381 |
project. |
382 |
|
383 |
Command-line interface |
384 |
---------------------- |
385 |
|
386 |
Quota can be queried per user or project:: |
387 |
|
388 |
# snf-manage user-show <id> --quota |
389 |
|
390 |
project resource limit effective_limit usage |
391 |
------------------------------------------------- |
392 |
uuid cyclades.vm 10 9 5 |
393 |
|
394 |
# snf-manage project-show <id> --quota |
395 |
|
396 |
resource limit usage |
397 |
------------------------ |
398 |
cyclades.vm 100 50 |
399 |
|
400 |
A new command ``snf-manage project-modify`` will automate the process of |
401 |
applying/approving applications in order to modify some project settings, |
402 |
such as the quota limits. |
403 |
|
404 |
Currently, the administrator can change the user base quota with: |
405 |
``snf-manage user-modify <id> --set-base-quota <resource> <capacity>``. |
406 |
This will be removed in favor of the ``project-modify`` command, so that all |
407 |
quota are handled in a uniform way. Similar to ``user-modify --all``, |
408 |
``project-modify`` will get options ``--all-base`` and ``--all-non-base`` to |
409 |
allow updating quota in bulk. |
410 |
|
411 |
Migration steps |
412 |
=============== |
413 |
|
414 |
Project conversion |
415 |
------------------ |
416 |
|
417 |
Existing projects need to be converted to resource-pool ones. The following |
418 |
steps must be taken in Astakos: |
419 |
* compute project-level limits for each resource as |
420 |
max_members * member-level limit |
421 |
* create base projects based on base quota for each user |
422 |
* make Quotaholder entries for projects and user/project pairs |
423 |
* assign all current usage to the base projects (both project |
424 |
and user/project entries) |
425 |
* set usage for all other entries to zero |
426 |
|
427 |
Cyclades and Pithos should initialize their project attribute on each resource |
428 |
with the user's base project, that is, the same UUID as the resource owner. |
429 |
|
430 |
Initial resource reassignment |
431 |
----------------------------- |
432 |
|
433 |
Once migration has finished, users will be off-quota on their base project, |
434 |
if they had used additional quota from projects. To alleviate this |
435 |
situation, each service can attempt to reassign resources to other projects, |
436 |
following this strategy: |
437 |
* consult Astakos for projects and quota for a given user |
438 |
* select resources that can fit in another project |
439 |
* issue a commission to decrease usage of the base project and likewise |
440 |
increase usage of the available project |
441 |
* record the new ProjectUUID for the reassigned resources |