|
1 |
Resource-pool projects
|
|
2 |
^^^^^^^^^^^^^^^^^^^^^^
|
|
3 |
|
|
4 |
This document describes the current state of the quota and projects system,
|
|
5 |
and proposes a new design for projects that would function as resource
|
|
6 |
pools. It sketches implementation details and migration concerns.
|
|
7 |
|
|
8 |
Current state and shortcomings
|
|
9 |
==============================
|
|
10 |
|
|
11 |
Each Synnefo user is granted quota for several resources. These quota
|
|
12 |
originate from two different sources: the system and projects. By default
|
|
13 |
a user holds so-called base quota granted by the system upon activation;
|
|
14 |
base quota can be customized per user. When a user joins a project,
|
|
15 |
resources offered by the project add up to the existing quota, increasing
|
|
16 |
the total amount of resources one can reserve.
|
|
17 |
|
|
18 |
This design fails to associate an actual (reserved) resource (e.g. VM) with
|
|
19 |
a particular project. There is no way to tell which project a resource
|
|
20 |
originates from and is thus not possible to employ any targeted policy when
|
|
21 |
a user leaves a project, such as reclaiming the granted resource. It is also
|
|
22 |
not possible to employ more advanced access control on resources, such as
|
|
23 |
sharing VMs among members of a project.
|
|
24 |
|
|
25 |
Proposed changes
|
|
26 |
================
|
|
27 |
|
|
28 |
We will alter project semantics so that a project is viewed as a pool of
|
|
29 |
finite resources. Each project member can reserve a portion of these
|
|
30 |
resources up to a specified limit. Each actual resource (e.g. VM) is
|
|
31 |
associated with a particular project. Admission of a user to a project will
|
|
32 |
no more result in increasing the user's existing overall quota, but in
|
|
33 |
defining new project-specific quota for the user.
|
|
34 |
|
|
35 |
A project defines a pair of limits for each resource that it grants (e.g.
|
|
36 |
cyclades.vm): project-level limit and member-level limit; The former is the
|
|
37 |
total amount of a resource that this project can grant; the latter is the
|
|
38 |
maximum amount that an individual user (project member) can reserve and
|
|
39 |
cannot exceed the former. A limit on the number of members allowed is still
|
|
40 |
enforced.
|
|
41 |
|
|
42 |
Projects will be the sole source of resources. Current base quota offered to
|
|
43 |
users by the system will be expressed in terms of special-purpose *base*
|
|
44 |
projects.
|
|
45 |
|
|
46 |
Implementation details
|
|
47 |
======================
|
|
48 |
|
|
49 |
Project-related quota holdings
|
|
50 |
------------------------------
|
|
51 |
|
|
52 |
The Quotaholder is responsible to record all resource allocations and
|
|
53 |
deallocations, and enforce the limits. It keeps counters of the following
|
|
54 |
structure:
|
|
55 |
* resource: the resource name (e.g. cyclades.vm)
|
|
56 |
* holder: the entity holding the resource (user or project)
|
|
57 |
* source: the origin of the resource; a user-holder reserves from a
|
|
58 |
project, a project is a top-level entity and reserves from nowhere (None)
|
|
59 |
* limit: maximum allowed allocation (an integer)
|
|
60 |
* usage: current allocation (an integer)
|
|
61 |
|
|
62 |
[Due to the transactional nature of the mechanism, there are actually two
|
|
63 |
usage fields (usage_min and usage_max). Details are beyond the scope of
|
|
64 |
this document.]
|
|
65 |
|
|
66 |
Creation of a new project triggers the creation of counters like::
|
|
67 |
|
|
68 |
resource holder source limit usage
|
|
69 |
------------|-------------------|--------|-------|------
|
|
70 |
cyclades.vm project:projectID None 50 0
|
|
71 |
|
|
72 |
When a user is admitted in a project, counters are created like::
|
|
73 |
|
|
74 |
resource holder source limit usage
|
|
75 |
------------|---------------|-------------------|-------|------
|
|
76 |
cyclades.vm user:userUUID project:ProjectID 5 0
|
|
77 |
|
|
78 |
Note that the two types of holders (and sources) are made distinguishable with
|
|
79 |
a prefix: ``user:`` or ``project:``.
|
|
80 |
|
|
81 |
When a user leaves a project, the latter limit is set to zero. This results
|
|
82 |
in the project-specific user quota being over limit and prohibits any
|
|
83 |
further allocation that would increase this counter. When a project
|
|
84 |
is deactivated, the limit of both types of counters is set to zero.
|
|
85 |
No user can perform any allocation related to this project. However, the
|
|
86 |
holdings cannot be deleted as long as a non-zero usage is recorded.
|
|
87 |
Deallocation is always allowed as long as usage does not fall below zero.
|
|
88 |
Counters with zero usage and limit could by garbage collected by Astakos, if
|
|
89 |
needed.
|
|
90 |
|
|
91 |
Base projects
|
|
92 |
-------------
|
|
93 |
|
|
94 |
For reasons of uniformity, we replace the base quota mechanism with projects.
|
|
95 |
In a similar vein to OpenStack tenants, we define new user-specific *base*
|
|
96 |
projects to account for the base quota for each user. These projects should
|
|
97 |
be clearly associated with a single user, restrict join/leave actions and
|
|
98 |
specify the quota granted by the system. When a new user is created,
|
|
99 |
their base project will be automatically created and linked back to the user.
|
|
100 |
User activation will trigger project activation, granting the default resource
|
|
101 |
quota. Base projects will have no owner, marked thusly as `system' projects.
|
|
102 |
The administrator can, following the usual project logic, alter quota by
|
|
103 |
modifying the project. Users cannot apply for modification of their base
|
|
104 |
projects.
|
|
105 |
|
|
106 |
Projects will, from now on, be identified by a UUID. Base projects will
|
|
107 |
receive the same UUID as the user itself. ProjectID, which appears above in
|
|
108 |
the Quotaholder entries, refers to the project UUID.
|
|
109 |
|
|
110 |
Base quota will be expressed both in terms of a project-level and a
|
|
111 |
member-level limit. This will result in two operationally equivalent
|
|
112 |
Quotaholder counters, as in the following example. In the future, we could
|
|
113 |
admit third-party users to a user's base project; in that case, those
|
|
114 |
counters would differ.
|
|
115 |
|
|
116 |
::
|
|
117 |
|
|
118 |
resource holder source limit usage
|
|
119 |
------------|--------------|--------------|-------|------
|
|
120 |
cyclades.vm project:uuid None 5 1
|
|
121 |
cyclades.vm user:uuid project:uuid 5 1
|
|
122 |
|
|
123 |
System default base quota
|
|
124 |
-------------------------
|
|
125 |
|
|
126 |
Each resource registered in the system is assigned a default quota limit.
|
|
127 |
A newly-activated user is given these limits as their base quota. Up to now,
|
|
128 |
a change in a default quota limit propagates to all users' base quota
|
|
129 |
(except if they have been customized). Since all users will now have, in a
|
|
130 |
way, `customized' base quota (through their base projects), this effect is
|
|
131 |
meaningless. Therefore, semantics of default quota limits will change to
|
|
132 |
affect only future users (i.e. construction of new base projects).
|
|
133 |
|
|
134 |
Private projects
|
|
135 |
----------------
|
|
136 |
|
|
137 |
Since the introduction of base projects will explode the number of total
|
|
138 |
projects, we will need to control their visibility. We add a new flag
|
|
139 |
*private* in project definitions. A private project can only be accessed by
|
|
140 |
its owner and members and not be advertised in the UI. Base projects are
|
|
141 |
marked as private.
|
|
142 |
|
|
143 |
Allocation of a new resource
|
|
144 |
----------------------------
|
|
145 |
|
|
146 |
When a service allocates a new resource, it should associate it both with a
|
|
147 |
user and a project. The commission issued to the Quotaholder should attempt
|
|
148 |
to update all related counters. For example, it should include the following
|
|
149 |
provisions::
|
|
150 |
|
|
151 |
"provisions": [
|
|
152 |
{
|
|
153 |
"holder": "user:user-uuid",
|
|
154 |
"source": "project:project-uuid",
|
|
155 |
"resource": "cyclades.vm",
|
|
156 |
"quantity": 1
|
|
157 |
},
|
|
158 |
{
|
|
159 |
"holder": "project:project-uuid",
|
|
160 |
"source": None,
|
|
161 |
"resource": "cyclades.vm",
|
|
162 |
"quantity": 1
|
|
163 |
},
|
|
164 |
{
|
|
165 |
"holder": "user:user-uuid",
|
|
166 |
"source": "project:project-uuid",
|
|
167 |
"resource": "cyclades.cpu",
|
|
168 |
"quantity": 2
|
|
169 |
},
|
|
170 |
{
|
|
171 |
"holder": "project:project-uuid",
|
|
172 |
"source": None,
|
|
173 |
"resource": "cyclades.cpu",
|
|
174 |
"quantity": 2
|
|
175 |
}
|
|
176 |
]
|
|
177 |
|
|
178 |
If any of these provisions fails, i.e. either on the project-level limits or
|
|
179 |
the user-level ones, the whole commission fails.
|
|
180 |
|
|
181 |
The astakosclient call ``issue_one_commission`` will be adapted to abstract
|
|
182 |
away the need to write both the user-level and the project-level provisions.
|
|
183 |
The previous commission will be issued with::
|
|
184 |
|
|
185 |
issue_one_commission(token, holder="user-uuid", source="project-uuid",
|
|
186 |
provisions={"cyclades.vm": 1, "cyclades.cpu": 2})
|
|
187 |
|
|
188 |
The service is responsible to record this resource-to-project association.
|
|
189 |
In Cyclades, each VM, floating IP, or other distinct resource should be
|
|
190 |
linked to a project. Pithos should link containers to projects.
|
|
191 |
|
|
192 |
Astakos will handle its own resource ``astakos.pending_app`` in a special
|
|
193 |
way: it will always be charged at the user's base project. This resource
|
|
194 |
is marked with ``allow_in_projects = False`` in its definition. Since quota
|
|
195 |
is now project-based, this flag will now be interpreted as forbidding usage
|
|
196 |
in non-base projects.
|
|
197 |
|
|
198 |
Resource reassignment
|
|
199 |
---------------------
|
|
200 |
|
|
201 |
The system will support reassigning a resource to a new project. One needs
|
|
202 |
to specify all related resource values. Astakosclient will provide a
|
|
203 |
convenience function ``reassign_resources`` to construct all needed
|
|
204 |
provisions. For instance, reassigning a VM with two CPUs can be done with::
|
|
205 |
|
|
206 |
reassign_resources(token, holder="user-uuid", from="from-uuid", to="to-uuid",
|
|
207 |
provisions={"cyclades.vm": 1, "cyclades.cpu": 2})
|
|
208 |
|
|
209 |
This will issue the following provisions to the Quotaholder::
|
|
210 |
|
|
211 |
"provisions": [
|
|
212 |
{
|
|
213 |
"holder": "user:user-uuid",
|
|
214 |
"source": "project:from-uuid",
|
|
215 |
"resource": "cyclades.vm",
|
|
216 |
"quantity": -1
|
|
217 |
},
|
|
218 |
{
|
|
219 |
"holder": "project:from-uuid",
|
|
220 |
"source": None,
|
|
221 |
"resource": "cyclades.vm",
|
|
222 |
"quantity": -1
|
|
223 |
},
|
|
224 |
{
|
|
225 |
"holder": "user:user-uuid",
|
|
226 |
"source": "project:from-uuid",
|
|
227 |
"resource": "cyclades.cpu",
|
|
228 |
"quantity": -2
|
|
229 |
},
|
|
230 |
{
|
|
231 |
"holder": "project:from-uuid",
|
|
232 |
"source": None,
|
|
233 |
"resource": "cyclades.cpu",
|
|
234 |
"quantity": -2
|
|
235 |
},
|
|
236 |
{
|
|
237 |
"holder": "user:user-uuid",
|
|
238 |
"source": "project:to-uuid",
|
|
239 |
"resource": "cyclades.vm",
|
|
240 |
"quantity": 1
|
|
241 |
},
|
|
242 |
{
|
|
243 |
"holder": "project:to-uuid",
|
|
244 |
"source": None,
|
|
245 |
"resource": "cyclades.vm",
|
|
246 |
"quantity": 1
|
|
247 |
}
|
|
248 |
{
|
|
249 |
"holder": "user:user-uuid",
|
|
250 |
"source": "project:to-uuid",
|
|
251 |
"resource": "cyclades.cpu",
|
|
252 |
"quantity": 2
|
|
253 |
},
|
|
254 |
{
|
|
255 |
"holder": "project:to-uuid",
|
|
256 |
"source": None,
|
|
257 |
"resource": "cyclades.cpu",
|
|
258 |
"quantity": 2
|
|
259 |
}
|
|
260 |
]
|
|
261 |
|
|
262 |
API extensions
|
|
263 |
--------------
|
|
264 |
|
|
265 |
API call ``GET /quotas`` is extended to incorporate project-level quota. The
|
|
266 |
response contains entries for all projects for which a user/project pair
|
|
267 |
exists in the quotaholder::
|
|
268 |
|
|
269 |
{
|
|
270 |
"project1-uuid": {
|
|
271 |
"cyclades.ram": {
|
|
272 |
"usage": 2147483648,
|
|
273 |
"limit": 2147483648,
|
|
274 |
"pending": 0,
|
|
275 |
"project_usage": ...,
|
|
276 |
"project_limit": ...,
|
|
277 |
"project_pending": ...
|
|
278 |
},
|
|
279 |
"cyclades.vm": {
|
|
280 |
...
|
|
281 |
}
|
|
282 |
}
|
|
283 |
"project2-uuid": {
|
|
284 |
...
|
|
285 |
}
|
|
286 |
}
|
|
287 |
|
|
288 |
An extra or differentiated call may be needed to retrieve the project quota
|
|
289 |
regardless of user::
|
|
290 |
|
|
291 |
GET /quotas?mode=projects
|
|
292 |
|
|
293 |
{
|
|
294 |
"project-uuid": {
|
|
295 |
"cyclades.ram": {
|
|
296 |
"project_usage": 2147483648,
|
|
297 |
"project_limit": 2147483648,
|
|
298 |
"project_pending": 0
|
|
299 |
}
|
|
300 |
"cyclades.vm": {
|
|
301 |
...
|
|
302 |
}
|
|
303 |
}
|
|
304 |
}
|
|
305 |
|
|
306 |
All service API calls that create resources can specify the project where
|
|
307 |
they will be attributed. For example ``POST /servers`` will receive an extra
|
|
308 |
argument ``project``. If it is missing, the user's base project will be
|
|
309 |
assumed. Moreover, extra calls will be needed for resource reassignment,
|
|
310 |
e.g::
|
|
311 |
|
|
312 |
POST /servers/<server-id>/action
|
|
313 |
|
|
314 |
{
|
|
315 |
"reassign": <project-id>
|
|
316 |
}
|
|
317 |
|
|
318 |
User interface
|
|
319 |
--------------
|
|
320 |
|
|
321 |
User quota will be presented per project, including the aggregate activity
|
|
322 |
of other project members: the Resource Usage page will include a drop-down
|
|
323 |
menu with all relevant projects. By default, user's base project will
|
|
324 |
be assumed. When choosing a project, usage for all resources will be
|
|
325 |
presented for the given project in the following style::
|
|
326 |
|
|
327 |
limit
|
|
328 |
used ^ taken by others
|
|
329 |
|::::::|..............|...........|::::::::::::::::::::::::::::::::::|
|
|
330 |
^ ^ ^
|
|
331 |
usage effective project
|
|
332 |
limit limit
|
|
333 |
|
|
334 |
|
|
335 |
limit
|
|
336 |
used ^ taken by others
|
|
337 |
|::::::|........|:::::|::::::::::::::::::::::::::::::::::::::::::::::|
|
|
338 |
^ ^ ^
|
|
339 |
usage effective project
|
|
340 |
limit limit
|
|
341 |
|
|
342 |
Text accompanying the bar could mention usage based on the effective limit,
|
|
343 |
e.g.: `usage` out of `effective limit` Virtual Machines. Likewise the shaded
|
|
344 |
`used` part of the bar could express the same ratio in percentage terms.
|
|
345 |
|
|
346 |
Given the above-mentioned response of the ``/quotas`` call, the effective
|
|
347 |
limit can be computed by::
|
|
348 |
|
|
349 |
taken_by_others = project_usage - usage
|
|
350 |
effective_limit = min(limit, project_limit - taken_by_others)
|
|
351 |
|
|
352 |
Projects show up in a number of service-specific user interactions, too.
|
|
353 |
When creating a Cyclades VM, the flavor-choosing window should first ask
|
|
354 |
for the project where the VM will be charged before showing the
|
|
355 |
available resource combinations. Likewise, creating a new container in
|
|
356 |
Pithos will prompt for picking a project to associate with.
|
|
357 |
|
|
358 |
Resource presentation (e.g. Cyclades VMs) will also mention the associated
|
|
359 |
project and provide an action to reassign the resource to a different
|
|
360 |
project.
|
|
361 |
|
|
362 |
Command-line interface
|
|
363 |
----------------------
|
|
364 |
|
|
365 |
Quota can be queried per user or project::
|
|
366 |
|
|
367 |
# snf-manage user-show <id> --quota
|
|
368 |
|
|
369 |
project resource limit effective_limit usage
|
|
370 |
-------------------------------------------------
|
|
371 |
uuid cyclades.vm 10 9 5
|
|
372 |
|
|
373 |
# snf-manage project-show <id> --quota
|
|
374 |
|
|
375 |
resource limit usage
|
|
376 |
------------------------
|
|
377 |
cyclades.vm 100 50
|
|
378 |
|
|
379 |
``snf-manage quota`` will no more be used to list quota; it may be preserved
|
|
380 |
for checking the integrity of the Quotaholder.
|
|
381 |
|
|
382 |
A new command ``snf-manage project-modify`` will automate the process of
|
|
383 |
applying/approving applications in order to modify some project settings,
|
|
384 |
such as the quota limits.
|
|
385 |
|
|
386 |
Currently, the administrator can change the user base quota with:
|
|
387 |
``snf-manage user-modify <id> --set-base-quota <resource> <capacity>``.
|
|
388 |
This will be removed in favor of the ``project-modify`` command, so that all
|
|
389 |
quota are handled in a uniform way.
|
|
390 |
|
|
391 |
Migration steps
|
|
392 |
===============
|
|
393 |
|
|
394 |
Project conversion
|
|
395 |
------------------
|
|
396 |
|
|
397 |
Existing projects need to be converted to resource-pool ones. The following
|
|
398 |
steps must be taken in Astakos:
|
|
399 |
* compute project-level limits for each resource as
|
|
400 |
max_members * member-level limit
|
|
401 |
* create base projects based on base quota for each user
|
|
402 |
* make Quotaholder entries for projects and user/project pairs
|
|
403 |
* assign all current usage to the base projects (both project
|
|
404 |
and user/project entries)
|
|
405 |
* set usage for all other entries to zero
|
|
406 |
|
|
407 |
Cyclades and Pithos should initialize their project attribute on each resource
|
|
408 |
with the user's base project, that is, the same UUID as the resource owner.
|
|
409 |
|
|
410 |
Initial resource reassignment
|
|
411 |
-----------------------------
|
|
412 |
|
|
413 |
Once migration has finished, users will be off-quota on their base project,
|
|
414 |
if they had used additional quota from projects. To alleviate this
|
|
415 |
situation, each service can attempt to reassign resources to other projects,
|
|
416 |
following this strategy:
|
|
417 |
* consult Astakos for projects and quota for a given user
|
|
418 |
* select resources that can fit in another project
|
|
419 |
* issue a commission to decrease usage of the base project and likewise
|
|
420 |
increase usage of the available project
|
|
421 |
* record the new ProjectUUID for the reassigned resources
|