Revision a6eb13e9
b/docs/source/devguide.rst | ||
---|---|---|
25 | 25 |
========================= ================================ |
26 | 26 |
Revision Description |
27 | 27 |
========================= ================================ |
28 |
0.4 (June 22, 2011) Support updating/deleting individual metadata with ``POST``. |
|
28 | 29 |
0.3 (June 14, 2011) Large object support with ``X-Object-Manifest``. |
29 | 30 |
\ Allow for publicly available objects via ``https://hostname/public``. |
30 | 31 |
\ Support time-variant account/container listings. |
31 |
\ Add source version when duplicating with PUT/COPY/MOVE.
|
|
32 |
\ Add source version when duplicating with PUT/COPY. |
|
32 | 33 |
\ Request version in object HEAD/GET requests (list versions with GET). |
33 | 34 |
0.2 (May 31, 2011) Add object meta listing and filtering in containers. |
34 | 35 |
\ Include underlying storage characteristics in container meta. |
... | ... | |
183 | 184 |
POST |
184 | 185 |
"""" |
185 | 186 |
|
187 |
====================== ============================================ |
|
188 |
Request Parameter Name Value |
|
189 |
====================== ============================================ |
|
190 |
update Do not replace metadata (no value parameter) |
|
191 |
====================== ============================================ |
|
192 |
|
|
193 |
| |
|
194 |
|
|
186 | 195 |
==================== =========================== |
187 | 196 |
Request Header Name Value |
188 | 197 |
==================== =========================== |
... | ... | |
191 | 200 |
|
192 | 201 |
No reply content/headers. |
193 | 202 |
|
194 |
The update operation will overwrite all user defined metadata.
|
|
203 |
The operation will overwrite all user defined metadata, except if ``update`` is defined.
|
|
195 | 204 |
|
196 | 205 |
================ =============================== |
197 | 206 |
Return Code Description |
... | ... | |
340 | 349 |
POST |
341 | 350 |
"""" |
342 | 351 |
|
352 |
====================== ============================================ |
|
353 |
Request Parameter Name Value |
|
354 |
====================== ============================================ |
|
355 |
update Do not replace metadata (no value parameter) |
|
356 |
====================== ============================================ |
|
357 |
|
|
358 |
| |
|
359 |
|
|
343 | 360 |
==================== ================================ |
344 | 361 |
Request Header Name Value |
345 | 362 |
==================== ================================ |
... | ... | |
348 | 365 |
|
349 | 366 |
No reply content/headers. |
350 | 367 |
|
351 |
The update operation will overwrite all user defined metadata.
|
|
368 |
The operation will overwrite all user defined metadata, except if ``update`` is defined.
|
|
352 | 369 |
|
353 | 370 |
================ =============================== |
354 | 371 |
Return Code Description |
... | ... | |
574 | 591 |
X-Object-Meta-* Optional user defined metadata |
575 | 592 |
==================== ================================ |
576 | 593 |
|
594 |
Refer to ``POST`` for a description of request headers. Metadata is also copied, updated with any values defined. |
|
595 |
|
|
577 | 596 |
No reply content/headers. |
578 | 597 |
|
579 | 598 |
=========================== ============================== |
... | ... | |
592 | 611 |
POST |
593 | 612 |
"""" |
594 | 613 |
|
614 |
====================== ============================================ |
|
615 |
Request Parameter Name Value |
|
616 |
====================== ============================================ |
|
617 |
update Do not replace metadata (no value parameter) |
|
618 |
====================== ============================================ |
|
619 |
|
|
620 |
| |
|
621 |
|
|
595 | 622 |
==================== ================================ |
596 | 623 |
Request Header Name Value |
597 | 624 |
==================== ================================ |
... | ... | |
606 | 633 |
X-Object-Meta-* Optional user defined metadata |
607 | 634 |
==================== ================================ |
608 | 635 |
|
609 |
The ``Content-Encoding``, ``Content-Disposition``, ``X-Object-Manifest``, ``X-Object-Public`` (**TBD**) and ``X-Object-Meta-*`` headers are considered to be user defined metadata. The update operation will overwrite all previous values and remove any keys not supplied.
|
|
636 |
The ``Content-Encoding``, ``Content-Disposition``, ``X-Object-Manifest``, ``X-Object-Public`` (**TBD**) and ``X-Object-Meta-*`` headers are considered to be user defined metadata. An operation without the ``update`` parameter will overwrite all previous values and remove any keys not supplied. When using ``update`` any metadata with an empty value will be deleted.
|
|
610 | 637 |
|
611 |
To update an object: |
|
638 |
To update an object's data:
|
|
612 | 639 |
|
640 |
* Set ``Content-Type`` to ``application/octet-stream``. If ``Content-Type`` has some other value, it will be ignored and only the metadata will be updated. |
|
613 | 641 |
* Supply ``Content-Length`` (except if using chunked transfers), ``Content-Type`` and ``Content-Range`` headers. |
614 |
* Set ``Content-Type`` to ``application/octet-stream``. |
|
615 | 642 |
* Set ``Content-Range`` as specified in RFC2616, with the following differences: |
616 | 643 |
|
617 | 644 |
* Client software MAY omit ``last-byte-pos`` of if the length of the range being transferred is unknown or difficult to determine. |
... | ... | |
636 | 663 |
202 (Accepted) The request has been accepted (not a data update) |
637 | 664 |
204 (No Content) The request succeeded (data updated) |
638 | 665 |
411 (Length Required) Missing ``Content-Length`` in the request |
639 |
416 (Range Not Satisfiable) The supplied range is out of limits or invalid size
|
|
666 |
416 (Range Not Satisfiable) The supplied range is invalid
|
|
640 | 667 |
=========================== ============================== |
641 | 668 |
|
642 | 669 |
|
... | ... | |
682 | 709 |
* All metadata replies, at all levels, include latest modification information. |
683 | 710 |
* At all levels, a ``GET`` request may use ``If-Modified-Since`` and ``If-Unmodified-Since`` headers. |
684 | 711 |
* Container/object lists include all associated metadata if the reply is of type json/xml. Some names are kept to their OOS API equivalents for compatibility. |
685 |
* Object metadata allowed, in addition to ``X-Object-Meta-*``: ``Content-Encoding``, ``Content-Disposition``, ``X-Object-Manifest``, ``X-Object-Public`` (**TBD**). These are all replaced with every update operation. |
|
712 |
* Object metadata allowed, in addition to ``X-Object-Meta-*``: ``Content-Encoding``, ``Content-Disposition``, ``X-Object-Manifest``, ``X-Object-Public`` (**TBD**). These are all replaced with every update operation, except if using the ``update`` parameter (in which case individual keys can also be deleted). Deleting meta by providing empty values also works when copying/moving an object.
|
|
686 | 713 |
* Multi-range object GET support as outlined in RFC2616. |
687 | 714 |
* Object hashmap retrieval through GET and the ``format`` parameter. |
688 | 715 |
* Partial object updates through POST, using the ``Content-Length``, ``Content-Type``, ``Content-Range`` and ``Transfer-Encoding`` headers. |
... | ... | |
702 | 729 |
* Container/object lists use a ``200`` return code if the reply is of type json/xml. The reply will include an empty json/xml. |
703 | 730 |
* In headers, dates are formatted according to RFC 1123. In extended information listings, dates are formatted according to ISO 8601. |
704 | 731 |
* The ``Last-Modified`` header value always reflects the actual latest change timestamp, regardless of time control parameters and version requests. Time precondition checks with ``If-Modified-Since`` and ``If-Unmodified-Since`` headers are applied to this value. |
732 |
* A copy/move using ``PUT``/``COPY``/``MOVE`` will always update metadata, keeping all old values except the ones redefined in the request headers. |
|
705 | 733 |
* A ``HEAD`` or ``GET`` for an ``X-Object-Manifest`` object, will include modified ``Content-Length`` and ``ETag`` headers, according to the characteristics of the objects under the specified prefix. The ``Etag`` will be the MD5 hash of the corresponding ETags concatenated. In extended container listings there is no metadata processing. |
706 | 734 |
|
707 | 735 |
The Pithos Client |
... | ... | |
796 | 824 |
-H "X-Auth-Token: 0000" \ |
797 | 825 |
https://pithos.dev.grnet.gr/v1/user/pithos?format=json |
798 | 826 |
|
827 |
It is recommended that extended replies are cached and subsequent requests utilize the ``If-Modified-Since`` header. |
|
828 |
|
|
799 | 829 |
* List metadata keys used by objects in a container |
800 | 830 |
|
801 | 831 |
Will be in the ``X-Container-Object-Meta`` reply header, included in container information or object list (``HEAD`` or ``GET``). |
b/pithos/api/functions.py | ||
---|---|---|
139 | 139 |
# unauthorized (401), |
140 | 140 |
# badRequest (400) |
141 | 141 |
|
142 |
meta = get_account_meta(request) |
|
143 |
backend.update_account_meta(request.user, v_account, meta, replace=True) |
|
142 |
meta = get_account_meta(request) |
|
143 |
replace = True |
|
144 |
if 'update' in request.GET: |
|
145 |
replace = False |
|
146 |
backend.update_account_meta(request.user, v_account, meta, replace) |
|
144 | 147 |
return HttpResponse(status=202) |
145 | 148 |
|
146 | 149 |
@api_method('GET', format_allowed=True) |
... | ... | |
248 | 251 |
# badRequest (400) |
249 | 252 |
|
250 | 253 |
meta = get_container_meta(request) |
254 |
replace = True |
|
255 |
if 'update' in request.GET: |
|
256 |
replace = False |
|
251 | 257 |
try: |
252 |
backend.update_container_meta(request.user, v_account, v_container, meta, replace=True)
|
|
258 |
backend.update_container_meta(request.user, v_account, v_container, meta, replace) |
|
253 | 259 |
except NameError: |
254 | 260 |
raise ItemNotFound('Container does not exist') |
255 | 261 |
return HttpResponse(status=202) |
b/pithos/api/util.py | ||
---|---|---|
77 | 77 |
"""Get all prefix-* request headers in a dict. Reformat keys with format_meta_key().""" |
78 | 78 |
|
79 | 79 |
prefix = 'HTTP_' + prefix.upper().replace('-', '_') |
80 |
return dict([(format_meta_key(k[5:]), v) for k, v in request.META.iteritems() if k.startswith(prefix)]) |
|
80 |
return dict([(format_meta_key(k[5:]), v) for k, v in request.META.iteritems() if k.startswith(prefix) and len(k) > len(prefix)])
|
|
81 | 81 |
|
82 | 82 |
def get_account_meta(request): |
83 | 83 |
"""Get metadata from an account request.""" |
... | ... | |
232 | 232 |
|
233 | 233 |
meta = get_object_meta(request) |
234 | 234 |
permissions = get_sharing(request) |
235 |
src_version = request.META.get('HTTP_X_SOURCE_VERSION') |
|
236 |
|
|
237 |
try: |
|
238 |
if move: |
|
239 |
src_meta = backend.get_object_meta(request.user, v_account, src_container, src_name) |
|
240 |
else: |
|
241 |
src_meta = backend.get_object_meta(request.user, v_account, src_container, src_name, src_version) |
|
242 |
except NameError, IndexError: |
|
243 |
raise ItemNotFound('Container or object does not exist') |
|
244 |
|
|
245 |
# Keep previous values of 'Content-Type' (if a new one is absent) and 'hash'. |
|
246 |
if 'Content-Type' in meta and 'Content-Type' in src_meta: |
|
247 |
del(src_meta['Content-Type']) |
|
248 |
for k in ('Content-Type', 'hash'): |
|
249 |
if k in src_meta: |
|
250 |
meta[k] = src_meta[k] |
|
251 |
|
|
235 |
src_version = request.META.get('HTTP_X_SOURCE_VERSION') |
|
252 | 236 |
try: |
253 | 237 |
if move: |
254 |
backend.move_object(request.user, v_account, src_container, src_name, dest_container, dest_name, meta, True, permissions)
|
|
238 |
backend.move_object(request.user, v_account, src_container, src_name, dest_container, dest_name, meta, False, permissions)
|
|
255 | 239 |
else: |
256 |
backend.copy_object(request.user, v_account, src_container, src_name, dest_container, dest_name, meta, True, permissions, src_version)
|
|
240 |
backend.copy_object(request.user, v_account, src_container, src_name, dest_container, dest_name, meta, False, permissions, src_version)
|
|
257 | 241 |
except NameError, IndexError: |
258 | 242 |
raise ItemNotFound('Container or object does not exist') |
259 | 243 |
except ValueError: |
b/pithos/backends/base.py | ||
---|---|---|
39 | 39 |
|
40 | 40 |
Note that the account level is always valid as it is checked from another subsystem. |
41 | 41 |
|
42 |
When not replacing metadata, keys with empty values should be deleted. |
|
43 |
|
|
42 | 44 |
The following variables should be available: |
43 | 45 |
'hash_algorithm': Suggested is 'sha256' |
44 | 46 |
'block_size': Suggested is 4MB |
... | ... | |
174 | 176 |
"""Update the metadata associated with the object. |
175 | 177 |
|
176 | 178 |
Parameters: |
177 |
'meta': Dictionary with metadata to update.
|
|
179 |
'meta': Dictionary with metadata to update |
|
178 | 180 |
'replace': Replace instead of update |
179 | 181 |
|
180 | 182 |
Raises: |
b/pithos/backends/simple.py | ||
---|---|---|
235 | 235 |
|
236 | 236 |
logger.debug("get_object_permissions: %s %s %s", account, container, name) |
237 | 237 |
path = self._get_objectinfo(account, container, name)[0] |
238 |
return self._get_permissions(path) |
|
238 |
perm_path, perms = self._get_permissions(path) |
|
239 |
if path == perm_path: |
|
240 |
return perms |
|
241 |
return {} |
|
239 | 242 |
|
240 | 243 |
def update_object_permissions(self, user, account, container, name, permissions): |
241 | 244 |
"""Update the permissions associated with the object.""" |
... | ... | |
476 | 479 |
|
477 | 480 |
src_version_id, dest_version_id = self._copy_version(path, path, not replace, True) |
478 | 481 |
for k, v in meta.iteritems(): |
479 |
sql = 'insert or replace into metadata (version_id, key, value) values (?, ?, ?)' |
|
480 |
self.con.execute(sql, (dest_version_id, k, v)) |
|
482 |
if not replace and v == '': |
|
483 |
sql = 'delete from metadata where version_id = ? and key = ?' |
|
484 |
self.con.execute(sql, (dest_version_id, k)) |
|
485 |
else: |
|
486 |
sql = 'insert or replace into metadata (version_id, key, value) values (?, ?, ?)' |
|
487 |
self.con.execute(sql, (dest_version_id, k, v)) |
|
481 | 488 |
self.con.commit() |
482 | 489 |
|
483 | 490 |
def _can_read(self, user, path): |
... | ... | |
510 | 517 |
return r, w |
511 | 518 |
|
512 | 519 |
def _get_permissions(self, path): |
513 |
sql = 'select read, write from permissions where name = ?' |
|
514 |
c = self.con.execute(sql, (path,)) |
|
520 |
# Check for permissions at path or above. |
|
521 |
sql = 'select name, read, write from permissions where ? like name || ?' |
|
522 |
c = self.con.execute(sql, (path, '%')) |
|
515 | 523 |
row = c.fetchone() |
516 | 524 |
if not row: |
517 |
return {} |
|
525 |
return path, {}
|
|
518 | 526 |
|
519 |
r, w = row |
|
527 |
name, r, w = row
|
|
520 | 528 |
if r == '' and w == '': |
521 | 529 |
return {'private': True} |
522 | 530 |
ret = {} |
... | ... | |
524 | 532 |
ret['write'] = w.split(',') |
525 | 533 |
if r != '': |
526 | 534 |
ret['read'] = r.split(',') |
527 |
return ret |
|
535 |
return name, ret
|
|
528 | 536 |
|
529 | 537 |
def _put_permissions(self, path, r, w): |
530 | 538 |
sql = 'insert or replace into permissions (name, read, write) values (?, ?, ?)' |
Also available in: Unified diff