Revision cb146cf9
b/docs/source/devguide.rst | ||
---|---|---|
295 | 295 |
x_object_version The object's version identifier |
296 | 296 |
x_object_version_timestamp The object's version timestamp |
297 | 297 |
x_object_manifest Object parts prefix in ``<container>/<object>`` form (optional) |
298 |
x_object_public Object is publicly accessible (optional) |
|
298 |
x_object_public Object is publicly accessible (optional) (**TBD**)
|
|
299 | 299 |
x_object_meta_* Optional user defined metadata |
300 | 300 |
========================== ====================================== |
301 | 301 |
|
... | ... | |
413 | 413 |
X-Object-Version The object's version identifier |
414 | 414 |
X-Object-Version-Timestamp The object's version timestamp |
415 | 415 |
X-Object-Manifest Object parts prefix in ``<container>/<object>`` form (optional) |
416 |
X-Object-Public Object is publicly accessible (optional) |
|
416 |
X-Object-Public Object is publicly accessible (optional) (**TBD**)
|
|
417 | 417 |
X-Object-Meta-* Optional user defined metadata |
418 | 418 |
========================== =============================== |
419 | 419 |
|
... | ... | |
422 | 422 |
================ =============================== |
423 | 423 |
Return Code Description |
424 | 424 |
================ =============================== |
425 |
204 (No Content) The request succeeded
|
|
425 |
200 (No Content) The request succeeded
|
|
426 | 426 |
================ =============================== |
427 | 427 |
|
428 | 428 |
|
... | ... | |
502 | 502 |
X-Object-Version The object's version identifier |
503 | 503 |
X-Object-Version-Timestamp The object's version timestamp |
504 | 504 |
X-Object-Manifest Object parts prefix in ``<container>/<object>`` form (optional) |
505 |
X-Object-Public Object is publicly accessible (optional) |
|
505 |
X-Object-Public Object is publicly accessible (optional) (**TBD**)
|
|
506 | 506 |
X-Object-Meta-* Optional user defined metadata |
507 | 507 |
========================== =============================== |
508 | 508 |
|
... | ... | |
531 | 531 |
Transfer-Encoding Set to ``chunked`` to specify incremental uploading (if used, ``Content-Length`` is ignored) |
532 | 532 |
X-Copy-From The source path in the form ``/<container>/<object>`` |
533 | 533 |
X-Move-From The source path in the form ``/<container>/<object>`` |
534 |
X-Source-Version The source version to copy/move from
|
|
534 |
X-Source-Version The source version to copy from |
|
535 | 535 |
Content-Encoding The encoding of the object (optional) |
536 | 536 |
Content-Disposition The presentation style of the object (optional) |
537 | 537 |
X-Object-Manifest Object parts prefix in ``<container>/<object>`` form (optional) |
538 |
X-Object-Public Object is publicly accessible (optional) |
|
538 |
X-Object-Public Object is publicly accessible (optional) (**TBD**)
|
|
539 | 539 |
X-Object-Meta-* Optional user defined metadata |
540 | 540 |
==================== ================================ |
541 | 541 |
|
... | ... | |
568 | 568 |
Content-Type The MIME content type of the object (optional) |
569 | 569 |
Content-Encoding The encoding of the object (optional) |
570 | 570 |
Content-Disposition The presentation style of the object (optional) |
571 |
X-Source-Version The source version to copy/move from
|
|
571 |
X-Source-Version The source version to copy from |
|
572 | 572 |
X-Object-Manifest Object parts prefix in ``<container>/<object>`` form (optional) |
573 |
X-Object-Public Object is publicly accessible (optional) |
|
573 |
X-Object-Public Object is publicly accessible (optional) (**TBD**)
|
|
574 | 574 |
X-Object-Meta-* Optional user defined metadata |
575 | 575 |
==================== ================================ |
576 | 576 |
|
... | ... | |
586 | 586 |
MOVE |
587 | 587 |
"""" |
588 | 588 |
|
589 |
Same as ``COPY``. |
|
589 |
Same as ``COPY``, without the ``X-Source-Version`` request header. The ``MOVE`` operation is always applied on the latest version.
|
|
590 | 590 |
|
591 | 591 |
|
592 | 592 |
POST |
... | ... | |
602 | 602 |
Content-Encoding The encoding of the object (optional) |
603 | 603 |
Content-Disposition The presentation style of the object (optional) |
604 | 604 |
X-Object-Manifest Object parts prefix in ``<container>/<object>`` form (optional) |
605 |
X-Object-Public Object is publicly accessible (optional) |
|
605 |
X-Object-Public Object is publicly accessible (optional) (**TBD**)
|
|
606 | 606 |
X-Object-Meta-* Optional user defined metadata |
607 | 607 |
==================== ================================ |
608 | 608 |
|
609 |
The ``Content-Encoding``, ``Content-Disposition``, ``X-Object-Manifest``, ``X-Object-Public`` and ``X-Object-Meta-*`` headers are considered to be user defined metadata. The update operation will overwrite all previous values and remove any keys not supplied. |
|
609 |
The ``Content-Encoding``, ``Content-Disposition``, ``X-Object-Manifest``, ``X-Object-Public`` (**TBD**) and ``X-Object-Meta-*`` headers are considered to be user defined metadata. The update operation will overwrite all previous values and remove any keys not supplied.
|
|
610 | 610 |
|
611 | 611 |
To update an object: |
612 | 612 |
|
... | ... | |
636 | 636 |
202 (Accepted) The request has been accepted (not a data update) |
637 | 637 |
204 (No Content) The request succeeded (data updated) |
638 | 638 |
411 (Length Required) Missing ``Content-Length`` in the request |
639 |
416 (Range Not Satisfiable) The supplied range is out of limits or invalid size (**TBD**)
|
|
639 |
416 (Range Not Satisfiable) The supplied range is out of limits or invalid size |
|
640 | 640 |
=========================== ============================== |
641 | 641 |
|
642 | 642 |
|
... | ... | |
656 | 656 |
Public Objects |
657 | 657 |
^^^^^^^^^^^^^^ |
658 | 658 |
|
659 |
Objects that are marked as public, via the ``X-Object-Public`` meta, are also available at the corresponding URI ``https://hostname/public/<account>/<container>/<object>`` for ``HEAD`` or ``GET``. Requests for public objects do not need to include an ``X-Auth-Token``. Pithos will ignore request parameters and only include the following headers in the reply (all ``X-Object-*`` meta is hidden). |
|
659 |
Objects that are marked as public, via the ``X-Object-Public`` meta (**TBD**), are also available at the corresponding URI ``https://hostname/public/<account>/<container>/<object>`` for ``HEAD`` or ``GET``. Requests for public objects do not need to include an ``X-Auth-Token``. Pithos will ignore request parameters and only include the following headers in the reply (all ``X-Object-*`` meta is hidden).
|
|
660 | 660 |
|
661 | 661 |
========================== =============================== |
662 | 662 |
Reply Header Name Value |
... | ... | |
682 | 682 |
* All metadata replies, at all levels, include latest modification information. |
683 | 683 |
* At all levels, a ``GET`` request may use ``If-Modified-Since`` and ``If-Unmodified-Since`` headers. |
684 | 684 |
* Container/object lists include all associated metadata if the reply is of type json/xml. Some names are kept to their OOS API equivalents for compatibility. |
685 |
* Object metadata allowed, in addition to ``X-Object-Meta-*``: ``Content-Encoding``, ``Content-Disposition``, ``X-Object-Manifest``, ``X-Object-Public``. These are all replaced with every update operation. |
|
685 |
* Object metadata allowed, in addition to ``X-Object-Meta-*``: ``Content-Encoding``, ``Content-Disposition``, ``X-Object-Manifest``, ``X-Object-Public`` (**TBD**). These are all replaced with every update operation.
|
|
686 | 686 |
* Multi-range object GET support as outlined in RFC2616. |
687 | 687 |
* Object hashmap retrieval through GET and the ``format`` parameter. |
688 | 688 |
* Partial object updates through POST, using the ``Content-Length``, ``Content-Type``, ``Content-Range`` and ``Transfer-Encoding`` headers. |
689 | 689 |
* Object ``MOVE`` support. |
690 | 690 |
* Time-variant account/container listings via the ``until`` parameter. |
691 |
* Object versions - parameter ``version`` in HEAD/GET (list versions with GET), ``X-Object-Version-*`` meta in replies, ``X-Source-Version`` in PUT/COPY/MOVE.
|
|
692 |
* Publicly accessible objects via ``https://hostname/public``. Control with ``X-Object-Public``. |
|
691 |
* Object versions - parameter ``version`` in HEAD/GET (list versions with GET), ``X-Object-Version-*`` meta in replies, ``X-Source-Version`` in PUT/COPY. |
|
692 |
* Publicly accessible objects via ``https://hostname/public``. Control with ``X-Object-Public`` (**TBD**).
|
|
693 | 693 |
* Large object support with ``X-Object-Manifest``. |
694 | 694 |
|
695 | 695 |
Clarifications/suggestions: |
b/pithos/api/functions.py | ||
---|---|---|
47 | 47 |
update_manifest_meta, validate_modification_preconditions, validate_matching_preconditions, |
48 | 48 |
split_container_object_string, copy_or_move_object, get_int_parameter, get_content_length, |
49 | 49 |
get_content_range, raw_input_socket, socket_read_iterator, object_data_response, |
50 |
hashmap_hash, api_method) |
|
50 |
put_object_block, hashmap_hash, api_method)
|
|
51 | 51 |
from pithos.backends import backend |
52 | 52 |
|
53 | 53 |
|
... | ... | |
377 | 377 |
|
378 | 378 |
update_manifest_meta(request, v_account, meta) |
379 | 379 |
|
380 |
response = HttpResponse(status=204)
|
|
380 |
response = HttpResponse(status=200)
|
|
381 | 381 |
put_object_meta(response, meta) |
382 | 382 |
return response |
383 | 383 |
|
... | ... | |
393 | 393 |
# notModified (304) |
394 | 394 |
|
395 | 395 |
version = get_int_parameter(request, 'version') |
396 |
if not version: |
|
397 |
version = request.GET.get('version') |
|
396 |
version_list = False |
|
397 |
if version is None and request.GET.get('version') == 'list': |
|
398 |
version_list = True |
|
398 | 399 |
try: |
399 | 400 |
meta = backend.get_object_meta(request.user, v_account, v_container, v_object, version) |
400 | 401 |
except NameError: |
... | ... | |
414 | 415 |
return response |
415 | 416 |
|
416 | 417 |
# Reply with the version list. |
417 |
if version == 'list':
|
|
418 |
if version_list:
|
|
418 | 419 |
if request.serialization == 'text': |
419 | 420 |
raise BadRequest('No format specified for version list.') |
420 | 421 |
|
... | ... | |
634 | 635 |
offset, length, total = ranges |
635 | 636 |
if offset is None: |
636 | 637 |
offset = size |
638 |
elif offset > size: |
|
639 |
raise RangeNotSatisfiable('Supplied offset is beyond object limits') |
|
637 | 640 |
if length is None or content_length == -1: |
638 | 641 |
length = content_length # Nevermind the error. |
639 | 642 |
elif length != content_length: |
... | ... | |
647 | 650 |
# TODO: Raise 408 (Request Timeout) if this takes too long. |
648 | 651 |
# TODO: Raise 499 (Client Disconnect) if a length is defined and we stop before getting this much data. |
649 | 652 |
data += d |
650 |
bi = int(offset / backend.block_size) |
|
651 |
bo = offset % backend.block_size |
|
652 |
bl = min(len(data), backend.block_size - bo) |
|
653 |
offset += bl |
|
654 |
h = backend.update_block(hashmap[bi], data[:bl], bo) |
|
655 |
if bi < len(hashmap): |
|
656 |
hashmap[bi] = h |
|
657 |
else: |
|
658 |
hashmap.append(h) |
|
659 |
data = data[bl:] |
|
653 |
bytes = put_object_block(hashmap, data, offset) |
|
654 |
offset += bytes |
|
655 |
data = data[bytes:] |
|
660 | 656 |
if len(data) > 0: |
661 |
bi = int(offset / backend.block_size) |
|
662 |
offset += len(data) |
|
663 |
h = backend.update_block(hashmap[bi], data) |
|
664 |
if bi < len(hashmap): |
|
665 |
hashmap[bi] = h |
|
666 |
else: |
|
667 |
hashmap.append(h) |
|
657 |
put_object_block(hashmap, data, offset) |
|
668 | 658 |
|
669 | 659 |
if offset > size: |
670 | 660 |
size = offset |
b/pithos/api/util.py | ||
---|---|---|
143 | 143 |
if not public: |
144 | 144 |
response['X-Object-Version'] = meta['version'] |
145 | 145 |
response['X-Object-Version-Timestamp'] = meta['version_timestamp'] |
146 |
response['X-Object-Size'] = meta['bytes'] |
|
147 | 146 |
for k in [x for x in meta.keys() if x.startswith('X-Object-Meta-')]: |
148 | 147 |
response[k.encode('utf-8')] = meta[k].encode('utf-8') |
149 | 148 |
for k in ('Content-Encoding', 'Content-Disposition', 'X-Object-Manifest', 'X-Object-Public'): |
... | ... | |
233 | 232 |
if k in src_meta: |
234 | 233 |
meta[k] = src_meta[k] |
235 | 234 |
|
236 |
src_version = request.META.get('HTTP_X_SOURCE_VERSION') |
|
237 | 235 |
try: |
238 | 236 |
if move: |
239 |
backend.move_object(request.user, v_account, src_container, src_name, dest_container, dest_name, meta, True, src_version)
|
|
237 |
backend.move_object(request.user, v_account, src_container, src_name, dest_container, dest_name, meta, True) |
|
240 | 238 |
else: |
239 |
src_version = request.META.get('HTTP_X_SOURCE_VERSION') |
|
241 | 240 |
backend.copy_object(request.user, v_account, src_container, src_name, dest_container, dest_name, meta, True, src_version) |
242 | 241 |
except NameError: |
243 | 242 |
raise ItemNotFound('Container or object does not exist') |
... | ... | |
528 | 527 |
response['Content-Type'] = 'multipart/byteranges; boundary=%s' % (boundary,) |
529 | 528 |
return response |
530 | 529 |
|
530 |
def put_object_block(hashmap, data, offset): |
|
531 |
"""Put one block of data at the given offset.""" |
|
532 |
|
|
533 |
bi = int(offset / backend.block_size) |
|
534 |
bo = offset % backend.block_size |
|
535 |
bl = min(len(data), backend.block_size - bo) |
|
536 |
if bi < len(hashmap): |
|
537 |
hashmap[bi] = backend.update_block(hashmap[bi], data[:bl], bo) |
|
538 |
else: |
|
539 |
hashmap.append(backend.put_block(('\x00' * bo) + data[:bl])) |
|
540 |
return bl # Return ammount of data written. |
|
541 |
|
|
531 | 542 |
def hashmap_hash(hashmap): |
532 | 543 |
"""Produce the root hash, treating the hashmap as a Merkle-like tree.""" |
533 | 544 |
|
b/pithos/backends/base.py | ||
---|---|---|
213 | 213 |
""" |
214 | 214 |
return |
215 | 215 |
|
216 |
def move_object(self, user, account, src_container, src_name, dest_container, dest_name, dest_meta={}, replace_meta=False, src_version=None):
|
|
216 |
def move_object(self, user, account, src_container, src_name, dest_container, dest_name, dest_meta={}, replace_meta=False): |
|
217 | 217 |
"""Move an object's data and metadata. |
218 | 218 |
|
219 | 219 |
Parameters: |
220 | 220 |
'dest_meta': Dictionary with metadata to changes from source to destination |
221 | 221 |
'replace_meta': Replace metadata instead of update |
222 |
'src_version': Copy from the version provided. |
|
223 | 222 |
|
224 | 223 |
Raises: |
225 | 224 |
NameError: Container/object does not exist |
226 |
IndexError: Version does not exist |
|
227 | 225 |
""" |
228 | 226 |
return |
229 | 227 |
|
b/pithos/backends/simple.py | ||
---|---|---|
272 | 272 |
self.con.execute(sql, (dest_version_id, k, v)) |
273 | 273 |
self.con.commit() |
274 | 274 |
|
275 |
def move_object(self, user, account, src_container, src_name, dest_container, dest_name, dest_meta={}, replace_meta=False, src_version=None):
|
|
275 |
def move_object(self, user, account, src_container, src_name, dest_container, dest_name, dest_meta={}, replace_meta=False): |
|
276 | 276 |
"""Move an object's data and metadata.""" |
277 | 277 |
|
278 |
logger.debug("move_object: %s %s %s %s, %s %s %s %s", account, src_container, src_name, dest_container, dest_name, dest_meta, replace_meta, src_version)
|
|
279 |
self.copy_object(user, account, src_container, src_name, dest_container, dest_name, dest_meta, replace_meta, src_version)
|
|
278 |
logger.debug("move_object: %s %s %s %s %s %s %s", account, src_container, src_name, dest_container, dest_name, dest_meta, replace_meta)
|
|
279 |
self.copy_object(user, account, src_container, src_name, dest_container, dest_name, dest_meta, replace_meta, None)
|
|
280 | 280 |
self.delete_object(user, account, src_container, src_name) |
281 | 281 |
|
282 | 282 |
def delete_object(self, user, account, container, name): |
... | ... | |
353 | 353 |
return int(row[0]), int(row[1]), int(tstamp) |
354 | 354 |
|
355 | 355 |
def _get_version(self, path, version=None): |
356 |
if version in [None, 'list']:
|
|
356 |
if version is None:
|
|
357 | 357 |
sql = '''select version_id, strftime('%s', tstamp), size, hide from versions where name = ? |
358 | 358 |
order by version_id desc limit 1''' |
359 | 359 |
c = self.con.execute(sql, (path,)) |
Also available in: Unified diff