Revision ab2e317e
b/docs/source/devguide.rst | ||
---|---|---|
25 | 25 |
========================= ================================ |
26 | 26 |
Revision Description |
27 | 27 |
========================= ================================ |
28 |
0.5 (July 11, 2011) Object update from another object's data. |
|
28 | 29 |
0.4 (July 01, 2011) Object permissions and account groups. |
29 | 30 |
\ Control versioning behavior and container quotas with container policy directives. |
30 | 31 |
\ Support updating/deleting individual metadata with ``POST``. |
... | ... | |
676 | 677 |
Transfer-Encoding Set to ``chunked`` to specify incremental uploading (if used, ``Content-Length`` is ignored) |
677 | 678 |
Content-Encoding The encoding of the object (optional) |
678 | 679 |
Content-Disposition The presentation style of the object (optional) |
680 |
X-Source-Object Update with data from the object at path ``/<container>/<object>`` (optional, to update) |
|
681 |
X-Source-Version The source version to update from (optional, to update) |
|
679 | 682 |
X-Object-Manifest Object parts prefix in ``<container>/<object>`` form (optional) |
680 | 683 |
X-Object-Sharing Object permissions (optional) |
681 | 684 |
X-Object-Public Object is publicly accessible (optional) |
... | ... | |
688 | 691 |
|
689 | 692 |
To update an object's data: |
690 | 693 |
|
691 |
* Set ``Content-Type`` to ``application/octet-stream``. If ``Content-Type`` has some other value, it will be ignored and only the metadata will be updated.
|
|
692 |
* Supply ``Content-Length`` (except if using chunked transfers), ``Content-Type`` and ``Content-Range`` headers.
|
|
694 |
* Either set ``Content-Type`` to ``application/octet-stream``, or provide an object with ``X-Source-Object``. If ``Content-Type`` has some other value, it will be ignored and only the metadata will be updated.
|
|
695 |
* If the data is supplied in the request (using ``Content-Type`` instead of ``X-Source-Object``), a valid ``Content-Length`` header is required - except if using chunked transfers (set ``Transfer-Encoding`` to ``chunked``).
|
|
693 | 696 |
* Set ``Content-Range`` as specified in RFC2616, with the following differences: |
694 | 697 |
|
695 | 698 |
* Client software MAY omit ``last-byte-pos`` of if the length of the range being transferred is unknown or difficult to determine. |
696 | 699 |
* Client software SHOULD not specify the ``instance-length`` (use a ``*``), unless there is a reason for performing a size check at the server. |
697 |
* If ``Content-Range`` used has a ``byte-range-resp-spec = *``, data supplied will be appended to the object.
|
|
700 |
* If ``Content-Range`` used has a ``byte-range-resp-spec = *``, data will be appended to the object. |
|
698 | 701 |
|
699 | 702 |
A data update will trigger an ETag change. The new ETag will not correspond to the object's MD5 sum (**TBD**) and will be included in reply headers. |
700 | 703 |
|
... | ... | |
767 | 770 |
* Object metadata allowed, in addition to ``X-Object-Meta-*``: ``Content-Encoding``, ``Content-Disposition``, ``X-Object-Manifest``. These are all replaced with every update operation, except if using the ``update`` parameter (in which case individual keys can also be deleted). Deleting meta by providing empty values also works when copying/moving an object. |
768 | 771 |
* Multi-range object GET support as outlined in RFC2616. |
769 | 772 |
* Object hashmap retrieval through GET and the ``format`` parameter. |
770 |
* Partial object updates through POST, using the ``Content-Length``, ``Content-Type``, ``Content-Range`` and ``Transfer-Encoding`` headers. |
|
773 |
* Partial object updates through POST, using the ``Content-Length``, ``Content-Type``, ``Content-Range`` and ``Transfer-Encoding`` headers. Use another object's data to update with ``X-Source-Object`` and ``X-Source-Version``.
|
|
771 | 774 |
* Object ``MOVE`` support. |
772 | 775 |
* Time-variant account/container listings via the ``until`` parameter. |
773 | 776 |
* Object versions - parameter ``version`` in HEAD/GET (list versions with GET), ``X-Object-Version-*`` meta in replies, ``X-Source-Version`` in PUT/COPY. |
... | ... | |
821 | 824 |
|
822 | 825 |
Pithos clients should use the ``pithos`` container for all Pithos objects. Object names use the ``/`` delimiter to impose a hierarchy of folders and files. |
823 | 826 |
|
824 |
At the object level, tags are implemented by managing metadata keys. The client software should allow the user to use any string as a tag (except ``trash``) and then set the corresponding ``X-Object-Meta-<tag>`` key at the server. The API extensions provided, allow for listing all tags in a container and filtering object listings based on one or more tags. The tag list is sufficient for implementing the ``tags`` element, either as a special, virtual folder (as done in the first version of Pithos), or as an application menu. |
|
825 |
|
|
826 |
To manage the deletion of files use the same API and the ``X-Object-Meta-Trash`` key. The string ``trash`` can not be used as a tag. The ``trash`` element should be presented as a folder, although with no hierarchy. |
|
827 |
At the object level, tags are implemented by managing metadata keys. The client software should allow the user to use any string as a tag and then set the corresponding ``X-Object-Meta-<tag>`` key at the server. The API extensions provided, allow for listing all tags in a container and filtering object listings based on one or more tags. The tag list is sufficient for implementing the ``tags`` element, either as a special, virtual folder (as done in the first version of Pithos), or as an application menu. |
|
827 | 828 |
|
828 | 829 |
The metadata specification is summarized in the following table. |
829 | 830 |
|
830 | 831 |
=========================== ============================== |
831 | 832 |
Metadata Name Value |
832 | 833 |
=========================== ============================== |
833 |
X-Object-Meta-Trash Set to ``true`` if the object has been moved to the trash |
|
834 | 834 |
X-Object-Meta-* Use for other tags that apply to the object |
835 | 835 |
=========================== ============================== |
836 | 836 |
|
... | ... | |
891 | 891 |
|
892 | 892 |
curl -X GET -D - \ |
893 | 893 |
-H "X-Auth-Token: 0000" \ |
894 |
https://pithos.dev.grnet.gr/v1/user/pithos?meta=trash |
|
895 |
|
|
896 |
This is the recommended way of tagging/retrieving objects in trash. |
|
894 |
https://pithos.dev.grnet.gr/v1/user/pithos?meta=favorites |
|
897 | 895 |
|
898 | 896 |
* Retrieve an object :: |
899 | 897 |
|
... | ... | |
969 | 967 |
curl -X DELETE -D - \ |
970 | 968 |
-H "X-Auth-Token: 0000" \ |
971 | 969 |
https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt |
972 |
|
b/pithos/api/functions.py | ||
---|---|---|
597 | 597 |
copy_from = request.META.get('HTTP_X_COPY_FROM') |
598 | 598 |
move_from = request.META.get('HTTP_X_MOVE_FROM') |
599 | 599 |
if copy_from or move_from: |
600 |
# TODO: Why is this required? Copy this ammount? |
|
601 |
content_length = get_content_length(request) |
|
600 |
content_length = get_content_length(request) # Required by the API. |
|
602 | 601 |
|
603 | 602 |
if move_from: |
604 | 603 |
try: |
... | ... | |
745 | 744 |
if k in prev_meta: |
746 | 745 |
meta[k] = prev_meta[k] |
747 | 746 |
|
748 |
# A Content-Type header indicates data updates. |
|
749 |
if not content_type or content_type != 'application/octet-stream': |
|
747 |
# A Content-Type or X-Source-Object header indicates data updates. |
|
748 |
src_object = request.META.get('HTTP_X_SOURCE_OBJECT') |
|
749 |
if (not content_type or content_type != 'application/octet-stream') and not src_object: |
|
750 | 750 |
# Do permissions first, as it may fail easier. |
751 | 751 |
if permissions is not None: |
752 | 752 |
try: |
... | ... | |
783 | 783 |
ranges = get_content_range(request) |
784 | 784 |
if not ranges: |
785 | 785 |
raise RangeNotSatisfiable('Invalid Content-Range header') |
786 |
# Require either a Content-Length, or 'chunked' Transfer-Encoding. |
|
787 |
content_length = -1 |
|
788 |
if request.META.get('HTTP_TRANSFER_ENCODING') != 'chunked': |
|
789 |
content_length = get_content_length(request) |
|
790 | 786 |
|
791 | 787 |
try: |
792 | 788 |
size, hashmap = backend.get_object_hashmap(request.user, v_account, v_container, v_object) |
... | ... | |
800 | 796 |
offset = size |
801 | 797 |
elif offset > size: |
802 | 798 |
raise RangeNotSatisfiable('Supplied offset is beyond object limits') |
803 |
if length is None or content_length == -1: |
|
804 |
length = content_length # Nevermind the error. |
|
805 |
elif length != content_length: |
|
806 |
raise BadRequest('Content length does not match range length') |
|
799 |
if src_object: |
|
800 |
src_container, src_name = split_container_object_string(src_object) |
|
801 |
src_version = request.META.get('HTTP_X_SOURCE_VERSION') |
|
802 |
try: |
|
803 |
src_size, src_hashmap = backend.get_object_hashmap(request.user, v_account, src_container, src_name, src_version) |
|
804 |
except NotAllowedError: |
|
805 |
raise Unauthorized('Access denied') |
|
806 |
except NameError: |
|
807 |
raise ItemNotFound('Source object does not exist') |
|
808 |
|
|
809 |
if length is None: |
|
810 |
length = src_size |
|
811 |
elif length > src_size: |
|
812 |
raise BadRequest('Object length is smaller than range length') |
|
813 |
else: |
|
814 |
# Require either a Content-Length, or 'chunked' Transfer-Encoding. |
|
815 |
content_length = -1 |
|
816 |
if request.META.get('HTTP_TRANSFER_ENCODING') != 'chunked': |
|
817 |
content_length = get_content_length(request) |
|
818 |
|
|
819 |
if length is None: |
|
820 |
length = content_length |
|
821 |
else: |
|
822 |
if content_length == -1: |
|
823 |
# TODO: Get up to length bytes in chunks. |
|
824 |
length = content_length |
|
825 |
elif length != content_length: |
|
826 |
raise BadRequest('Content length does not match range length') |
|
807 | 827 |
if total is not None and (total != size or offset >= size or (length > 0 and offset + length >= size)): |
808 | 828 |
raise RangeNotSatisfiable('Supplied range will change provided object limits') |
809 | 829 |
|
810 |
sock = raw_input_socket(request) |
|
811 |
data = '' |
|
812 |
for d in socket_read_iterator(sock, length, backend.block_size): |
|
813 |
# TODO: Raise 408 (Request Timeout) if this takes too long. |
|
814 |
# TODO: Raise 499 (Client Disconnect) if a length is defined and we stop before getting this much data. |
|
815 |
data += d |
|
816 |
bytes = put_object_block(hashmap, data, offset) |
|
817 |
offset += bytes |
|
818 |
data = data[bytes:] |
|
819 |
if len(data) > 0: |
|
820 |
put_object_block(hashmap, data, offset) |
|
830 |
if src_object: |
|
831 |
if offset % backend.block_size == 0: |
|
832 |
# Update the hashes only. |
|
833 |
sbi = 0 |
|
834 |
while length > 0: |
|
835 |
bi = int(offset / backend.block_size) |
|
836 |
bl = min(length, backend.block_size) |
|
837 |
if bi < len(hashmap): |
|
838 |
if bl == backend.block_size: |
|
839 |
hashmap[bi] = src_hashmap[sbi] |
|
840 |
else: |
|
841 |
data = backend.get_block(src_hashmap[sbi]) |
|
842 |
hashmap[bi] = backend.update_block(hashmap[bi], data[:bl], 0) |
|
843 |
else: |
|
844 |
hashmap.append(src_hashmap[sbi]) |
|
845 |
offset += bl |
|
846 |
length -= bl |
|
847 |
sbi += 1 |
|
848 |
else: |
|
849 |
data = '' |
|
850 |
sbi = 0 |
|
851 |
while length > 0: |
|
852 |
data += backend.get_block(src_hashmap[sbi]) |
|
853 |
if length < backend.block_size: |
|
854 |
data = data[:length] |
|
855 |
bytes = put_object_block(hashmap, data, offset) |
|
856 |
offset += bytes |
|
857 |
data = data[bytes:] |
|
858 |
length -= bytes |
|
859 |
sbi += 1 |
|
860 |
else: |
|
861 |
sock = raw_input_socket(request) |
|
862 |
data = '' |
|
863 |
for d in socket_read_iterator(sock, length, backend.block_size): |
|
864 |
# TODO: Raise 408 (Request Timeout) if this takes too long. |
|
865 |
# TODO: Raise 499 (Client Disconnect) if a length is defined and we stop before getting this much data. |
|
866 |
data += d |
|
867 |
bytes = put_object_block(hashmap, data, offset) |
|
868 |
offset += bytes |
|
869 |
data = data[bytes:] |
|
870 |
if len(data) > 0: |
|
871 |
put_object_block(hashmap, data, offset) |
|
821 | 872 |
|
822 | 873 |
if offset > size: |
823 | 874 |
size = offset |
b/pithos/api/util.py | ||
---|---|---|
263 | 263 |
raise Conflict(json.dumps(e.data)) |
264 | 264 |
if public is not None: |
265 | 265 |
try: |
266 |
backend.update_object_public(request.user, v_account, v_container, v_object, public)
|
|
266 |
backend.update_object_public(request.user, v_account, dest_container, dest_name, public)
|
|
267 | 267 |
except NotAllowedError: |
268 | 268 |
raise Unauthorized('Access denied') |
269 | 269 |
except NameError: |
Also available in: Unified diff