1 Pithos v2 Developer Guide
2 =========================
7 Pithos is a storage service implemented by GRNET (http://www.grnet.gr). Data is stored as objects, organized in containers, belonging to an account. This hierarchy of storage layers has been inspired by the OpenStack Object Storage (OOS) API and similar CloudFiles API by Rackspace. The Pithos API follows the OOS API as closely as possible. One of the design requirements has been to be able to use Pithos with clients built for the OOS, without changes.
9 However, to be able to take full advantage of the Pithos infrastructure, client software should be aware of the extensions that differentiate Pithos from OOS. Pithos objects can be updated, or appended to. Pithos will store sharing permissions per object and enforce corresponding authorization policies. Automatic version management, allows taking account and container listings back in time, as well as reading previous instances of objects.
11 The storage backend of Pithos is block oriented, permitting efficient, deduplicated data placement. The block structure of objects is exposed at the API layer, in order to encourage external software to implement advanced data management operations.
13 This document's goals are:
15 * Define the Pithos ReST API that allows the storage and retrieval of data and metadata via HTTP calls
16 * Specify metadata semantics and user interface guidelines for a common experience across client software implementations
18 The present document is meant to be read alongside the OOS API documentation. Thus, it is suggested that the reader is familiar with associated technologies, the OOS API as well as the first version of the Pithos API. This document refers to the second version of Pithos. Information on the first version of the storage API can be found at http://code.google.com/p/gss.
20 Whatever marked as to be determined (**TBD**), should not be considered by implementors.
25 ========================= ================================
27 ========================= ================================
28 0.6 (Sept 12, 2011) Reply with Merkle hash as the ETag when updating objects.
29 \ Include version id in object replace/change replies.
30 \ Change conflict (409) replies format to text.
31 \ Tags should be migrated to a meta value.
32 \ Container ``PUT`` updates metadata/policy.
33 0.5 (July 22, 2011) Object update from another object's data.
34 \ Support object truncate.
35 \ Create object using a standard HTML form.
36 \ Purge container/object history.
37 \ List other accounts that share objects with a user.
38 \ List shared containers/objects.
39 \ Update implementation guidelines.
40 \ Check preconditions when creating/updating objects.
41 0.4 (July 01, 2011) Object permissions and account groups.
42 \ Control versioning behavior and container quotas with container policy directives.
43 \ Support updating/deleting individual metadata with ``POST``.
44 \ Create object using hashmap.
45 0.3 (June 14, 2011) Large object support with ``X-Object-Manifest``.
46 \ Allow for publicly available objects via ``https://hostname/public``.
47 \ Support time-variant account/container listings.
48 \ Add source version when duplicating with ``PUT``/``COPY``.
49 \ Request version in object ``HEAD``/``GET`` requests (list versions with ``GET``).
50 0.2 (May 31, 2011) Add object meta listing and filtering in containers.
51 \ Include underlying storage characteristics in container meta.
52 \ Support for partial object updates through ``POST``.
53 \ Expose object hashmaps through ``GET``.
54 \ Support for multi-range object ``GET`` requests.
55 0.1 (May 17, 2011) Initial release. Based on OpenStack Object Storage Developer Guide API v1 (Apr. 15, 2011).
56 ========================= ================================
61 The URI requests supported by the Pithos API follow one of the following forms:
63 * Top level: ``https://hostname/v1/``
64 * Account level: ``https://hostname/v1/<account>``
65 * Container level: ``https://hostname/v1/<account>/<container>``
66 * Object level: ``https://hostname/v1/<account>/<container>/<object>``
68 All requests must include an ``X-Auth-Token`` - as a header, or a parameter. The process of obtaining the token is still to be determined (**TBD**).
70 The allowable request operations and respective return codes per level are presented in the remainder of this chapter. Common to all requests are the following return codes.
72 ========================= ================================
73 Return Code Description
74 ========================= ================================
75 400 (Bad Request) The request is invalid
76 401 (Unauthorized) Request not allowed
77 404 (Not Found) The requested resource was not found
78 503 (Service Unavailable) The request cannot be completed because of an internal error
79 ========================= ================================
86 ========= ==================
88 ========= ==================
89 GET Authentication (for compatibility with the OOS API) or list allowed accounts
90 ========= ==================
95 If the ``X-Auth-User`` and ``X-Auth-Key`` headers are given, a dummy ``X-Auth-Token`` and ``X-Storage-Url`` will be replied, which can be used as a guest token/namespace for testing Pithos.
97 ================ =====================
98 Return Code Description
99 ================ =====================
100 204 (No Content) The request succeeded
101 ================ =====================
103 If an ``X-Auth-Token`` is already present, the operation will be interpreted as a request to list other accounts that share objects to the user.
105 ====================== =========================
106 Request Parameter Name Value
107 ====================== =========================
108 limit The amount of results requested (default is 10000)
109 marker Return containers with name lexicographically after marker
110 format Optional extended reply type (can be ``json`` or ``xml``)
111 ====================== =========================
113 The reply is a list of account names.
114 If a ``format=xml`` or ``format=json`` argument is given, extended information on the accounts will be returned, serialized in the chosen format.
115 For each account, the information will include the following (names will be in lower case and with hyphens replaced with underscores):
117 =========================== ============================
119 =========================== ============================
120 name The name of the account
121 last_modified The last account modification date (regardless of ``until``)
122 =========================== ============================
124 Example ``format=json`` reply:
128 [{"name": "user", "last_modified": "2011-07-19T10:48:16"}, ...]
130 Example ``format=xml`` reply:
134 <?xml version="1.0" encoding="UTF-8"?>
138 <last_modified>2011-07-19T10:48:16</last_modified>
140 <account>...</account>
143 =========================== =====================
144 Return Code Description
145 =========================== =====================
146 200 (OK) The request succeeded
147 204 (No Content) The user has no access to other accounts (only for non-extended replies)
148 =========================== =====================
150 Will use a ``200`` return code if the reply is of type json/xml.
157 ========= ==================
158 Operation Description
159 ========= ==================
160 HEAD Retrieve account metadata
162 POST Update account metadata
163 ========= ==================
168 ==================== ===========================
169 Request Header Name Value
170 ==================== ===========================
171 If-Modified-Since Retrieve if account has changed since provided timestamp
172 If-Unmodified-Since Retrieve if account has not changed since provided timestamp
173 ==================== ===========================
177 ====================== ===================================
178 Request Parameter Name Value
179 ====================== ===================================
180 until Optional timestamp
181 ====================== ===================================
183 Cross-user requests are not allowed to use ``until`` and only include the account modification date in the reply.
185 ========================== =====================
186 Reply Header Name Value
187 ========================== =====================
188 X-Account-Container-Count The total number of containers
189 X-Account-Object-Count The total number of objects (**TBD**)
190 X-Account-Bytes-Used The total number of bytes stored
191 X-Account-Bytes-Remaining The total number of bytes remaining (**TBD**)
192 X-Account-Last-Login The last login (**TBD**)
193 X-Account-Until-Timestamp The last account modification date until the timestamp provided
194 X-Account-Group-* Optional user defined groups
195 X-Account-Meta-* Optional user defined metadata
196 Last-Modified The last account modification date (regardless of ``until``)
197 ========================== =====================
201 ================ =====================
202 Return Code Description
203 ================ =====================
204 204 (No Content) The request succeeded
205 ================ =====================
211 ==================== ===========================
212 Request Header Name Value
213 ==================== ===========================
214 If-Modified-Since Retrieve if account has changed since provided timestamp
215 If-Unmodified-Since Retrieve if account has not changed since provided timestamp
216 ==================== ===========================
220 ====================== =========================
221 Request Parameter Name Value
222 ====================== =========================
223 limit The amount of results requested (default is 10000)
224 marker Return containers with name lexicographically after marker
225 format Optional extended reply type (can be ``json`` or ``xml``)
226 shared Show only shared containers (no value parameter)
227 until Optional timestamp
228 ====================== =========================
230 The reply is a list of container names. Account headers (as in a ``HEAD`` request) will also be included.
231 Cross-user requests are not allowed to use ``until`` and only include the account/container modification dates in the reply.
233 If a ``format=xml`` or ``format=json`` argument is given, extended information on the containers will be returned, serialized in the chosen format.
234 For each container, the information will include all container metadata (names will be in lower case and with hyphens replaced with underscores):
236 =========================== ============================
238 =========================== ============================
239 name The name of the container
240 count The number of objects inside the container
241 bytes The total size of the objects inside the container
242 last_modified The last container modification date (regardless of ``until``)
243 x_container_until_timestamp The last container modification date until the timestamp provided
244 x_container_policy_* Container behavior and limits
245 x_container_meta_* Optional user defined metadata
246 =========================== ============================
248 For examples of container details returned in JSON/XML formats refer to the OOS API documentation.
250 =========================== =====================
251 Return Code Description
252 =========================== =====================
253 200 (OK) The request succeeded
254 204 (No Content) The account has no containers (only for non-extended replies)
255 304 (Not Modified) The account has not been modified
256 412 (Precondition Failed) The condition set can not be satisfied
257 =========================== =====================
259 Will use a ``200`` return code if the reply is of type json/xml.
265 ==================== ===========================
266 Request Header Name Value
267 ==================== ===========================
268 X-Account-Group-* Optional user defined groups
269 X-Account-Meta-* Optional user defined metadata
270 ==================== ===========================
274 ====================== ============================================
275 Request Parameter Name Value
276 ====================== ============================================
277 update Do not replace metadata/groups (no value parameter)
278 ====================== ============================================
280 No reply content/headers.
282 The operation will overwrite all user defined metadata, except if ``update`` is defined.
283 To create a group, include an ``X-Account-Group-*`` header with the name in the key and a comma separated list of user names in the value. If no ``X-Account-Group-*`` header is present, no changes will be applied to groups. The ``update`` parameter also applies to groups. To delete a specific group, use ``update`` and an empty header value.
285 ================ ===============================
286 Return Code Description
287 ================ ===============================
288 202 (Accepted) The request has been accepted
289 ================ ===============================
297 ========= ============================
298 Operation Description
299 ========= ============================
300 HEAD Retrieve container metadata
302 PUT Create/update container
303 POST Update container metadata
304 DELETE Delete container
305 ========= ============================
311 ==================== ===========================
312 Request Header Name Value
313 ==================== ===========================
314 If-Modified-Since Retrieve if container has changed since provided timestamp
315 If-Unmodified-Since Retrieve if container has not changed since provided timestamp
316 ==================== ===========================
320 ====================== ===================================
321 Request Parameter Name Value
322 ====================== ===================================
323 until Optional timestamp
324 ====================== ===================================
326 Cross-user requests are not allowed to use ``until`` and only include the container modification date in the reply.
328 =========================== ===============================
329 Reply Header Name Value
330 =========================== ===============================
331 X-Container-Object-Count The total number of objects in the container
332 X-Container-Bytes-Used The total number of bytes of all objects stored
333 X-Container-Block-Size The block size used by the storage backend
334 X-Container-Block-Hash The hash algorithm used for block identifiers in object hashmaps
335 X-Container-Until-Timestamp The last container modification date until the timestamp provided
336 X-Container-Object-Meta A list with all meta keys used by objects (**TBD**)
337 X-Container-Policy-* Container behavior and limits
338 X-Container-Meta-* Optional user defined metadata
339 Last-Modified The last container modification date (regardless of ``until``)
340 =========================== ===============================
342 The keys returned in ``X-Container-Object-Meta`` are all the unique strings after the ``X-Object-Meta-`` prefix, formatted as a comma-separated list. See container ``PUT`` for a reference of policy directives. (**TBD**)
344 ================ ===============================
345 Return Code Description
346 ================ ===============================
347 204 (No Content) The request succeeded
348 ================ ===============================
354 ==================== ===========================
355 Request Header Name Value
356 ==================== ===========================
357 If-Modified-Since Retrieve if container has changed since provided timestamp
358 If-Unmodified-Since Retrieve if container has not changed since provided timestamp
359 ==================== ===========================
363 ====================== ===================================
364 Request Parameter Name Value
365 ====================== ===================================
366 limit The amount of results requested (default is 10000)
367 marker Return containers with name lexicographically after marker
368 prefix Return objects starting with prefix
369 delimiter Return objects up to the delimiter (discussion follows)
370 path Assume ``prefix=path`` and ``delimiter=/``
371 format Optional extended reply type (can be ``json`` or ``xml``)
372 meta Return objects having the specified meta keys (can be a comma separated list)
373 shared Show only shared objects (no value parameter)
374 until Optional timestamp
375 ====================== ===================================
377 The ``path`` parameter overrides ``prefix`` and ``delimiter``. When using ``path``, results will include objects ending in ``delimiter``.
379 The keys given with ``meta`` will be matched with the strings after the ``X-Object-Meta-`` prefix.
381 The reply is a list of object names. Container headers (as in a ``HEAD`` request) will also be included.
382 Cross-user requests are not allowed to use ``until`` and include the following limited set of headers in the reply:
384 =========================== ===============================
385 Reply Header Name Value
386 =========================== ===============================
387 X-Container-Block-Size The block size used by the storage backend
388 X-Container-Block-Hash The hash algorithm used for block identifiers in object hashmaps
389 X-Container-Object-Meta A list with all meta keys used by allowed objects (**TBD**)
390 Last-Modified The last container modification date
391 =========================== ===============================
393 If a ``format=xml`` or ``format=json`` argument is given, extended information on the objects will be returned, serialized in the chosen format.
394 For each object, the information will include all object metadata (names will be in lower case and with hyphens replaced with underscores):
396 ========================== ======================================
398 ========================== ======================================
399 name The name of the object
400 hash The ETag of the object
401 bytes The size of the object
402 content_type The MIME content type of the object
403 content_encoding The encoding of the object (optional)
404 content-disposition The presentation style of the object (optional)
405 last_modified The last object modification date (regardless of version)
406 x_object_version The object's version identifier
407 x_object_version_timestamp The object's version timestamp
408 x_object_modified_by The user that committed the object's version
409 x_object_manifest Object parts prefix in ``<container>/<object>`` form (optional)
410 x_object_sharing Object permissions (optional)
411 x_object_shared_by Object inheriting permissions (optional)
412 x_object_public Object's publicly accessible URI (optional)
413 x_object_meta_* Optional user defined metadata
414 ========================== ======================================
416 Extended replies may also include virtual directory markers in separate sections of the ``json`` or ``xml`` results.
417 Virtual directory markers are only included when ``delimiter`` is explicitly set. They correspond to the substrings up to and including the first occurrence of the delimiter.
418 In JSON results they appear as dictionaries with only a ``"subdir"`` key. In XML results they appear interleaved with ``<object>`` tags as ``<subdir name="..." />``.
419 In case there is an object with the same name as a virtual directory marker, the object will be returned.
421 For examples of object details returned in JSON/XML formats refer to the OOS API documentation.
423 =========================== ===============================
424 Return Code Description
425 =========================== ===============================
426 200 (OK) The request succeeded
427 204 (No Content) The account has no containers (only for non-extended replies)
428 304 (Not Modified) The container has not been modified
429 412 (Precondition Failed) The condition set can not be satisfied
430 =========================== ===============================
432 Will use a ``200`` return code if the reply is of type json/xml.
438 ==================== ================================
439 Request Header Name Value
440 ==================== ================================
441 X-Container-Policy-* Container behavior and limits
442 X-Container-Meta-* Optional user defined metadata
443 ==================== ================================
445 No reply content/headers.
447 If no policy is defined, the container will be created with the default values.
448 Available policy directives:
450 * ``versioning``: Set to ``auto``, ``manual`` or ``none`` (default is ``manual``)
451 * ``quota``: Size limit in KB (default is ``0`` - unlimited)
453 If the container already exists, the operation is equal to a ``POST`` with ``update`` defined.
455 ================ ===============================
456 Return Code Description
457 ================ ===============================
458 201 (Created) The container has been created
459 202 (Accepted) The request has been accepted
460 ================ ===============================
466 ==================== ================================
467 Request Header Name Value
468 ==================== ================================
469 X-Container-Policy-* Container behavior and limits
470 X-Container-Meta-* Optional user defined metadata
471 ==================== ================================
475 ====================== ============================================
476 Request Parameter Name Value
477 ====================== ============================================
478 update Do not replace metadata/policy (no value parameter)
479 ====================== ============================================
481 No reply content/headers.
483 The operation will overwrite all user defined metadata, except if ``update`` is defined.
484 To change policy, include an ``X-Container-Policy-*`` header with the name in the key. If no ``X-Container-Policy-*`` header is present, no changes will be applied to policy. The ``update`` parameter also applies to policy - deleted values will revert to defaults. To delete/revert a specific policy directive, use ``update`` and an empty header value. See container ``PUT`` for a reference of policy directives.
486 ================ ===============================
487 Return Code Description
488 ================ ===============================
489 202 (Accepted) The request has been accepted
490 ================ ===============================
496 ====================== ===================================
497 Request Parameter Name Value
498 ====================== ===================================
499 until Optional timestamp
500 ====================== ===================================
502 If ``until`` is defined, the container is "purged" up to that time (the history of all objects up to then is deleted).
504 No reply content/headers.
506 ================ ===============================
507 Return Code Description
508 ================ ===============================
509 204 (No Content) The request succeeded
510 409 (Conflict) The container is not empty
511 ================ ===============================
519 ========= =================================
520 Operation Description
521 ========= =================================
522 HEAD Retrieve object metadata
524 PUT Write object data or copy/move object
527 POST Update object metadata/data
529 ========= =================================
535 ==================== ================================
536 Request Header Name Value
537 ==================== ================================
538 If-Match Retrieve if ETags match
539 If-None-Match Retrieve if ETags don't match
540 If-Modified-Since Retrieve if object has changed since provided timestamp
541 If-Unmodified-Since Retrieve if object has not changed since provided timestamp
542 ==================== ================================
546 ====================== ===================================
547 Request Parameter Name Value
548 ====================== ===================================
549 version Optional version identifier
550 ====================== ===================================
554 ========================== ===============================
555 Reply Header Name Value
556 ========================== ===============================
557 ETag The ETag of the object
558 Content-Length The size of the object
559 Content-Type The MIME content type of the object
560 Last-Modified The last object modification date (regardless of version)
561 Content-Encoding The encoding of the object (optional)
562 Content-Disposition The presentation style of the object (optional)
563 X-Object-Version The object's version identifier
564 X-Object-Version-Timestamp The object's version timestamp
565 X-Object-Modified-By The user that comitted the object's version
566 X-Object-Manifest Object parts prefix in ``<container>/<object>`` form (optional)
567 X-Object-Sharing Object permissions (optional)
568 X-Object-Shared-By Object inheriting permissions (optional)
569 X-Object-Public Object's publicly accessible URI (optional)
570 X-Object-Meta-* Optional user defined metadata
571 ========================== ===============================
575 ================ ===============================
576 Return Code Description
577 ================ ===============================
578 200 (No Content) The request succeeded
579 ================ ===============================
585 ==================== ================================
586 Request Header Name Value
587 ==================== ================================
588 Range Optional range of data to retrieve
589 If-Range Retrieve the missing part if entity is unchanged; otherwise, retrieve the entire new entity (used together with Range header)
590 If-Match Retrieve if ETags match
591 If-None-Match Retrieve if ETags don't match
592 If-Modified-Since Retrieve if object has changed since provided timestamp
593 If-Unmodified-Since Retrieve if object has not changed since provided timestamp
594 ==================== ================================
598 ====================== ===================================
599 Request Parameter Name Value
600 ====================== ===================================
601 format Optional extended reply type (can be ``json`` or ``xml``)
602 version Optional version identifier or ``list`` (specify a format if requesting a list)
603 ====================== ===================================
605 The reply is the object's data (or part of it), except if a hashmap is requested with the ``format`` parameter, or a version list with ``version=list`` (in which case an extended reply format must be specified). Object headers (as in a ``HEAD`` request) are always included.
607 Hashmaps expose the underlying storage format of the object. Note that each hash is computed after trimming trailing null bytes of the corresponding block.
609 Example ``format=json`` reply:
613 {"block_hash": "sha1", "hashes": ["7295c41da03d7f916440b98e32c4a2a39351546c", ...], "block_size": 131072, "bytes": 242}
615 Example ``format=xml`` reply:
619 <?xml version="1.0" encoding="UTF-8"?>
620 <object name="file" bytes="24223726" block_size="131072" block_hash="sha1">
621 <hash>7295c41da03d7f916440b98e32c4a2a39351546c</hash>
625 Version lists include the version identifier and timestamp for each available object version. Version identifiers can be arbitrary strings, so use the timestamp to find newer versions.
627 Example ``format=json`` reply:
631 {"versions": [[23, 1307700892], [28, 1307700898], ...]}
633 Example ``format=xml`` reply:
637 <?xml version="1.0" encoding="UTF-8"?>
639 <version timestamp="1307700892">23</version>
640 <version timestamp="1307700898">28</version>
641 <version timestamp="...">...</version>
644 The ``Range`` header may include multiple ranges, as outlined in RFC2616. Then the ``Content-Type`` of the reply will be ``multipart/byteranges`` and each part will include a ``Content-Range`` header.
646 ========================== ===============================
647 Reply Header Name Value
648 ========================== ===============================
649 ETag The ETag of the object
650 Content-Length The size of the data returned
651 Content-Type The MIME content type of the object
652 Content-Range The range of data included (only on a single range request)
653 Last-Modified The last object modification date (regardless of version)
654 Content-Encoding The encoding of the object (optional)
655 Content-Disposition The presentation style of the object (optional)
656 X-Object-Version The object's version identifier
657 X-Object-Version-Timestamp The object's version timestamp
658 X-Object-Modified-By The user that comitted the object's version
659 X-Object-Manifest Object parts prefix in ``<container>/<object>`` form (optional)
660 X-Object-Sharing Object permissions (optional)
661 X-Object-Shared-By Object inheriting permissions (optional)
662 X-Object-Public Object's publicly accessible URI (optional)
663 X-Object-Meta-* Optional user defined metadata
664 ========================== ===============================
668 =========================== ==============================
669 Return Code Description
670 =========================== ==============================
671 200 (OK) The request succeeded
672 206 (Partial Content) The range request succeeded
673 304 (Not Modified) The object has not been modified
674 412 (Precondition Failed) The condition set can not be satisfied
675 416 (Range Not Satisfiable) The requested range is out of limits
676 =========================== ==============================
682 ==================== ================================
683 Request Header Name Value
684 ==================== ================================
685 If-Match Put if ETags match with current object
686 If-None-Match Put if ETags don't match with current object
687 ETag The MD5 hash of the object (optional to check written data)
688 Content-Length The size of the data written
689 Content-Type The MIME content type of the object
690 Transfer-Encoding Set to ``chunked`` to specify incremental uploading (if used, ``Content-Length`` is ignored)
691 X-Copy-From The source path in the form ``/<container>/<object>``
692 X-Move-From The source path in the form ``/<container>/<object>``
693 X-Source-Version The source version to copy from
694 Content-Encoding The encoding of the object (optional)
695 Content-Disposition The presentation style of the object (optional)
696 X-Object-Manifest Object parts prefix in ``<container>/<object>`` form (optional)
697 X-Object-Sharing Object permissions (optional)
698 X-Object-Public Object is publicly accessible (optional)
699 X-Object-Meta-* Optional user defined metadata
700 ==================== ================================
704 ====================== ===================================
705 Request Parameter Name Value
706 ====================== ===================================
707 format Optional extended request type (can be ``json``) to create the object by suppling its hashmap instead
708 ====================== ===================================
710 The request is the object's data (or part of it), except if a hashmap is provided with the ``format`` parameter. If format is used and all different parts are stored in the server, the object is created, otherwise the server returns Conflict (409) with the list of the missing parts (in a simple text format, with one hash per line).
712 Hashmaps expose the underlying storage format of the object.
714 Example ``format=json`` request:
718 {"block_hash": "sha1", "hashes": ["7295c41da03d7f916440b98e32c4a2a39351546c", ...], "block_size": 131072, "bytes": 242}
720 Example ``format=xml`` request:
724 <?xml version="1.0" encoding="UTF-8"?>
725 <object name="file" bytes="24223726" block_size="131072" block_hash="sha1">
726 <hash>7295c41da03d7f916440b98e32c4a2a39351546c</hash>
730 ========================== ===============================
731 Reply Header Name Value
732 ========================== ===============================
733 ETag The MD5 hash of the object (on create)
734 X-Object-Version The object's new version
735 ========================== ===============================
737 The ``X-Object-Sharing`` header may include either a ``read=...`` comma-separated user/group list, or a ``write=...`` comma-separated user/group list, or both separated by a semicolon (``;``). Groups are specified as ``<account>:<group>``. To publish the object, set ``X-Object-Public`` to ``true``. To unpublish, set to ``false``, or use an empty header value.
739 =========================== ==============================
740 Return Code Description
741 =========================== ==============================
742 201 (Created) The object has been created
743 409 (Conflict) The object can not be created from the provided hashmap, or there are conflicting permissions (a list of missing hashes, or a list of conflicting sharing paths will be included in the reply - in simple text format)
744 411 (Length Required) Missing ``Content-Length`` or ``Content-Type`` in the request
745 422 (Unprocessable Entity) The MD5 checksum of the data written to the storage system does not match the (optionally) supplied ETag value
746 =========================== ==============================
752 ==================== ================================
753 Request Header Name Value
754 ==================== ================================
755 If-Match Proceed if ETags match with object
756 If-None-Match Proceed if ETags don't match with object
757 Destination The destination path in the form ``/<container>/<object>``
758 Content-Type The MIME content type of the object (optional)
759 Content-Encoding The encoding of the object (optional)
760 Content-Disposition The presentation style of the object (optional)
761 X-Source-Version The source version to copy from
762 X-Object-Manifest Object parts prefix in ``<container>/<object>`` form (optional)
763 X-Object-Sharing Object permissions (optional)
764 X-Object-Public Object is publicly accessible (optional)
765 X-Object-Meta-* Optional user defined metadata
766 ==================== ================================
768 Refer to ``PUT``/``POST`` for a description of request headers. Metadata is also copied, updated with any values defined. Sharing/publishing options are not copied.
770 ========================== ===============================
771 Reply Header Name Value
772 ========================== ===============================
773 X-Object-Version The object's new version
774 ========================== ===============================
778 =========================== ==============================
779 Return Code Description
780 =========================== ==============================
781 201 (Created) The object has been created
782 409 (Conflict) There are conflicting permissions (a list of conflicting sharing paths will be included in the reply - in simple text format)
783 =========================== ==============================
789 Same as ``COPY``, without the ``X-Source-Version`` request header. The ``MOVE`` operation is always applied on the latest version.
795 ==================== ================================
796 Request Header Name Value
797 ==================== ================================
798 If-Match Proceed if ETags match with object
799 If-None-Match Proceed if ETags don't match with object
800 Content-Length The size of the data written (optional, to update)
801 Content-Type The MIME content type of the object (optional, to update)
802 Content-Range The range of data supplied (optional, to update)
803 Transfer-Encoding Set to ``chunked`` to specify incremental uploading (if used, ``Content-Length`` is ignored)
804 Content-Encoding The encoding of the object (optional)
805 Content-Disposition The presentation style of the object (optional)
806 X-Source-Object Update with data from the object at path ``/<container>/<object>`` (optional, to update)
807 X-Source-Version The source version to update from (optional, to update)
808 X-Object-Bytes The updated object's final size (optional, when updating)
809 X-Object-Manifest Object parts prefix in ``<container>/<object>`` form (optional)
810 X-Object-Sharing Object permissions (optional)
811 X-Object-Public Object is publicly accessible (optional)
812 X-Object-Meta-* Optional user defined metadata
813 ==================== ================================
817 ====================== ============================================
818 Request Parameter Name Value
819 ====================== ============================================
820 update Do not replace metadata (no value parameter)
821 ====================== ============================================
823 The ``Content-Encoding``, ``Content-Disposition``, ``X-Object-Manifest`` and ``X-Object-Meta-*`` headers are considered to be user defined metadata. An operation without the ``update`` parameter will overwrite all previous values and remove any keys not supplied. When using ``update`` any metadata with an empty value will be deleted.
825 To change permissions, include an ``X-Object-Sharing`` header (as defined in ``PUT``). To publish, include an ``X-Object-Public`` header, with a value of ``true``. If no such headers are defined, no changes will be applied to sharing/public. Use empty values to remove permissions/unpublish (unpublishing also works with ``false`` as a header value). Sharing options are applied to the object - not its versions.
827 To update an object's data:
829 * Either set ``Content-Type`` to ``application/octet-stream``, or provide an object with ``X-Source-Object``. If ``Content-Type`` has some other value, it will be ignored and only the metadata will be updated.
830 * If the data is supplied in the request (using ``Content-Type`` instead of ``X-Source-Object``), a valid ``Content-Length`` header is required - except if using chunked transfers (set ``Transfer-Encoding`` to ``chunked``).
831 * Set ``Content-Range`` as specified in RFC2616, with the following differences:
833 * Client software MAY omit ``last-byte-pos`` of if the length of the range being transferred is unknown or difficult to determine.
834 * Client software SHOULD not specify the ``instance-length`` (use a ``*``), unless there is a reason for performing a size check at the server.
835 * If ``Content-Range`` used has a ``byte-range-resp-spec = *``, data will be appended to the object.
837 Optionally, truncate the updated object to the desired length with the ``X-Object-Bytes`` header.
839 A data update will trigger an ETag change. Updated ETags correspond to the single Merkle hash of the object's hashmap (refer to http://bittorrent.org/beps/bep_0030.html for more information).
841 No reply content. No reply headers if only metadata is updated.
843 ========================== ===============================
844 Reply Header Name Value
845 ========================== ===============================
846 ETag The new ETag of the object (data updated)
847 X-Object-Version The object's new version
848 ========================== ===============================
852 =========================== ==============================
853 Return Code Description
854 =========================== ==============================
855 202 (Accepted) The request has been accepted (not a data update)
856 204 (No Content) The request succeeded (data updated)
857 409 (Conflict) There are conflicting permissions (a list of conflicting sharing paths will be included in the reply - in simple text format)
858 411 (Length Required) Missing ``Content-Length`` in the request
859 416 (Range Not Satisfiable) The supplied range is invalid
860 =========================== ==============================
862 The ``POST`` method can also be used for creating an object via a standard HTML form. If the request ``Content-Type`` is ``multipart/form-data``, none of the above headers will be processed. The form should have exactly two fields, as in the following example. ::
864 <form method="post" action="https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt" enctype="multipart/form-data">
865 <input type="hidden" name="X-Auth-Token" value="0000">
866 <input type="file" name="X-Object-Data">
867 <input type="submit">
870 This will create/override the object with the given name, as if using ``PUT``. The ``Content-Type`` of the object will be set to the value of the corresponding header sent in the part of the request containing the data. Metadata, sharing and other object attributes can not be set this way.
872 ========================== ===============================
873 Reply Header Name Value
874 ========================== ===============================
875 ETag The MD5 hash of the object
876 X-Object-Version The object's new version
877 ========================== ===============================
881 =========================== ==============================
882 Return Code Description
883 =========================== ==============================
884 201 (Created) The object has been created
885 =========================== ==============================
891 ====================== ===================================
892 Request Parameter Name Value
893 ====================== ===================================
894 until Optional timestamp
895 ====================== ===================================
897 If ``until`` is defined, the object is "purged" up to that time (the history up to then is deleted).
899 No reply content/headers.
901 =========================== ==============================
902 Return Code Description
903 =========================== ==============================
904 204 (No Content) The request succeeded
905 =========================== ==============================
907 Sharing and Public Objects
908 ^^^^^^^^^^^^^^^^^^^^^^^^^^
910 Read and write control in Pithos is managed by setting appropriate permissions with the ``X-Object-Sharing`` header. The permissions are applied using prefix-based inheritance. Thus, each set of authorization directives is applied to all objects sharing the same prefix with the object where the corresponding ``X-Object-Sharing`` header is defined. For simplicity, nested/overlapping permissions are not allowed. Setting ``X-Object-Sharing`` will fail, if the object is already "covered", or another object with a longer common-prefix name already has permissions. When retrieving an object, the ``X-Object-Shared-By`` header reports where it gets its permissions from. If not present, the object is the actual source of authorization directives.
912 A user may ``GET`` another account or container. The result will include a limited reply, containing only the allowed containers or objects respectively. A top-level request with an authentication token, will return a list of allowed accounts, so the user can easily find out which other users share objects.
914 Objects that are marked as public, via the ``X-Object-Public`` meta, are also available at the corresponding URI returned for ``HEAD`` or ``GET``. Requests for public objects do not need to include an ``X-Auth-Token``. Pithos will ignore request parameters and only include the following headers in the reply (all ``X-Object-*`` meta is hidden):
916 ========================== ===============================
917 Reply Header Name Value
918 ========================== ===============================
919 ETag The ETag of the object
920 Content-Length The size of the data returned
921 Content-Type The MIME content type of the object
922 Content-Range The range of data included (only on a single range request)
923 Last-Modified The last object modification date (regardless of version)
924 Content-Encoding The encoding of the object (optional)
925 Content-Disposition The presentation style of the object (optional)
926 ========================== ===============================
928 Public objects are not included and do not influence cross-user listings. They are, however, readable by all users.
933 List of differences from the OOS API:
935 * Support for ``X-Account-Meta-*`` style headers at the account level. Use ``POST`` to update.
936 * Support for ``X-Container-Meta-*`` style headers at the container level. Can be set when creating via ``PUT``. Use ``POST`` to update.
937 * Header ``X-Container-Object-Meta`` at the container level and parameter ``meta`` in container listings. (**TBD**)
938 * Container policies to manage behavior and limits.
939 * Headers ``X-Container-Block-*`` at the container level, exposing the underlying storage characteristics.
940 * All metadata replies, at all levels, include latest modification information.
941 * At all levels, a ``HEAD`` or ``GET`` request may use ``If-Modified-Since`` and ``If-Unmodified-Since`` headers.
942 * Container/object lists include all associated metadata if the reply is of type json/xml. Some names are kept to their OOS API equivalents for compatibility.
943 * Option to include only shared containers/objects in listings.
944 * Object metadata allowed, in addition to ``X-Object-Meta-*``: ``Content-Encoding``, ``Content-Disposition``, ``X-Object-Manifest``. These are all replaced with every update operation, except if using the ``update`` parameter (in which case individual keys can also be deleted). Deleting meta by providing empty values also works when copying/moving an object.
945 * Multi-range object ``GET`` support as outlined in RFC2616.
946 * Object hashmap retrieval through ``GET`` and the ``format`` parameter.
947 * Object create via hashmap through ``PUT`` and the ``format`` parameter.
948 * Object create using ``POST`` to support standard HTML forms.
949 * Partial object updates through ``POST``, using the ``Content-Length``, ``Content-Type``, ``Content-Range`` and ``Transfer-Encoding`` headers. Use another object's data to update with ``X-Source-Object`` and ``X-Source-Version``. Truncate with ``X-Object-Bytes``. New ETag corresponds to the Merkle hash of the object's hashmap.
950 * Include new version identifier in replies for object replace/change requests.
951 * Object ``MOVE`` support.
952 * Conditional object create/update operations, using ``If-Match`` and ``If-None-Match`` headers.
953 * Time-variant account/container listings via the ``until`` parameter.
954 * Object versions - parameter ``version`` in ``HEAD``/``GET`` (list versions with ``GET``), ``X-Object-Version-*`` meta in replies, ``X-Source-Version`` in ``PUT``/``COPY``.
955 * Sharing/publishing with ``X-Object-Sharing``, ``X-Object-Public`` at the object level. Cross-user operations are allowed - controlled by sharing directives. Permissions may include groups defined with ``X-Account-Group-*`` at the account level. These apply to the object - not its versions.
956 * Support for prefix-based inheritance when enforcing permissions. Parent object carrying the authorization directives is reported in ``X-Object-Shared-By``.
957 * Large object support with ``X-Object-Manifest``.
958 * Trace the user that created/modified an object with ``X-Object-Modified-By``.
959 * Purge container/object history with the ``until`` parameter in ``DELETE``.
961 Clarifications/suggestions:
963 * Authentication is done by another system. The token is used in the same way, but it is obtained differently. The top level ``GET`` request is kept compatible with the OOS API and allows for guest/testing operations.
964 * Some processing is done in the variable part of all ``X-*-Meta-*`` headers. If it includes underscores, they will be converted to dashes and the first letter of all intra-dash strings will be capitalized.
965 * A ``GET`` reply for a level will include all headers of the corresponding ``HEAD`` request.
966 * To avoid conflicts between objects and virtual directory markers in container listings, it is recommended that object names do not end with the delimiter used.
967 * The ``Accept`` header may be used in requests instead of the ``format`` parameter to specify the desired reply format. The parameter overrides the header (**TBD**).
968 * Container/object lists use a ``200`` return code if the reply is of type json/xml. The reply will include an empty json/xml.
969 * In headers, dates are formatted according to RFC 1123. In extended information listings, the ``last_modified`` field is formatted according to ISO 8601 (for OOS API compatibility). All other fields (Pithos extensions) use integer tiemstamps.
970 * The ``Last-Modified`` header value always reflects the actual latest change timestamp, regardless of time control parameters and version requests. Time precondition checks with ``If-Modified-Since`` and ``If-Unmodified-Since`` headers are applied to this value.
971 * A copy/move using ``PUT``/``COPY``/``MOVE`` will always update metadata, keeping all old values except the ones redefined in the request headers.
972 * A ``HEAD`` or ``GET`` for an ``X-Object-Manifest`` object, will include modified ``Content-Length`` and ``ETag`` headers, according to the characteristics of the objects under the specified prefix. The ``Etag`` will be the MD5 hash of the corresponding ETags concatenated. In extended container listings there is no metadata processing.
980 Hopefully this API will allow for a multitude of client implementations, each supporting a different device or operating system. All clients will be able to manipulate containers and objects - even software only designed for OOS API compatibility. But a Pithos interface should not be only about showing containers and folders. There are some extra user interface elements and functionalities that should be common to all implementations.
982 Upon entrance to the service, a user is presented with the following elements - which can be represented as folders or with other related icons:
984 * The ``home`` element, which is used as the default entry point to the user's "files". Objects under ``home`` are represented in the usual hierarchical organization of folders and files.
985 * The ``trash`` element, which contains files that have been marked for deletion, but can still be recovered.
986 * The ``shared`` element, which contains all objects shared by the user to other users of the system.
987 * The ``others`` element, which contains all objects that other users share with the user.
988 * The ``groups`` element, which contains the names of groups the user has defined. Each group consists of a user list. Group creation, deletion, and manipulation is carried out by actions originating here.
989 * The ``history`` element, which allows browsing past instances of ``home`` and - optionally - ``trash``.
991 Objects in Pithos can be:
993 * Moved to trash and then deleted.
994 * Shared with specific permissions.
995 * Made public (shared with non-Pithos users).
996 * Restored from previous versions.
998 Some of these functions are performed by the client software and some by the Pithos server.
1000 In the first version of Pithos, objects could also be assigned custom tags. This is no longer supported. Existing deployments can migrate tags into a specific metadata value, i.e. ``X-Object-Meta-Tags``.
1002 Implementation Guidelines
1003 ^^^^^^^^^^^^^^^^^^^^^^^^^
1005 Pithos clients should use the ``pithos`` and ``trash`` containers for active and inactive objects respectively. If any of these containers is not found, the client software should create it, without interrupting the user's workflow. The ``home`` element corresponds to ``pithos`` and the ``trash`` element to ``trash``. Use ``PUT`` with the ``X-Move-From`` header, or ``MOVE`` to transfer objects from one container to the other. Use ``DELETE`` to remove from ``pithos`` without trashing, or to remove from ``trash``. When moving objects, detect naming conflicts with the ``If-Match`` or ``If-None-Match`` headers. Such conflicts should be resolved by the user.
1007 Object names should use the ``/`` delimiter to impose a hierarchy of folders and files.
1009 The ``shared`` element should be implemented as a read-only view of the ``pithos`` container, using the ``shared`` parameter when listing objects. The ``others`` element, should start with a top-level ``GET`` to retrieve the list of accounts accessible to the user. It is suggested that the client software hides the next step of navigation - the container - if it only includes ``pithos`` and forwards the user directly to the objects.
1011 Public objects are not included in ``shared`` and ``others`` listings. It is suggested that they are marked in a visually distinctive way in ``pithos`` listings (for example using an icon overlay).
1013 A special application menu, or a section in application preferences, should be devoted to managing groups (the ``groups`` element). All group-related actions are implemented at the account level.
1015 Browsing past versions of objects should be available both at the object and the container level. At the object level, a list of past versions can be included in the screen showing details or more information on the object (metadata, permissions, etc.). At the container level, it is suggested that clients use a ``history`` element, which presents to the user a read-only, time-variable view of ``pithos`` contents. This can be accomplished via the ``until`` parameter in listings. Optionally, ``history`` may include ``trash``.
1017 Recommended Practices and Examples
1018 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1020 Assuming an authentication token is obtained (**TBD**), the following high-level operations are available - shown with ``curl``:
1022 * Get account information ::
1025 -H "X-Auth-Token: 0000" \
1026 https://pithos.dev.grnet.gr/v1/user
1028 * List available containers ::
1031 -H "X-Auth-Token: 0000" \
1032 https://pithos.dev.grnet.gr/v1/user
1034 * Get container information ::
1037 -H "X-Auth-Token: 0000" \
1038 https://pithos.dev.grnet.gr/v1/user/pithos
1040 * Add a new container ::
1043 -H "X-Auth-Token: 0000" \
1044 https://pithos.dev.grnet.gr/v1/user/test
1046 * Delete a container ::
1048 curl -X DELETE -D - \
1049 -H "X-Auth-Token: 0000" \
1050 https://pithos.dev.grnet.gr/v1/user/test
1052 * List objects in a container ::
1055 -H "X-Auth-Token: 0000" \
1056 https://pithos.dev.grnet.gr/v1/user/pithos
1058 * List objects in a container (extended reply) ::
1061 -H "X-Auth-Token: 0000" \
1062 https://pithos.dev.grnet.gr/v1/user/pithos?format=json
1064 It is recommended that extended replies are cached and subsequent requests utilize the ``If-Modified-Since`` header.
1066 * List metadata keys used by objects in a container
1068 Will be in the ``X-Container-Object-Meta`` reply header, included in container information or object list (``HEAD`` or ``GET``). (**TBD**)
1070 * List objects in a container having a specific meta defined ::
1073 -H "X-Auth-Token: 0000" \
1074 https://pithos.dev.grnet.gr/v1/user/pithos?meta=favorites
1076 * Retrieve an object ::
1079 -H "X-Auth-Token: 0000" \
1080 https://pithos.dev.grnet.gr/v1/user/pithos/README.txt
1082 * Retrieve an object (specific ranges of data) ::
1085 -H "X-Auth-Token: 0000" \
1086 -H "Range: bytes=0-9" \
1087 https://pithos.dev.grnet.gr/v1/user/pithos/README.txt
1089 This will return the first 10 bytes. To get the first 10, bytes 30-39 and the last 100 use ``Range: bytes=0-9,30-39,-100``.
1091 * Add a new object (folder type) (**TBD**) ::
1094 -H "X-Auth-Token: 0000" \
1095 -H "Content-Type: application/folder" \
1096 https://pithos.dev.grnet.gr/v1/user/pithos/folder
1098 * Add a new object ::
1101 -H "X-Auth-Token: 0000" \
1102 -H "Content-Type: text/plain" \
1104 https://pithos.dev.grnet.gr/v1/user/pithos/folder/EXAMPLE.txt
1106 * Update an object ::
1109 -H "X-Auth-Token: 0000" \
1110 -H "Content-Length: 10" \
1111 -H "Content-Type: application/octet-stream" \
1112 -H "Content-Range: bytes 10-19/*" \
1114 https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt
1116 This will update bytes 10-19 with the data specified.
1118 * Update an object (append) ::
1121 -H "X-Auth-Token: 0000" \
1122 -H "Content-Length: 10" \
1123 -H "Content-Type: application/octet-stream" \
1124 -H "Content-Range: bytes */*" \
1126 https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt
1128 * Update an object (truncate) ::
1131 -H "X-Auth-Token: 0000" \
1132 -H "X-Source-Object: /folder/EXAMPLE.txt" \
1133 -H "Content-Range: bytes 0-0/*" \
1134 -H "X-Object-Bytes: 0" \
1135 https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt
1137 This will truncate the object to 0 bytes.
1139 * Add object metadata ::
1142 -H "X-Auth-Token: 0000" \
1143 -H "X-Object-Meta-First: first_meta_value" \
1144 -H "X-Object-Meta-Second: second_meta_value" \
1145 https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt
1147 * Delete object metadata ::
1150 -H "X-Auth-Token: 0000" \
1151 -H "X-Object-Meta-First: first_meta_value" \
1152 https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt
1154 Metadata can only be "set". To delete ``X-Object-Meta-Second``, reset all metadata.
1156 * Delete an object ::
1158 curl -X DELETE -D - \
1159 -H "X-Auth-Token: 0000" \
1160 https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt
projects / pithos / blob