1 Pithos v2 Developer Guide
2 =========================
7 Pithos is a storage service implemented by GRNET (http://www.grnet.gr). Data is stored as objects, organized in containers, belonging to an account. This hierarchy of storage layers has been inspired by the OpenStack Object Storage (OOS) API and similar CloudFiles API by Rackspace. The Pithos API follows the OOS API as closely as possible. One of the design requirements has been to be able to use Pithos with clients built for the OOS, without changes.
9 However, to be able to take full advantage of the Pithos infrastructure, client software should be aware of the extensions that differentiate Pithos from OOS. Pithos objects can be updated, or appended to. Pithos will store sharing permissions per object and enforce corresponding authorization policies. Automatic version management, allows taking account and container listings back in time, as well as reading previous instances of objects.
11 The storage backend of Pithos is block oriented, permitting efficient, deduplicated data placement. The block structure of objects is exposed at the API layer, in order to encourage external software to implement advanced data management operations.
13 This document's goals are:
15 * Define the Pithos ReST API that allows the storage and retrieval of data and metadata via HTTP calls
16 * Specify metadata semantics and user interface guidelines for a common experience across client software implementations
18 The present document is meant to be read alongside the OOS API documentation. Thus, it is suggested that the reader is familiar with associated technologies, the OOS API as well as the first version of the Pithos API. This document refers to the second version of Pithos. Information on the first version of the storage API can be found at http://code.google.com/p/gss.
20 Whatever marked as to be determined (**TBD**), should not be considered by implementors.
25 ========================= ================================
27 ========================= ================================
28 0.4 (June 30, 2011) Object permissions and account groups.
29 \ Control versioning behavior and container quotas with container policy directives.
30 \ Support updating/deleting individual metadata with ``POST``.
31 0.3 (June 14, 2011) Large object support with ``X-Object-Manifest``.
32 \ Allow for publicly available objects via ``https://hostname/public``.
33 \ Support time-variant account/container listings.
34 \ Add source version when duplicating with PUT/COPY.
35 \ Request version in object HEAD/GET requests (list versions with GET).
36 0.2 (May 31, 2011) Add object meta listing and filtering in containers.
37 \ Include underlying storage characteristics in container meta.
38 \ Support for partial object updates through POST.
39 \ Expose object hashmaps through GET.
40 \ Support for multi-range object GET requests.
41 0.1 (May 17, 2011) Initial release. Based on OpenStack Object Storage Developer Guide API v1 (Apr. 15, 2011).
42 ========================= ================================
47 The URI requests supported by the Pithos API follow one of the following forms:
49 * Top level: ``https://hostname/v1/``
50 * Account level: ``https://hostname/v1/<account>``
51 * Container level: ``https://hostname/v1/<account>/<container>``
52 * Object level: ``https://hostname/v1/<account>/<container>/<object>``
54 All requests must include an ``X-Auth-Token``. The process of obtaining the token is still to be determined (**TBD**).
56 The allowable request operations and respective return codes per level are presented in the remainder of this chapter. Common to all requests are the following return codes.
58 ========================= ================================
59 Return Code Description
60 ========================= ================================
61 400 (Bad Request) The request is invalid
62 401 (Unauthorized) Request not allowed
63 404 (Not Found) The requested resource was not found
64 503 (Service Unavailable) The request cannot be completed because of an internal error
65 ========================= ================================
72 ========= ==================
74 ========= ==================
75 GET Authentication. This is kept for compatibility with the OOS API
76 ========= ==================
81 If the ``X-Auth-User`` and ``X-Auth-Key`` headers are given, a dummy ``X-Auth-Token`` and ``X-Storage-Url`` will be replied, which can be used as a guest token/namespace for testing Pithos.
83 ================ =====================
84 Return Code Description
85 ================ =====================
86 204 (No Content) The request succeeded
87 ================ =====================
95 ========= ==================
97 ========= ==================
98 HEAD Retrieve account metadata
100 POST Update account metadata
101 ========= ==================
106 ====================== ===================================
107 Request Parameter Name Value
108 ====================== ===================================
109 until Optional timestamp
110 ====================== ===================================
114 ========================== =====================
115 Reply Header Name Value
116 ========================== =====================
117 X-Account-Container-Count The total number of containers
118 X-Account-Object-Count The total number of objects (**TBD**)
119 X-Account-Bytes-Used The total number of bytes stored
120 X-Account-Bytes-Remaining The total number of bytes remaining (**TBD**)
121 X-Account-Last-Login The last login (**TBD**)
122 X-Account-Until-Timestamp The last account modification date until the timestamp provided
123 X-Account-Group-* Optional user defined groups
124 X-Account-Meta-* Optional user defined metadata
125 Last-Modified The last account modification date (regardless of ``until``)
126 ========================== =====================
130 ================ =====================
131 Return Code Description
132 ================ =====================
133 204 (No Content) The request succeeded
134 ================ =====================
140 ==================== ===========================
141 Request Header Name Value
142 ==================== ===========================
143 If-Modified-Since Retrieve if account has changed since provided timestamp
144 If-Unmodified-Since Retrieve if account has not changed since provided timestamp
145 ==================== ===========================
149 ====================== =========================
150 Request Parameter Name Value
151 ====================== =========================
152 limit The amount of results requested (default is 10000)
153 marker Return containers with name lexicographically after marker
154 format Optional extended reply type (can be ``json`` or ``xml``)
155 until Optional timestamp
156 ====================== =========================
158 The reply is a list of container names. Account headers (as in a ``HEAD`` request) will also be included.
159 If a ``format=xml`` or ``format=json`` argument is given, extended information on the containers will be returned, serialized in the chosen format.
160 For each container, the information will include all container metadata (names will be in lower case and with hyphens replaced with underscores):
162 =========================== ============================
164 =========================== ============================
165 name The name of the container
166 count The number of objects inside the container
167 bytes The total size of the objects inside the container
168 last_modified The last container modification date (regardless of ``until``)
169 x_container_until_timestamp The last container modification date until the timestamp provided
170 x_container_policy_* Container behavior and limits
171 x_container_meta_* Optional user defined metadata
172 =========================== ============================
174 For examples of container details returned in JSON/XML formats refer to the OOS API documentation.
176 =========================== =====================
177 Return Code Description
178 =========================== =====================
179 200 (OK) The request succeeded
180 204 (No Content) The account has no containers (only for non-extended replies)
181 304 (Not Modified) The account has not been modified
182 412 (Precondition Failed) The condition set can not be satisfied
183 =========================== =====================
185 Will use a ``200`` return code if the reply is of type json/xml.
191 ====================== ============================================
192 Request Parameter Name Value
193 ====================== ============================================
194 update Do not replace metadata/groups (no value parameter)
195 ====================== ============================================
199 ==================== ===========================
200 Request Header Name Value
201 ==================== ===========================
202 X-Account-Group-* Optional user defined groups
203 X-Account-Meta-* Optional user defined metadata
204 ==================== ===========================
206 No reply content/headers.
208 The operation will overwrite all user defined metadata, except if ``update`` is defined.
209 To create a group, include an ``X-Account-Group-*`` header with the name in the key and a comma separated list of user names in the value. If no ``X-Account-Group-*`` header is present, no changes will be applied to groups. The ``update`` parameter also applies to groups. To delete a specific group, use ``update`` and an empty header value.
211 ================ ===============================
212 Return Code Description
213 ================ ===============================
214 202 (Accepted) The request has been accepted
215 ================ ===============================
223 ========= ============================
224 Operation Description
225 ========= ============================
226 HEAD Retrieve container metadata
228 PUT Create/update container
229 POST Update container metadata
230 DELETE Delete container
231 ========= ============================
237 ====================== ===================================
238 Request Parameter Name Value
239 ====================== ===================================
240 until Optional timestamp
241 ====================== ===================================
245 =========================== ===============================
246 Reply Header Name Value
247 =========================== ===============================
248 X-Container-Object-Count The total number of objects in the container
249 X-Container-Bytes-Used The total number of bytes of all objects stored
250 X-Container-Block-Size The block size used by the storage backend
251 X-Container-Block-Hash The hash algorithm used for block identifiers in object hashmaps
252 X-Container-Until-Timestamp The last container modification date until the timestamp provided
253 X-Container-Object-Meta A list with all meta keys used by objects
254 X-Container-Policy-* Container behavior and limits
255 X-Container-Meta-* Optional user defined metadata
256 Last-Modified The last container modification date (regardless of ``until``)
257 =========================== ===============================
259 The keys returned in ``X-Container-Object-Meta`` are all the unique strings after the ``X-Object-Meta-`` prefix. See container ``PUT`` for a reference of policy directives.
261 ================ ===============================
262 Return Code Description
263 ================ ===============================
264 204 (No Content) The request succeeded
265 ================ ===============================
271 ==================== ===========================
272 Request Header Name Value
273 ==================== ===========================
274 If-Modified-Since Retrieve if container has changed since provided timestamp
275 If-Unmodified-Since Retrieve if container has not changed since provided timestamp
276 ==================== ===========================
280 ====================== ===================================
281 Request Parameter Name Value
282 ====================== ===================================
283 limit The amount of results requested (default is 10000)
284 marker Return containers with name lexicographically after marker
285 prefix Return objects starting with prefix
286 delimiter Return objects up to the delimiter (discussion follows)
287 path Assume ``prefix=path`` and ``delimiter=/``
288 format Optional extended reply type (can be ``json`` or ``xml``)
289 meta Return objects having the specified meta keys (can be a comma separated list)
290 until Optional timestamp
291 ====================== ===================================
293 The ``path`` parameter overrides ``prefix`` and ``delimiter``. When using ``path``, results will include objects ending in ``delimiter``.
295 The keys given with ``meta`` will be matched with the strings after the ``X-Object-Meta-`` prefix.
297 The reply is a list of object names. Container headers (as in a ``HEAD`` request) will also be included.
298 If a ``format=xml`` or ``format=json`` argument is given, extended information on the objects will be returned, serialized in the chosen format.
299 For each object, the information will include all object metadata (names will be in lower case and with hyphens replaced with underscores):
301 ========================== ======================================
303 ========================== ======================================
304 name The name of the object
305 hash The ETag of the object
306 bytes The size of the object
307 content_type The MIME content type of the object
308 content_encoding The encoding of the object (optional)
309 content-disposition The presentation style of the object (optional)
310 last_modified The last object modification date (regardless of version)
311 x_object_version The object's version identifier
312 x_object_version_timestamp The object's version timestamp
313 x_object_modified_by The user that committed the object's version
314 x_object_manifest Object parts prefix in ``<container>/<object>`` form (optional)
315 x_object_sharing Object permissions (optional)
316 x_object_shared_by Object inheriting permissions (optional)
317 x_object_public Object's publicly accessible URI (optional)
318 x_object_meta_* Optional user defined metadata
319 ========================== ======================================
321 Extended replies may also include virtual directory markers in separate sections of the ``json`` or ``xml`` results.
322 Virtual directory markers are only included when ``delimiter`` is explicitly set. They correspond to the substrings up to and including the first occurrence of the delimiter.
323 In JSON results they appear as dictionaries with only a ``"subdir"`` key. In XML results they appear interleaved with ``<object>`` tags as ``<subdir name="..." />``.
324 In case there is an object with the same name as a virtual directory marker, the object will be returned.
326 For examples of object details returned in JSON/XML formats refer to the OOS API documentation.
328 =========================== ===============================
329 Return Code Description
330 =========================== ===============================
331 200 (OK) The request succeeded
332 204 (No Content) The account has no containers (only for non-extended replies)
333 304 (Not Modified) The container has not been modified
334 412 (Precondition Failed) The condition set can not be satisfied
335 =========================== ===============================
337 Will use a ``200`` return code if the reply is of type json/xml.
343 ==================== ================================
344 Request Header Name Value
345 ==================== ================================
346 X-Container-Policy-* Container behavior and limits
347 X-Container-Meta-* Optional user defined metadata
348 ==================== ================================
350 No reply content/headers.
352 If no policy is defined, the container will be created with the default values.
353 Available policy directives:
355 * ``versioning``: Set to ``auto``, ``manual`` or ``none`` (default is ``manual``)
356 * ``quota``: Size limit in KB (default is ``0`` - unlimited)
358 ================ ===============================
359 Return Code Description
360 ================ ===============================
361 201 (Created) The container has been created
362 202 (Accepted) The request has been accepted
363 ================ ===============================
369 ====================== ============================================
370 Request Parameter Name Value
371 ====================== ============================================
372 update Do not replace metadata/policy (no value parameter)
373 ====================== ============================================
377 ==================== ================================
378 Request Header Name Value
379 ==================== ================================
380 X-Container-Policy-* Container behavior and limits
381 X-Container-Meta-* Optional user defined metadata
382 ==================== ================================
384 No reply content/headers.
386 The operation will overwrite all user defined metadata, except if ``update`` is defined.
387 To change policy, include an ``X-Container-Policy-*`` header with the name in the key. If no ``X-Container-Policy-*`` header is present, no changes will be applied to policy. The ``update`` parameter also applies to policy - deleted values will revert to defaults. To delete/revert a specific policy directive, use ``update`` and an empty header value. See container ``PUT`` for a reference of policy directives.
389 ================ ===============================
390 Return Code Description
391 ================ ===============================
392 202 (Accepted) The request has been accepted
393 ================ ===============================
399 No request parameters/headers.
401 No reply content/headers.
403 ================ ===============================
404 Return Code Description
405 ================ ===============================
406 204 (No Content) The request succeeded
407 409 (Conflict) The container is not empty
408 ================ ===============================
416 ========= =================================
417 Operation Description
418 ========= =================================
419 HEAD Retrieve object metadata
421 PUT Write object data or copy/move object
424 POST Update object metadata/data
426 ========= =================================
432 ====================== ===================================
433 Request Parameter Name Value
434 ====================== ===================================
435 version Optional version identifier
436 ====================== ===================================
440 ========================== ===============================
441 Reply Header Name Value
442 ========================== ===============================
443 ETag The ETag of the object
444 Content-Length The size of the object
445 Content-Type The MIME content type of the object
446 Last-Modified The last object modification date (regardless of version)
447 Content-Encoding The encoding of the object (optional)
448 Content-Disposition The presentation style of the object (optional)
449 X-Object-Version The object's version identifier
450 X-Object-Version-Timestamp The object's version timestamp
451 X-Object-Modified-By The user that comitted the object's version
452 X-Object-Manifest Object parts prefix in ``<container>/<object>`` form (optional)
453 X-Object-Sharing Object permissions (optional)
454 X-Object-Shared-By Object inheriting permissions (optional)
455 X-Object-Public Object's publicly accessible URI (optional)
456 X-Object-Meta-* Optional user defined metadata
457 ========================== ===============================
461 ================ ===============================
462 Return Code Description
463 ================ ===============================
464 200 (No Content) The request succeeded
465 ================ ===============================
471 ==================== ================================
472 Request Header Name Value
473 ==================== ================================
474 Range Optional range of data to retrieve
475 If-Match Retrieve if ETags match
476 If-None-Match Retrieve if ETags don't match
477 If-Modified-Since Retrieve if object has changed since provided timestamp
478 If-Unmodified-Since Retrieve if object has not changed since provided timestamp
479 ==================== ================================
483 ====================== ===================================
484 Request Parameter Name Value
485 ====================== ===================================
486 format Optional extended reply type (can be ``json`` or ``xml``)
487 version Optional version identifier or ``list`` (specify a format if requesting a list)
488 ====================== ===================================
490 The reply is the object's data (or part of it), except if a hashmap is requested with the ``format`` parameter, or a version list with ``version=list`` (in which case an extended reply format must be specified). Object headers (as in a ``HEAD`` request) are always included.
492 Hashmaps expose the underlying storage format of the object. Note that each hash is computed after trimming trailing null bytes of the corresponding block.
494 Example ``format=json`` reply:
498 {"block_hash": "sha1", "hashes": ["7295c41da03d7f916440b98e32c4a2a39351546c", ...], "block_size": 131072, "bytes": 242}
500 Example ``format=xml`` reply:
504 <?xml version="1.0" encoding="UTF-8"?>
505 <object name="file" bytes="24223726" block_size="131072" block_hash="sha1">
506 <hash>7295c41da03d7f916440b98e32c4a2a39351546c</hash>
510 Version lists include the version identifier and timestamp for each available object version. Version identifiers can be arbitrary strings, so use the timestamp to find newer versions.
512 Example ``format=json`` reply:
516 {"versions": [[23, 1307700892], [28, 1307700898], ...]}
518 Example ``format=xml`` reply:
522 <?xml version="1.0" encoding="UTF-8"?>
524 <version timestamp="1307700892">23</version>
525 <version timestamp="1307700898">28</version>
526 <version timestamp="...">...</version>
529 The ``Range`` header may include multiple ranges, as outlined in RFC2616. Then the ``Content-Type`` of the reply will be ``multipart/byteranges`` and each part will include a ``Content-Range`` header.
531 ========================== ===============================
532 Reply Header Name Value
533 ========================== ===============================
534 ETag The ETag of the object
535 Content-Length The size of the data returned
536 Content-Type The MIME content type of the object
537 Content-Range The range of data included (only on a single range request)
538 Last-Modified The last object modification date (regardless of version)
539 Content-Encoding The encoding of the object (optional)
540 Content-Disposition The presentation style of the object (optional)
541 X-Object-Version The object's version identifier
542 X-Object-Version-Timestamp The object's version timestamp
543 X-Object-Modified-By The user that comitted the object's version
544 X-Object-Manifest Object parts prefix in ``<container>/<object>`` form (optional)
545 X-Object-Sharing Object permissions (optional)
546 X-Object-Shared-By Object inheriting permissions (optional)
547 X-Object-Public Object's publicly accessible URI (optional)
548 X-Object-Meta-* Optional user defined metadata
549 ========================== ===============================
553 =========================== ==============================
554 Return Code Description
555 =========================== ==============================
556 200 (OK) The request succeeded
557 206 (Partial Content) The range request succeeded
558 304 (Not Modified) The object has not been modified
559 412 (Precondition Failed) The condition set can not be satisfied
560 416 (Range Not Satisfiable) The requested range is out of limits
561 =========================== ==============================
567 ==================== ================================
568 Request Header Name Value
569 ==================== ================================
570 ETag The MD5 hash of the object (optional to check written data)
571 Content-Length The size of the data written
572 Content-Type The MIME content type of the object
573 Transfer-Encoding Set to ``chunked`` to specify incremental uploading (if used, ``Content-Length`` is ignored)
574 X-Copy-From The source path in the form ``/<container>/<object>``
575 X-Move-From The source path in the form ``/<container>/<object>``
576 X-Source-Version The source version to copy from
577 Content-Encoding The encoding of the object (optional)
578 Content-Disposition The presentation style of the object (optional)
579 X-Object-Manifest Object parts prefix in ``<container>/<object>`` form (optional)
580 X-Object-Sharing Object permissions (optional)
581 X-Object-Public Object is publicly accessible (optional)
582 X-Object-Meta-* Optional user defined metadata
583 ==================== ================================
587 ========================== ===============================
588 Reply Header Name Value
589 ========================== ===============================
590 ETag The MD5 hash of the object (on create)
591 ========================== ===============================
593 The ``X-Object-Sharing`` header may include either a ``read=...`` comma-separated user/group list, or a ``write=...`` comma-separated user/group list, or both separated by a semicolon (``;``). To publish the object, set ``X-Object-Public`` to ``true``. To unpublish, set to ``false``, or use an empty header value.
595 =========================== ==============================
596 Return Code Description
597 =========================== ==============================
598 201 (Created) The object has been created
599 411 (Length Required) Missing ``Content-Length`` or ``Content-Type`` in the request
600 422 (Unprocessable Entity) The MD5 checksum of the data written to the storage system does not match the (optionally) supplied ETag value
601 =========================== ==============================
607 ==================== ================================
608 Request Header Name Value
609 ==================== ================================
610 Destination The destination path in the form ``/<container>/<object>``
611 Content-Type The MIME content type of the object (optional)
612 Content-Encoding The encoding of the object (optional)
613 Content-Disposition The presentation style of the object (optional)
614 X-Source-Version The source version to copy from
615 X-Object-Manifest Object parts prefix in ``<container>/<object>`` form (optional)
616 X-Object-Sharing Object permissions (optional)
617 X-Object-Public Object is publicly accessible (optional)
618 X-Object-Meta-* Optional user defined metadata
619 ==================== ================================
621 Refer to ``PUT``/``POST`` for a description of request headers. Metadata is also copied, updated with any values defined. Sharing/publishing options are not copied.
623 No reply content/headers.
625 =========================== ==============================
626 Return Code Description
627 =========================== ==============================
628 201 (Created) The object has been created
629 =========================== ==============================
635 Same as ``COPY``, without the ``X-Source-Version`` request header. The ``MOVE`` operation is always applied on the latest version.
641 ====================== ============================================
642 Request Parameter Name Value
643 ====================== ============================================
644 update Do not replace metadata (no value parameter)
645 ====================== ============================================
649 ==================== ================================
650 Request Header Name Value
651 ==================== ================================
652 Content-Length The size of the data written (optional, to update)
653 Content-Type The MIME content type of the object (optional, to update)
654 Content-Range The range of data supplied (optional, to update)
655 Transfer-Encoding Set to ``chunked`` to specify incremental uploading (if used, ``Content-Length`` is ignored)
656 Content-Encoding The encoding of the object (optional)
657 Content-Disposition The presentation style of the object (optional)
658 X-Object-Manifest Object parts prefix in ``<container>/<object>`` form (optional)
659 X-Object-Sharing Object permissions (optional)
660 X-Object-Public Object is publicly accessible (optional)
661 X-Object-Meta-* Optional user defined metadata
662 ==================== ================================
664 The ``Content-Encoding``, ``Content-Disposition``, ``X-Object-Manifest`` and ``X-Object-Meta-*`` headers are considered to be user defined metadata. An operation without the ``update`` parameter will overwrite all previous values and remove any keys not supplied. When using ``update`` any metadata with an empty value will be deleted.
666 To change permissions, include an ``X-Object-Sharing`` header (as defined in ``PUT``). To publish, include an ``X-Object-Public`` header, with a value of ``true``. If no such headers are defined, no changes will be applied to sharing/public. Use empty values to remove permissions/unpublish (unpublishing also works with ``false`` as a header value). Sharing options are applied to the object - not its versions.
668 To update an object's data:
670 * Set ``Content-Type`` to ``application/octet-stream``. If ``Content-Type`` has some other value, it will be ignored and only the metadata will be updated.
671 * Supply ``Content-Length`` (except if using chunked transfers), ``Content-Type`` and ``Content-Range`` headers.
672 * Set ``Content-Range`` as specified in RFC2616, with the following differences:
674 * Client software MAY omit ``last-byte-pos`` of if the length of the range being transferred is unknown or difficult to determine.
675 * Client software SHOULD not specify the ``instance-length`` (use a ``*``), unless there is a reason for performing a size check at the server.
676 * If ``Content-Range`` used has a ``byte-range-resp-spec = *``, data supplied will be appended to the object.
678 A data update will trigger an ETag change. The new ETag will not correspond to the object's MD5 sum (**TBD**) and will be included in reply headers.
680 No reply content. No reply headers if only metadata is updated.
682 ========================== ===============================
683 Reply Header Name Value
684 ========================== ===============================
685 ETag The new ETag of the object (data updated)
686 ========================== ===============================
690 =========================== ==============================
691 Return Code Description
692 =========================== ==============================
693 202 (Accepted) The request has been accepted (not a data update)
694 204 (No Content) The request succeeded (data updated)
695 411 (Length Required) Missing ``Content-Length`` in the request
696 416 (Range Not Satisfiable) The supplied range is invalid
697 =========================== ==============================
703 No request parameters/headers.
705 No reply content/headers.
707 =========================== ==============================
708 Return Code Description
709 =========================== ==============================
710 204 (No Content) The request succeeded
711 =========================== ==============================
713 Sharing and Public Objects
714 ^^^^^^^^^^^^^^^^^^^^^^^^^^
716 Read and write control in Pithos is managed by setting appropriate permissions with the ``X-Object-Sharing`` header. The permissions are applied using prefix-based inheritance. Thus, each set of authorization directives is applied to all objects sharing the same prefix with the object where the corresponding ``X-Object-Sharing`` header is defined. For simplicity, nested/overlapping permissions are not allowed. Setting ``X-Object-Sharing`` will fail, if the object is already "covered", or another object with a longer common-prefix name already has permissions. When retrieving an object, the ``X-Object-Shared-By`` header reports where it gets its permissions from. If not present, the object is the actual source of authorization directives.
718 Objects that are marked as public, via the ``X-Object-Public`` meta, are also available at the corresponding URI returned for ``HEAD`` or ``GET``. Requests for public objects do not need to include an ``X-Auth-Token``. Pithos will ignore request parameters and only include the following headers in the reply (all ``X-Object-*`` meta is hidden).
720 ========================== ===============================
721 Reply Header Name Value
722 ========================== ===============================
723 ETag The ETag of the object
724 Content-Length The size of the data returned
725 Content-Type The MIME content type of the object
726 Content-Range The range of data included (only on a single range request)
727 Last-Modified The last object modification date (regardless of version)
728 Content-Encoding The encoding of the object (optional)
729 Content-Disposition The presentation style of the object (optional)
730 ========================== ===============================
735 List of differences from the OOS API:
737 * Support for ``X-Account-Meta-*`` style headers at the account level. Use ``POST`` to update.
738 * Support for ``X-Container-Meta-*`` style headers at the container level. Can be set when creating via ``PUT``. Use ``POST`` to update.
739 * Header ``X-Container-Object-Meta`` at the container level and parameter ``meta`` in container listings.
740 * Container policies to manage behavior and limits.
741 * Headers ``X-Container-Block-*`` at the container level, exposing the underlying storage characteristics.
742 * All metadata replies, at all levels, include latest modification information.
743 * At all levels, a ``GET`` request may use ``If-Modified-Since`` and ``If-Unmodified-Since`` headers.
744 * Container/object lists include all associated metadata if the reply is of type json/xml. Some names are kept to their OOS API equivalents for compatibility.
745 * Object metadata allowed, in addition to ``X-Object-Meta-*``: ``Content-Encoding``, ``Content-Disposition``, ``X-Object-Manifest``. These are all replaced with every update operation, except if using the ``update`` parameter (in which case individual keys can also be deleted). Deleting meta by providing empty values also works when copying/moving an object.
746 * Multi-range object GET support as outlined in RFC2616.
747 * Object hashmap retrieval through GET and the ``format`` parameter.
748 * Partial object updates through POST, using the ``Content-Length``, ``Content-Type``, ``Content-Range`` and ``Transfer-Encoding`` headers.
749 * Object ``MOVE`` support.
750 * Time-variant account/container listings via the ``until`` parameter.
751 * Object versions - parameter ``version`` in HEAD/GET (list versions with GET), ``X-Object-Version-*`` meta in replies, ``X-Source-Version`` in PUT/COPY.
752 * Sharing/publishing with ``X-Object-Sharing``, ``X-Object-Public`` at the object level. Permissions may include groups defined with ``X-Account-Group-*`` at the account level. These apply to the object - not its versions.
753 * Support for prefix-based inheritance when enforcing permissions. Parent object carrying the authorization directives is reported in ``X-Object-Shared-By``.
754 * Large object support with ``X-Object-Manifest``.
755 * Trace the user that created/modified an object with ``X-Object-Modified-By``.
757 Clarifications/suggestions:
759 * Authentication is done by another system. The token is used in the same way, but it is obtained differently. The top level ``GET`` request is kept compatible with the OOS API and allows for guest/testing operations.
760 * Some processing is done in the variable part of all ``X-*-Meta-*`` headers. If it includes underscores, they will be converted to dashes and the first letter of all intra-dash strings will be capitalized.
761 * A ``GET`` reply for a level will include all headers of the corresponding ``HEAD`` request.
762 * To avoid conflicts between objects and virtual directory markers in container listings, it is recommended that object names do not end with the delimiter used.
763 * The ``Accept`` header may be used in requests instead of the ``format`` parameter to specify the desired reply format. The parameter overrides the header.
764 * Container/object lists use a ``200`` return code if the reply is of type json/xml. The reply will include an empty json/xml.
765 * In headers, dates are formatted according to RFC 1123. In extended information listings, dates are formatted according to ISO 8601.
766 * The ``Last-Modified`` header value always reflects the actual latest change timestamp, regardless of time control parameters and version requests. Time precondition checks with ``If-Modified-Since`` and ``If-Unmodified-Since`` headers are applied to this value.
767 * A copy/move using ``PUT``/``COPY``/``MOVE`` will always update metadata, keeping all old values except the ones redefined in the request headers.
768 * A ``HEAD`` or ``GET`` for an ``X-Object-Manifest`` object, will include modified ``Content-Length`` and ``ETag`` headers, according to the characteristics of the objects under the specified prefix. The ``Etag`` will be the MD5 hash of the corresponding ETags concatenated. In extended container listings there is no metadata processing.
776 Hopefully this API will allow for a multitude of client implementations, each supporting a different device or operating system. All clients will be able to manipulate containers and objects - even software only designed for OOS API compatibility. But a Pithos interface should not be only about showing containers and folders. There are some extra user interface elements and functionalities that should be common to all implementations.
778 Upon entrance to the service, a user is presented with the following elements - which can be represented as folders or with other related icons:
780 * The ``home`` element, which is used as the default entry point to the user's "files". Objects under ``home`` are represented in the usual hierarchical organization of folders and files.
781 * The ``trash`` element, which contains files that have been marked for deletion, but can still be recovered.
782 * The ``shared`` element, which contains all objects shared by the user to other users of the system.
783 * The ``others`` element, which contains all objects that other users share with the user.
784 * The ``tags`` element, which lists the names of tags the user has defined. This can be an entry point to list all files that have been assigned a specific tag or manage tags in general (remove a tag completely, rename a tag etc.).
785 * The ``groups`` element, which contains the names of groups the user has defined. Each group consists of a user list. Group creation, deletion, and manipulation is carried out by actions originating here.
787 Objects in Pithos can be:
789 * Assigned custom tags.
790 * Moved to trash and then deleted.
791 * Shared with specific permissions.
792 * Made public (shared with non-Pithos users).
793 * Restored from previous versions.
795 Some of these functions are performed by the client software and some by the Pithos server. Client-driven functionality is based on specific metadata that should be handled equally across implementations. These metadata names are discussed in the next chapter.
797 Conventions and Metadata Specification
798 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
800 Pithos clients should use the ``pithos`` container for all Pithos objects. Object names use the ``/`` delimiter to impose a hierarchy of folders and files.
802 At the object level, tags are implemented by managing metadata keys. The client software should allow the user to use any string as a tag (except ``trash``) and then set the corresponding ``X-Object-Meta-<tag>`` key at the server. The API extensions provided, allow for listing all tags in a container and filtering object listings based on one or more tags. The tag list is sufficient for implementing the ``tags`` element, either as a special, virtual folder (as done in the first version of Pithos), or as an application menu.
804 To manage the deletion of files use the same API and the ``X-Object-Meta-Trash`` key. The string ``trash`` can not be used as a tag. The ``trash`` element should be presented as a folder, although with no hierarchy.
806 The metadata specification is summarized in the following table.
808 =========================== ==============================
810 =========================== ==============================
811 X-Object-Meta-Trash Set to ``true`` if the object has been moved to the trash
812 X-Object-Meta-* Use for other tags that apply to the object
813 =========================== ==============================
815 Recommended Practices and Examples
816 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
818 Assuming an authentication token is obtained (**TBD**), the following high-level operations are available - shown with ``curl``:
820 * Get account information ::
823 -H "X-Auth-Token: 0000" \
824 https://pithos.dev.grnet.gr/v1/user
826 * List available containers ::
829 -H "X-Auth-Token: 0000" \
830 https://pithos.dev.grnet.gr/v1/user
832 * Get container information ::
835 -H "X-Auth-Token: 0000" \
836 https://pithos.dev.grnet.gr/v1/user/pithos
838 * Add a new container ::
841 -H "X-Auth-Token: 0000" \
842 https://pithos.dev.grnet.gr/v1/user/test
844 * Delete a container ::
846 curl -X DELETE -D - \
847 -H "X-Auth-Token: 0000" \
848 https://pithos.dev.grnet.gr/v1/user/test
850 * List objects in a container ::
853 -H "X-Auth-Token: 0000" \
854 https://pithos.dev.grnet.gr/v1/user/pithos
856 * List objects in a container (extended reply) ::
859 -H "X-Auth-Token: 0000" \
860 https://pithos.dev.grnet.gr/v1/user/pithos?format=json
862 It is recommended that extended replies are cached and subsequent requests utilize the ``If-Modified-Since`` header.
864 * List metadata keys used by objects in a container
866 Will be in the ``X-Container-Object-Meta`` reply header, included in container information or object list (``HEAD`` or ``GET``).
868 * List objects in a container having a specific meta defined ::
871 -H "X-Auth-Token: 0000" \
872 https://pithos.dev.grnet.gr/v1/user/pithos?meta=trash
874 This is the recommended way of tagging/retrieving objects in trash.
876 * Retrieve an object ::
879 -H "X-Auth-Token: 0000" \
880 https://pithos.dev.grnet.gr/v1/user/pithos/README.txt
882 * Retrieve an object (specific ranges of data) ::
885 -H "X-Auth-Token: 0000" \
886 -H "Range: bytes=0-9" \
887 https://pithos.dev.grnet.gr/v1/user/pithos/README.txt
889 This will return the first 10 bytes. To get the first 10, bytes 30-39 and the last 100 use ``Range: bytes=0-9,30-39,-100``.
891 * Add a new object (folder type) (**TBD**) ::
894 -H "X-Auth-Token: 0000" \
895 -H "Content-Type: application/folder" \
896 https://pithos.dev.grnet.gr/v1/user/pithos/folder
898 * Add a new object ::
901 -H "X-Auth-Token: 0000" \
902 -H "Content-Type: text/plain" \
904 https://pithos.dev.grnet.gr/v1/user/pithos/folder/EXAMPLE.txt
906 * Update an object ::
909 -H "X-Auth-Token: 0000" \
910 -H "Content-Length: 10" \
911 -H "Content-Type: application/octet-stream" \
912 -H "Content-Range: bytes 10-19/*" \
914 https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt
916 This will update bytes 10-19 with the data specified.
918 * Update an object (append) ::
921 -H "X-Auth-Token: 0000" \
922 -H "Content-Length: 10" \
923 -H "Content-Type: application/octet-stream" \
924 -H "Content-Range: bytes */*" \
926 https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt
928 * Add object metadata ::
931 -H "X-Auth-Token: 0000" \
932 -H "X-Object-Meta-First: first_meta_value" \
933 -H "X-Object-Meta-Second: second_meta_value" \
934 https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt
936 * Delete object metadata ::
939 -H "X-Auth-Token: 0000" \
940 -H "X-Object-Meta-First: first_meta_value" \
941 https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt
943 Metadata can only be "set". To delete ``X-Object-Meta-Second``, reset all metadata.
945 * Delete an object ::
947 curl -X DELETE -D - \
948 -H "X-Auth-Token: 0000" \
949 https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt
projects / pithos / blob