1 Pithos v2 Developer Guide
2 =========================
7 Pithos is a storage service implemented by GRNET (http://www.grnet.gr). Data is stored as objects, organized in containers, belonging to an account. This hierarchy of storage layers has been inspired by the OpenStack Object Storage (OOS) API and similar CloudFiles API by Rackspace. The Pithos API follows the OOS API as closely as possible. One of the design requirements has been to be able to use Pithos with clients built for the OOS, without changes.
9 However, to be able to take full advantage of the Pithos infrastructure, client software should be aware of the extensions that differentiate Pithos from OOS. Pithos objects can be updated, or appended to. Pithos will store sharing permissions per object and enforce corresponding authorization policies. Automatic version management, allows taking account and container listings back in time, as well as reading previous instances of objects.
11 The storage backend of Pithos is block oriented, permitting efficient, deduplicated data placement. The block structure of objects is exposed at the API layer, in order to encourage external software to implement advanced data management operations.
13 This document's goals are:
15 * Define the Pithos ReST API that allows the storage and retrieval of data and metadata via HTTP calls
16 * Specify metadata semantics and user interface guidelines for a common experience across client software implementations
18 The present document is meant to be read alongside the OOS API documentation. Thus, it is suggested that the reader is familiar with associated technologies, the OOS API as well as the first version of the Pithos API. This document refers to the second version of Pithos. Information on the first version of the storage API can be found at http://code.google.com/p/gss.
20 Whatever marked as to be determined (**TBD**), should not be considered by implementors.
22 More info about Pithos can be found here: https://code.grnet.gr/projects/pithos
27 ========================= ================================
29 ========================= ================================
30 0.8 (Feb 9, 2012) Update allowed versioning values.
31 \ Change policy/meta formatting in JSON/XML replies.
32 \ Document that all non-ASCII characters in headers should be URL-encoded.
33 \ Support metadata-based queries when listing objects at the container level.
34 \ Note Content-Type issue when using the internal django web server.
35 \ Add object UUID field.
36 \ Always reply with the MD5 in the ETag.
37 \ Note that ``/login`` will only work if an external authentication system is defined.
38 \ Include option to ignore Content-Type on ``COPY``/``MOVE``.
39 \ Use format parameter for conflict (409) and uploaded hash list (container level) replies.
40 \ Change permissions model.
41 0.7 (Nov 21, 2011) Suggest upload/download methods using hashmaps.
42 \ Propose syncing algorithm.
43 \ Support cross-account object copy and move.
44 \ Pass token as a request parameter when using ``POST`` via an HTML form.
45 \ Optionally use source account to update object from another object.
46 \ Use container ``POST`` to upload missing blocks of data.
47 \ Report policy in account headers.
48 \ Add insufficient quota reply.
49 \ Use special meta to always report Merkle hash.
50 0.6 (Sept 13, 2011) Reply with Merkle hash as the ETag when updating objects.
51 \ Include version id in object replace/change replies.
52 \ Change conflict (409) replies format to text.
53 \ Tags should be migrated to a meta value.
54 \ Container ``PUT`` updates metadata/policy.
55 \ Report allowed actions in shared object replies.
56 \ Provide ``https://hostname/login`` for Shibboleth authentication.
57 \ Use ``hashmap`` parameter in object ``GET``/``PUT`` to use hashmaps.
58 0.5 (July 22, 2011) Object update from another object's data.
59 \ Support object truncate.
60 \ Create object using a standard HTML form.
61 \ Purge container/object history.
62 \ List other accounts that share objects with a user.
63 \ List shared containers/objects.
64 \ Update implementation guidelines.
65 \ Check preconditions when creating/updating objects.
66 0.4 (July 01, 2011) Object permissions and account groups.
67 \ Control versioning behavior and container quotas with container policy directives.
68 \ Support updating/deleting individual metadata with ``POST``.
69 \ Create object using hashmap.
70 0.3 (June 14, 2011) Large object support with ``X-Object-Manifest``.
71 \ Allow for publicly available objects via ``https://hostname/public``.
72 \ Support time-variant account/container listings.
73 \ Add source version when duplicating with ``PUT``/``COPY``.
74 \ Request version in object ``HEAD``/``GET`` requests (list versions with ``GET``).
75 0.2 (May 31, 2011) Add object meta listing and filtering in containers.
76 \ Include underlying storage characteristics in container meta.
77 \ Support for partial object updates through ``POST``.
78 \ Expose object hashmaps through ``GET``.
79 \ Support for multi-range object ``GET`` requests.
80 0.1 (May 17, 2011) Initial release. Based on OpenStack Object Storage Developer Guide API v1 (Apr. 15, 2011).
81 ========================= ================================
83 Pithos Users and Authentication
84 -------------------------------
86 In Pithos, each user is uniquely identified by a token. All API requests require a token and each token is internally resolved to an account string. The API uses the account string to identify the user's own files, thus whether a request is local or cross-account.
88 Pithos does not keep a user database. For development and testing purposes, user identifiers and their corresponding tokens can be defined in the settings file. However, Pithos is designed with an external authentication service in mind. This service must handle the details of validating user credentials and communicate with Pithos via a middleware software component that, given a token, fills in the internal request account variable.
90 Client software using Pithos, if not already knowing a user's identifier and token, should forward to the ``/login`` URI. The Pithos server, depending on its configuration will redirect to the appropriate login page.
92 The login URI accepts the following parameters:
94 ====================== =========================
95 Request Parameter Name Value
96 ====================== =========================
97 next The URI to redirect to when the process is finished
98 renew Force token renewal (no value parameter)
99 ====================== =========================
101 When done with logging in, the service's login URI should redirect to the URI provided with ``next``, adding ``user`` and ``token`` parameters, which contain the account and token fields respectively.
103 A user management service that implements a login URI according to these conventions is Astakos (https://code.grnet.gr/projects/astakos), by GRNET.
108 The URI requests supported by the Pithos API follow one of the following forms:
110 * Top level: ``https://hostname/v1/``
111 * Account level: ``https://hostname/v1/<account>``
112 * Container level: ``https://hostname/v1/<account>/<container>``
113 * Object level: ``https://hostname/v1/<account>/<container>/<object>``
115 All requests must include an ``X-Auth-Token`` - as a header, or a parameter.
117 The allowable request operations and respective return codes per level are presented in the remainder of this chapter. Common to all requests are the following return codes.
119 ========================= ================================
120 Return Code Description
121 ========================= ================================
122 400 (Bad Request) The request is invalid
123 401 (Unauthorized) Missing or invalid token
124 403 (Forbidden) Request not allowed
125 404 (Not Found) The requested resource was not found
126 503 (Service Unavailable) The request cannot be completed because of an internal error
127 ========================= ================================
134 ========= ==================
135 Operation Description
136 ========= ==================
137 GET Authentication (for compatibility with the OOS API) or list allowed accounts
138 ========= ==================
143 If the ``X-Auth-User`` and ``X-Auth-Key`` headers are given, a dummy ``X-Auth-Token`` and ``X-Storage-Url`` will be replied, which can be used as a guest token/namespace for testing Pithos.
145 ================ =====================
146 Return Code Description
147 ================ =====================
148 204 (No Content) The request succeeded
149 ================ =====================
151 If an ``X-Auth-Token`` is already present, the operation will be interpreted as a request to list other accounts that share objects to the user.
153 ====================== =========================
154 Request Parameter Name Value
155 ====================== =========================
156 limit The amount of results requested (default is 10000)
157 marker Return containers with name lexicographically after marker
158 format Optional extended reply type (can be ``json`` or ``xml``)
159 ====================== =========================
161 The reply is a list of account names.
162 If a ``format=xml`` or ``format=json`` argument is given, extended information on the accounts will be returned, serialized in the chosen format.
163 For each account, the information will include the following (names will be in lower case and with hyphens replaced with underscores):
165 =========================== ============================
167 =========================== ============================
168 name The name of the account
169 last_modified The last account modification date (regardless of ``until``)
170 =========================== ============================
172 Example ``format=json`` reply:
176 [{"name": "user", "last_modified": "2011-12-02T08:10:41.565891+00:00"}, ...]
178 Example ``format=xml`` reply:
182 <?xml version="1.0" encoding="UTF-8"?>
186 <last_modified>2011-12-02T08:10:41.565891+00:00</last_modified>
188 <account>...</account>
191 =========================== =====================
192 Return Code Description
193 =========================== =====================
194 200 (OK) The request succeeded
195 204 (No Content) The user has no access to other accounts (only for non-extended replies)
196 =========================== =====================
198 Will use a ``200`` return code if the reply is of type JSON/XML.
205 ========= ==================
206 Operation Description
207 ========= ==================
208 HEAD Retrieve account metadata
210 POST Update account metadata
211 ========= ==================
216 ==================== ===========================
217 Request Header Name Value
218 ==================== ===========================
219 If-Modified-Since Retrieve if account has changed since provided timestamp
220 If-Unmodified-Since Retrieve if account has not changed since provided timestamp
221 ==================== ===========================
225 ====================== ===================================
226 Request Parameter Name Value
227 ====================== ===================================
228 until Optional timestamp
229 ====================== ===================================
231 Cross-user requests are not allowed to use ``until`` and only include the account modification date in the reply.
233 ========================== =====================
234 Reply Header Name Value
235 ========================== =====================
236 X-Account-Container-Count The total number of containers
237 X-Account-Bytes-Used The total number of bytes stored
238 X-Account-Until-Timestamp The last account modification date until the timestamp provided
239 X-Account-Group-* Optional user defined groups
240 X-Account-Policy-* Account behavior and limits
241 X-Account-Meta-* Optional user defined metadata
242 Last-Modified The last account modification date (regardless of ``until``)
243 ========================== =====================
247 ================ =====================
248 Return Code Description
249 ================ =====================
250 204 (No Content) The request succeeded
251 ================ =====================
257 ==================== ===========================
258 Request Header Name Value
259 ==================== ===========================
260 If-Modified-Since Retrieve if account has changed since provided timestamp
261 If-Unmodified-Since Retrieve if account has not changed since provided timestamp
262 ==================== ===========================
266 ====================== =========================
267 Request Parameter Name Value
268 ====================== =========================
269 limit The amount of results requested (default is 10000)
270 marker Return containers with name lexicographically after marker
271 format Optional extended reply type (can be ``json`` or ``xml``)
272 shared Show only shared containers (no value parameter)
273 until Optional timestamp
274 ====================== =========================
276 The reply is a list of container names. Account headers (as in a ``HEAD`` request) will also be included.
277 Cross-user requests are not allowed to use ``until`` and only include the account/container modification dates in the reply.
279 If a ``format=xml`` or ``format=json`` argument is given, extended information on the containers will be returned, serialized in the chosen format.
280 For each container, the information will include all container metadata (names will be in lower case and with hyphens replaced with underscores):
282 =========================== ============================
284 =========================== ============================
285 name The name of the container
286 count The number of objects inside the container
287 bytes The total size of the objects inside the container
288 last_modified The last container modification date (regardless of ``until``)
289 x_container_until_timestamp The last container modification date until the timestamp provided
290 x_container_policy_* Container behavior and limits
291 x_container_meta_* Optional user defined metadata
292 =========================== ============================
294 Example ``format=json`` reply:
301 "last_modified": "2011-12-02T08:10:41.565891+00:00",
302 "x_container_policy": {"quota": "53687091200", "versioning": "auto"},
303 "x_container_meta": {"a": "b", "1": "2"}}, ...]
305 Example ``format=xml`` reply:
309 <?xml version="1.0" encoding="UTF-8"?>
310 <account name="user">
315 <last_modified>2011-12-02T08:10:41.565891+00:00</last_modified>
317 <key>quota</key><value>53687091200</value>
318 <key>versioning</key><value>auto</value>
319 </x_container_policy>
321 <key>a</key><value>b</value>
322 <key>1</key><value>2</value>
325 <container>...</container>
328 For more examples of container details returned in JSON/XML formats refer to the OOS API documentation. In addition to the OOS API, Pithos returns all fields. Policy and metadata values are grouped and returned as key-value pairs.
330 =========================== =====================
331 Return Code Description
332 =========================== =====================
333 200 (OK) The request succeeded
334 204 (No Content) The account has no containers (only for non-extended replies)
335 304 (Not Modified) The account has not been modified
336 412 (Precondition Failed) The condition set can not be satisfied
337 =========================== =====================
339 Will use a ``200`` return code if the reply is of type JSON/XML.
345 ==================== ===========================
346 Request Header Name Value
347 ==================== ===========================
348 X-Account-Group-* Optional user defined groups
349 X-Account-Meta-* Optional user defined metadata
350 ==================== ===========================
354 ====================== ============================================
355 Request Parameter Name Value
356 ====================== ============================================
357 update Do not replace metadata/groups (no value parameter)
358 ====================== ============================================
360 No reply content/headers.
362 The operation will overwrite all user defined metadata, except if ``update`` is defined.
363 To create a group, include an ``X-Account-Group-*`` header with the name in the key and a comma separated list of user names in the value. If no ``X-Account-Group-*`` header is present, no changes will be applied to groups. The ``update`` parameter also applies to groups. To delete a specific group, use ``update`` and an empty header value.
365 ================ ===============================
366 Return Code Description
367 ================ ===============================
368 202 (Accepted) The request has been accepted
369 ================ ===============================
377 ========= ============================
378 Operation Description
379 ========= ============================
380 HEAD Retrieve container metadata
382 PUT Create/update container
383 POST Update container metadata
384 DELETE Delete container
385 ========= ============================
391 ==================== ===========================
392 Request Header Name Value
393 ==================== ===========================
394 If-Modified-Since Retrieve if container has changed since provided timestamp
395 If-Unmodified-Since Retrieve if container has not changed since provided timestamp
396 ==================== ===========================
400 ====================== ===================================
401 Request Parameter Name Value
402 ====================== ===================================
403 until Optional timestamp
404 ====================== ===================================
406 Cross-user requests are not allowed to use ``until`` and only include the container modification date in the reply.
408 =========================== ===============================
409 Reply Header Name Value
410 =========================== ===============================
411 X-Container-Object-Count The total number of objects in the container
412 X-Container-Bytes-Used The total number of bytes of all objects stored
413 X-Container-Block-Size The block size used by the storage backend
414 X-Container-Block-Hash The hash algorithm used for block identifiers in object hashmaps
415 X-Container-Until-Timestamp The last container modification date until the timestamp provided
416 X-Container-Object-Meta A list with all meta keys used by objects (**TBD**)
417 X-Container-Policy-* Container behavior and limits
418 X-Container-Meta-* Optional user defined metadata
419 Last-Modified The last container modification date (regardless of ``until``)
420 =========================== ===============================
422 The keys returned in ``X-Container-Object-Meta`` are all the unique strings after the ``X-Object-Meta-`` prefix, formatted as a comma-separated list. See container ``PUT`` for a reference of policy directives. (**TBD**)
424 ================ ===============================
425 Return Code Description
426 ================ ===============================
427 204 (No Content) The request succeeded
428 ================ ===============================
434 ==================== ===========================
435 Request Header Name Value
436 ==================== ===========================
437 If-Modified-Since Retrieve if container has changed since provided timestamp
438 If-Unmodified-Since Retrieve if container has not changed since provided timestamp
439 ==================== ===========================
443 ====================== ===================================
444 Request Parameter Name Value
445 ====================== ===================================
446 limit The amount of results requested (default is 10000)
447 marker Return containers with name lexicographically after marker
448 prefix Return objects starting with prefix
449 delimiter Return objects up to the delimiter (discussion follows)
450 path Assume ``prefix=path`` and ``delimiter=/``
451 format Optional extended reply type (can be ``json`` or ``xml``)
452 meta Return objects that satisfy the key queries in the specified comma separated list (use ``<key>``, ``!<key>`` for existence queries, ``<key><op><value>`` for value queries, where ``<op>`` can be one of ``=``, ``!=``, ``<=``, ``>=``, ``<``, ``>``)
453 shared Show only shared objects (no value parameter)
454 until Optional timestamp
455 ====================== ===================================
457 The ``path`` parameter overrides ``prefix`` and ``delimiter``. When using ``path``, results will include objects ending in ``delimiter``.
459 The keys given with ``meta`` will be matched with the strings after the ``X-Object-Meta-`` prefix.
461 The reply is a list of object names. Container headers (as in a ``HEAD`` request) will also be included.
462 Cross-user requests are not allowed to use ``until`` and include the following limited set of headers in the reply:
464 =========================== ===============================
465 Reply Header Name Value
466 =========================== ===============================
467 X-Container-Block-Size The block size used by the storage backend
468 X-Container-Block-Hash The hash algorithm used for block identifiers in object hashmaps
469 X-Container-Object-Meta A list with all meta keys used by allowed objects (**TBD**)
470 Last-Modified The last container modification date
471 =========================== ===============================
473 If a ``format=xml`` or ``format=json`` argument is given, extended information on the objects will be returned, serialized in the chosen format.
474 For each object, the information will include all object metadata (names will be in lower case and with hyphens replaced with underscores):
476 ========================== ======================================
478 ========================== ======================================
479 name The name of the object
480 hash The ETag of the object
481 bytes The size of the object
482 content_type The MIME content type of the object
483 content_encoding The encoding of the object (optional)
484 content-disposition The presentation style of the object (optional)
485 last_modified The last object modification date (regardless of version)
486 x_object_hash The Merkle hash
487 x_object_uuid The object's UUID
488 x_object_version The object's version identifier
489 x_object_version_timestamp The object's version timestamp
490 x_object_modified_by The user that committed the object's version
491 x_object_manifest Object parts prefix in ``<container>/<object>`` form (optional)
492 x_object_sharing Object permissions (optional)
493 x_object_shared_by Object inheriting permissions (optional)
494 x_object_allowed_to Allowed actions on object (optional)
495 x_object_public Object's publicly accessible URI (optional)
496 x_object_meta_* Optional user defined metadata
497 ========================== ======================================
499 Sharing metadata will only be returned if there is no ``until`` parameter defined.
501 Extended replies may also include virtual directory markers in separate sections of the ``json`` or ``xml`` results.
502 Virtual directory markers are only included when ``delimiter`` is explicitly set. They correspond to the substrings up to and including the first occurrence of the delimiter.
503 In JSON results they appear as dictionaries with only a ``subdir`` key. In XML results they appear interleaved with ``<object>`` tags as ``<subdir name="..." />``.
504 In case there is an object with the same name as a virtual directory marker, the object will be returned.
506 Example ``format=json`` reply:
512 "hash": "d41d8cd98f00b204e9800998ecf8427e",
513 "content_type": "application/octet-stream",
514 "last_modified": "2011-12-02T08:10:41.565891+00:00",
515 "x_object_meta": {"asdf": "qwerty"},
516 "x_object_hash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
517 "x_object_uuid": "8ed9af1b-c948-4bb6-82b0-48344f5c822c",
518 "x_object_version": 98,
519 "x_object_version_timestamp": "1322813441.565891",
520 "x_object_modified_by": "user"}, ...]
522 Example ``format=xml`` reply:
526 <?xml version="1.0" encoding="UTF-8"?>
527 <container name="pithos">
531 <hash>d41d8cd98f00b204e9800998ecf8427e</hash>
532 <content_type>application/octet-stream</content_type>
533 <last_modified>2011-12-02T08:10:41.565891+00:00</last_modified>
535 <key>asdf</key><value>qwerty</value>
537 <x_object_hash>e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855</x_object_hash>
538 <x_object_uuid>8ed9af1b-c948-4bb6-82b0-48344f5c822c</x_object_uuid>
539 <x_object_version>98</x_object_version>
540 <x_object_version_timestamp>1322813441.565891</x_object_version_timestamp>
541 <x_object_modified_by>chazapis</x_object_modified_by>
546 For more examples of container details returned in JSON/XML formats refer to the OOS API documentation. In addition to the OOS API, Pithos returns all fields. Metadata values are grouped and returned as key-value pairs.
548 =========================== ===============================
549 Return Code Description
550 =========================== ===============================
551 200 (OK) The request succeeded
552 204 (No Content) The account has no containers (only for non-extended replies)
553 304 (Not Modified) The container has not been modified
554 412 (Precondition Failed) The condition set can not be satisfied
555 =========================== ===============================
557 Will use a ``200`` return code if the reply is of type JSON/XML.
563 ==================== ================================
564 Request Header Name Value
565 ==================== ================================
566 X-Container-Policy-* Container behavior and limits
567 X-Container-Meta-* Optional user defined metadata
568 ==================== ================================
570 No reply content/headers.
572 If no policy is defined, the container will be created with the default values.
573 Available policy directives:
575 * ``versioning``: Set to ``auto`` or ``none`` (default is ``auto``)
576 * ``quota``: Size limit in KB (default is ``0`` - unlimited)
578 If the container already exists, the operation is equal to a ``POST`` with ``update`` defined.
580 ================ ===============================
581 Return Code Description
582 ================ ===============================
583 201 (Created) The container has been created
584 202 (Accepted) The request has been accepted
585 ================ ===============================
591 ==================== ================================
592 Request Header Name Value
593 ==================== ================================
594 Content-Length The size of the supplied data (optional, to upload)
595 Content-Type The MIME content type of the supplied data (optional, to upload)
596 Transfer-Encoding Set to ``chunked`` to specify incremental uploading (if used, ``Content-Length`` is ignored)
597 X-Container-Policy-* Container behavior and limits
598 X-Container-Meta-* Optional user defined metadata
599 ==================== ================================
603 ====================== ============================================
604 Request Parameter Name Value
605 ====================== ============================================
606 format Optional hash list reply type (can be ``json`` or ``xml``)
607 update Do not replace metadata/policy (no value parameter)
608 ====================== ============================================
610 No reply content/headers, except when uploading data, where the reply consists of a list of hashes for the blocks received (in the format specified).
612 The operation will overwrite all user defined metadata, except if ``update`` is defined.
613 To change policy, include an ``X-Container-Policy-*`` header with the name in the key. If no ``X-Container-Policy-*`` header is present, no changes will be applied to policy. The ``update`` parameter also applies to policy - deleted values will revert to defaults. To delete/revert a specific policy directive, use ``update`` and an empty header value. See container ``PUT`` for a reference of policy directives.
615 To upload blocks of data to the container, set ``Content-Type`` to ``application/octet-stream`` and ``Content-Length`` to a valid value (except if using ``chunked`` as the ``Transfer-Encoding``).
617 ================ ===============================
618 Return Code Description
619 ================ ===============================
620 202 (Accepted) The request has been accepted
621 ================ ===============================
627 ====================== ===================================
628 Request Parameter Name Value
629 ====================== ===================================
630 until Optional timestamp
631 ====================== ===================================
633 If ``until`` is defined, the container is "purged" up to that time (the history of all objects up to then is deleted).
635 No reply content/headers.
637 ================ ===============================
638 Return Code Description
639 ================ ===============================
640 204 (No Content) The request succeeded
641 409 (Conflict) The container is not empty
642 ================ ===============================
650 ========= =================================
651 Operation Description
652 ========= =================================
653 HEAD Retrieve object metadata
655 PUT Write object data or copy/move object
658 POST Update object metadata/data
660 ========= =================================
666 ==================== ================================
667 Request Header Name Value
668 ==================== ================================
669 If-Match Retrieve if ETags match
670 If-None-Match Retrieve if ETags don't match
671 If-Modified-Since Retrieve if object has changed since provided timestamp
672 If-Unmodified-Since Retrieve if object has not changed since provided timestamp
673 ==================== ================================
677 ====================== ===================================
678 Request Parameter Name Value
679 ====================== ===================================
680 version Optional version identifier
681 ====================== ===================================
685 ========================== ===============================
686 Reply Header Name Value
687 ========================== ===============================
688 ETag The ETag of the object
689 Content-Length The size of the object
690 Content-Type The MIME content type of the object
691 Last-Modified The last object modification date (regardless of version)
692 Content-Encoding The encoding of the object (optional)
693 Content-Disposition The presentation style of the object (optional)
694 X-Object-Hash The Merkle hash
695 X-Object-UUID The object's UUID
696 X-Object-Version The object's version identifier
697 X-Object-Version-Timestamp The object's version timestamp
698 X-Object-Modified-By The user that comitted the object's version
699 X-Object-Manifest Object parts prefix in ``<container>/<object>`` form (optional)
700 X-Object-Sharing Object permissions (optional)
701 X-Object-Shared-By Object inheriting permissions (optional)
702 X-Object-Allowed-To Allowed actions on object (optional)
703 X-Object-Public Object's publicly accessible URI (optional)
704 X-Object-Meta-* Optional user defined metadata
705 ========================== ===============================
709 ================ ===============================
710 Return Code Description
711 ================ ===============================
712 200 (No Content) The request succeeded
713 ================ ===============================
719 ==================== ================================
720 Request Header Name Value
721 ==================== ================================
722 Range Optional range of data to retrieve
723 If-Range Retrieve the missing part if entity is unchanged; otherwise, retrieve the entire new entity (used together with Range header)
724 If-Match Retrieve if ETags match
725 If-None-Match Retrieve if ETags don't match
726 If-Modified-Since Retrieve if object has changed since provided timestamp
727 If-Unmodified-Since Retrieve if object has not changed since provided timestamp
728 ==================== ================================
732 ====================== ===================================
733 Request Parameter Name Value
734 ====================== ===================================
735 format Optional extended reply type (can be ``json`` or ``xml``)
736 hashmap Optional request for hashmap (no value parameter)
737 version Optional version identifier or ``list`` (specify a format if requesting a list)
738 ====================== ===================================
740 The reply is the object's data (or part of it), except if a hashmap is requested with ``hashmap``, or a version list with ``version=list`` (in both cases an extended reply format must be specified). Object headers (as in a ``HEAD`` request) are always included.
742 Hashmaps expose the underlying storage format of the object. Note that each hash is computed after trimming trailing null bytes of the corresponding block. The ``X-Object-Hash`` header reports the single Merkle hash of the object's hashmap (refer to http://bittorrent.org/beps/bep_0030.html for more information).
744 Example ``format=json`` reply:
748 {"block_hash": "sha1", "hashes": ["7295c41da03d7f916440b98e32c4a2a39351546c", ...], "block_size": 131072, "bytes": 242}
750 Example ``format=xml`` reply:
754 <?xml version="1.0" encoding="UTF-8"?>
755 <object name="file" bytes="24223726" block_size="131072" block_hash="sha1">
756 <hash>7295c41da03d7f916440b98e32c4a2a39351546c</hash>
760 Version lists include the version identifier and timestamp for each available object version. Version identifiers can be arbitrary strings, so use the timestamp to find newer versions.
762 Example ``format=json`` reply:
766 {"versions": [[85, "1322734861.248469"], [86, "1322734905.009272"], ...]}
768 Example ``format=xml`` reply:
772 <?xml version="1.0" encoding="UTF-8"?>
774 <version timestamp="1322734861.248469">85</version>
775 <version timestamp="1322734905.009272">86</version>
776 <version timestamp="...">...</version>
779 The ``Range`` header may include multiple ranges, as outlined in RFC2616. Then the ``Content-Type`` of the reply will be ``multipart/byteranges`` and each part will include a ``Content-Range`` header.
781 ========================== ===============================
782 Reply Header Name Value
783 ========================== ===============================
784 ETag The ETag of the object
785 Content-Length The size of the data returned
786 Content-Type The MIME content type of the object
787 Content-Range The range of data included (only on a single range request)
788 Last-Modified The last object modification date (regardless of version)
789 Content-Encoding The encoding of the object (optional)
790 Content-Disposition The presentation style of the object (optional)
791 X-Object-Hash The Merkle hash
792 X-Object-UUID The object's UUID
793 X-Object-Version The object's version identifier
794 X-Object-Version-Timestamp The object's version timestamp
795 X-Object-Modified-By The user that comitted the object's version
796 X-Object-Manifest Object parts prefix in ``<container>/<object>`` form (optional)
797 X-Object-Sharing Object permissions (optional)
798 X-Object-Shared-By Object inheriting permissions (optional)
799 X-Object-Allowed-To Allowed actions on object (optional)
800 X-Object-Public Object's publicly accessible URI (optional)
801 X-Object-Meta-* Optional user defined metadata
802 ========================== ===============================
804 Sharing headers (``X-Object-Sharing``, ``X-Object-Shared-By`` and ``X-Object-Allowed-To``) are only included if the request is for the object's latest version (no specific ``version`` parameter is set).
806 =========================== ==============================
807 Return Code Description
808 =========================== ==============================
809 200 (OK) The request succeeded
810 206 (Partial Content) The range request succeeded
811 304 (Not Modified) The object has not been modified
812 412 (Precondition Failed) The condition set can not be satisfied
813 416 (Range Not Satisfiable) The requested range is out of limits
814 =========================== ==============================
820 ==================== ================================
821 Request Header Name Value
822 ==================== ================================
823 If-Match Put if ETags match with current object
824 If-None-Match Put if ETags don't match with current object
825 ETag The MD5 hash of the object (optional to check written data)
826 Content-Length The size of the data written
827 Content-Type The MIME content type of the object
828 Transfer-Encoding Set to ``chunked`` to specify incremental uploading (if used, ``Content-Length`` is ignored)
829 X-Copy-From The source path in the form ``/<container>/<object>``
830 X-Move-From The source path in the form ``/<container>/<object>``
831 X-Source-Account The source account to copy/move from
832 X-Source-Version The source version to copy from
833 Content-Encoding The encoding of the object (optional)
834 Content-Disposition The presentation style of the object (optional)
835 X-Object-Manifest Object parts prefix in ``<container>/<object>`` form (optional)
836 X-Object-Sharing Object permissions (optional)
837 X-Object-Public Object is publicly accessible (optional)
838 X-Object-Meta-* Optional user defined metadata
839 ==================== ================================
843 ====================== ===================================
844 Request Parameter Name Value
845 ====================== ===================================
846 format Optional extended request/conflict response type (can be ``json`` or ``xml``)
847 hashmap Optional hashmap provided instead of data (no value parameter)
848 ====================== ===================================
850 The request is the object's data (or part of it), except if a hashmap is provided (using ``hashmap`` and ``format`` parameters). If using a hashmap and all different parts are stored in the server, the object is created. Otherwise the server returns Conflict (409) with the list of the missing parts (in simple text format, with one hash per line, or in JSON/XML - depending on the ``format`` parameter).
852 Hashmaps should be formatted as outlined in ``GET``.
854 ========================== ===============================
855 Reply Header Name Value
856 ========================== ===============================
857 ETag The MD5 hash of the object
858 X-Object-Version The object's new version
859 ========================== ===============================
861 The ``X-Object-Sharing`` header may include either a ``read=...`` comma-separated user/group list, or a ``write=...`` comma-separated user/group list, or both separated by a semicolon (``;``). Groups are specified as ``<account>:<group>``. To publish the object, set ``X-Object-Public`` to ``true``. To unpublish, set to ``false``, or use an empty header value.
863 ============================== ==============================
864 Return Code Description
865 ============================== ==============================
866 201 (Created) The object has been created
867 409 (Conflict) The object can not be created from the provided hashmap (a list of missing hashes will be included in the reply)
868 411 (Length Required) Missing ``Content-Length`` or ``Content-Type`` in the request
869 413 (Request Entity Too Large) Insufficient quota to complete the request
870 422 (Unprocessable Entity) The MD5 checksum of the data written to the storage system does not match the (optionally) supplied ETag value
871 ============================== ==============================
877 ==================== ================================
878 Request Header Name Value
879 ==================== ================================
880 If-Match Proceed if ETags match with object
881 If-None-Match Proceed if ETags don't match with object
882 Destination The destination path in the form ``/<container>/<object>``
883 Destination-Account The destination account to copy to
884 Content-Type The MIME content type of the object (optional :sup:`*`)
885 Content-Encoding The encoding of the object (optional)
886 Content-Disposition The presentation style of the object (optional)
887 X-Source-Version The source version to copy from
888 X-Object-Manifest Object parts prefix in ``<container>/<object>`` form (optional)
889 X-Object-Sharing Object permissions (optional)
890 X-Object-Public Object is publicly accessible (optional)
891 X-Object-Meta-* Optional user defined metadata
892 ==================== ================================
894 :sup:`*` *When using django locally with the supplied web server, use the ignore_content_type parameter, or do provide a valid Content-Type, as a type of text/plain is applied by default to all requests. Client software should always state ignore_content_type, except when a Content-Type is explicitly defined by the user.*
896 ====================== ===================================
897 Request Parameter Name Value
898 ====================== ===================================
899 format Optional conflict response type (can be ``json`` or ``xml``)
900 ignore_content_type Ignore the supplied Content-Type
901 ====================== ===================================
903 Refer to ``PUT``/``POST`` for a description of request headers. Metadata is also copied, updated with any values defined. Sharing/publishing options are not copied.
905 ========================== ===============================
906 Reply Header Name Value
907 ========================== ===============================
908 X-Object-Version The object's new version
909 ========================== ===============================
913 ============================== ==============================
914 Return Code Description
915 ============================== ==============================
916 201 (Created) The object has been created
917 413 (Request Entity Too Large) Insufficient quota to complete the request
918 ============================== ==============================
924 Same as ``COPY``, without the ``X-Source-Version`` request header. The ``MOVE`` operation is always applied on the latest version.
930 ==================== ================================
931 Request Header Name Value
932 ==================== ================================
933 If-Match Proceed if ETags match with object
934 If-None-Match Proceed if ETags don't match with object
935 Content-Length The size of the data written (optional, to update)
936 Content-Type The MIME content type of the object (optional, to update)
937 Content-Range The range of data supplied (optional, to update)
938 Transfer-Encoding Set to ``chunked`` to specify incremental uploading (if used, ``Content-Length`` is ignored)
939 Content-Encoding The encoding of the object (optional)
940 Content-Disposition The presentation style of the object (optional)
941 X-Source-Object Update with data from the object at path ``/<container>/<object>`` (optional, to update)
942 X-Source-Account The source account to update from
943 X-Source-Version The source version to update from (optional, to update)
944 X-Object-Bytes The updated object's final size (optional, when updating)
945 X-Object-Manifest Object parts prefix in ``<container>/<object>`` form (optional)
946 X-Object-Sharing Object permissions (optional)
947 X-Object-Public Object is publicly accessible (optional)
948 X-Object-Meta-* Optional user defined metadata
949 ==================== ================================
953 ====================== ============================================
954 Request Parameter Name Value
955 ====================== ============================================
956 format Optional conflict response type (can be ``json`` or ``xml``)
957 update Do not replace metadata (no value parameter)
958 ====================== ============================================
960 The ``Content-Encoding``, ``Content-Disposition``, ``X-Object-Manifest`` and ``X-Object-Meta-*`` headers are considered to be user defined metadata. An operation without the ``update`` parameter will overwrite all previous values and remove any keys not supplied. When using ``update`` any metadata with an empty value will be deleted.
962 To change permissions, include an ``X-Object-Sharing`` header (as defined in ``PUT``). To publish, include an ``X-Object-Public`` header, with a value of ``true``. If no such headers are defined, no changes will be applied to sharing/public. Use empty values to remove permissions/unpublish (unpublishing also works with ``false`` as a header value). Sharing options are applied to the object - not its versions.
964 To update an object's data:
966 * Either set ``Content-Type`` to ``application/octet-stream``, or provide an object with ``X-Source-Object``. If ``Content-Type`` has some other value, it will be ignored and only the metadata will be updated.
967 * If the data is supplied in the request (using ``Content-Type`` instead of ``X-Source-Object``), a valid ``Content-Length`` header is required - except if using chunked transfers (set ``Transfer-Encoding`` to ``chunked``).
968 * Set ``Content-Range`` as specified in RFC2616, with the following differences:
970 * Client software MAY omit ``last-byte-pos`` of if the length of the range being transferred is unknown or difficult to determine.
971 * Client software SHOULD not specify the ``instance-length`` (use a ``*``), unless there is a reason for performing a size check at the server.
972 * If ``Content-Range`` used has a ``byte-range-resp-spec = *``, data will be appended to the object.
974 Optionally, truncate the updated object to the desired length with the ``X-Object-Bytes`` header.
976 A data update will trigger an ETag change. Updated ETags may happen asynchronously and appear at the server with a delay.
978 No reply content. No reply headers if only metadata is updated.
980 ========================== ===============================
981 Reply Header Name Value
982 ========================== ===============================
983 ETag The new ETag of the object (data updated)
984 X-Object-Version The object's new version
985 ========================== ===============================
989 ============================== ==============================
990 Return Code Description
991 ============================== ==============================
992 202 (Accepted) The request has been accepted (not a data update)
993 204 (No Content) The request succeeded (data updated)
994 411 (Length Required) Missing ``Content-Length`` in the request
995 413 (Request Entity Too Large) Insufficient quota to complete the request
996 416 (Range Not Satisfiable) The supplied range is invalid
997 ============================== ==============================
999 The ``POST`` method can also be used for creating an object via a standard HTML form. If the request ``Content-Type`` is ``multipart/form-data``, none of the above headers will be processed. The form should have an ``X-Object-Data`` field, as in the following example. The token is passed as a request parameter. ::
1001 <form method="post" action="https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt?X-Auth-Token=0000" enctype="multipart/form-data">
1002 <input type="file" name="X-Object-Data">
1003 <input type="submit">
1006 This will create/override the object with the given name, as if using ``PUT``. The ``Content-Type`` of the object will be set to the value of the corresponding header sent in the part of the request containing the data (usually, automatically handled by the browser). Metadata, sharing and other object attributes can not be set this way. The response will contain the object's ETag.
1008 ========================== ===============================
1009 Reply Header Name Value
1010 ========================== ===============================
1011 ETag The MD5 hash of the object
1012 X-Object-Version The object's new version
1013 ========================== ===============================
1017 ============================== ==============================
1018 Return Code Description
1019 ============================== ==============================
1020 201 (Created) The object has been created
1021 413 (Request Entity Too Large) Insufficient quota to complete the request
1022 ============================== ==============================
1028 ====================== ===================================
1029 Request Parameter Name Value
1030 ====================== ===================================
1031 until Optional timestamp
1032 ====================== ===================================
1034 If ``until`` is defined, the object is "purged" up to that time (the history up to then is deleted).
1036 No reply content/headers.
1038 =========================== ==============================
1039 Return Code Description
1040 =========================== ==============================
1041 204 (No Content) The request succeeded
1042 =========================== ==============================
1044 Sharing and Public Objects
1045 ^^^^^^^^^^^^^^^^^^^^^^^^^^
1047 Read and write control in Pithos is managed by setting appropriate permissions with the ``X-Object-Sharing`` header. The permissions are applied using directory-based inheritance. A directory is an object with the corresponding content type. The default delimiter is ``/``. Thus, each set of authorization directives is applied to all objects in the directory object where the corresponding ``X-Object-Sharing`` header is defined. If there are nested/overlapping permissions, the closest to the object is applied. When retrieving an object, the ``X-Object-Shared-By`` header reports where it gets its permissions from. If not present, the object is the actual source of authorization directives.
1049 A user may ``GET`` another account or container. The result will include a limited reply, containing only the allowed containers or objects respectively. A top-level request with an authentication token, will return a list of allowed accounts, so the user can easily find out which other users share objects. The ``X-Object-Allowed-To`` header lists the actions allowed on an object, if it does not belong to the requesting user.
1051 Objects that are marked as public, via the ``X-Object-Public`` meta, are also available at the corresponding URI returned for ``HEAD`` or ``GET``. Requests for public objects do not need to include an ``X-Auth-Token``. Pithos will ignore request parameters and only include the following headers in the reply (all ``X-Object-*`` meta is hidden):
1053 ========================== ===============================
1054 Reply Header Name Value
1055 ========================== ===============================
1056 ETag The ETag of the object
1057 Content-Length The size of the data returned
1058 Content-Type The MIME content type of the object
1059 Content-Range The range of data included (only on a single range request)
1060 Last-Modified The last object modification date (regardless of version)
1061 Content-Encoding The encoding of the object (optional)
1062 Content-Disposition The presentation style of the object (optional)
1063 ========================== ===============================
1065 Public objects are not included and do not influence cross-user listings. They are, however, readable by all users.
1070 List of differences from the OOS API:
1072 * Support for ``X-Account-Meta-*`` style headers at the account level. Use ``POST`` to update.
1073 * Support for ``X-Container-Meta-*`` style headers at the container level. Can be set when creating via ``PUT``. Use ``POST`` to update.
1074 * Header ``X-Container-Object-Meta`` at the container level and parameter ``meta`` in container listings. (**TBD**)
1075 * Account and container policies to manage behavior and limits. Container behavior overrides account settings. Account quota sets the maximum bytes limit, regardless of container values.
1076 * Headers ``X-Container-Block-*`` at the container level, exposing the underlying storage characteristics.
1077 * All metadata replies, at all levels, include latest modification information.
1078 * At all levels, a ``HEAD`` or ``GET`` request may use ``If-Modified-Since`` and ``If-Unmodified-Since`` headers.
1079 * Container/object lists include all associated metadata if the reply is of type JSON/XML. Some names are kept to their OOS API equivalents for compatibility.
1080 * Option to include only shared containers/objects in listings.
1081 * Object metadata allowed, in addition to ``X-Object-Meta-*``: ``Content-Encoding``, ``Content-Disposition``, ``X-Object-Manifest``. These are all replaced with every update operation, except if using the ``update`` parameter (in which case individual keys can also be deleted). Deleting meta by providing empty values also works when copying/moving an object.
1082 * Multi-range object ``GET`` support as outlined in RFC2616.
1083 * Object hashmap retrieval through ``GET`` and the ``format`` parameter.
1084 * Object create via hashmap through ``PUT`` and the ``format`` parameter.
1085 * The object's Merkle hash is always returned in the ``X-Object-Hash`` header.
1086 * The object's UUID is always returned in the ``X-Object-UUID`` header. The UUID remains unchanged, even when the object's data or metadata changes, or the object is moved to another path (is renamed). A new UUID is assigned when creating or copying an object.
1087 * Object create using ``POST`` to support standard HTML forms.
1088 * Partial object updates through ``POST``, using the ``Content-Length``, ``Content-Type``, ``Content-Range`` and ``Transfer-Encoding`` headers. Use another object's data to update with ``X-Source-Object`` and ``X-Source-Version``. Truncate with ``X-Object-Bytes``.
1089 * Include new version identifier in replies for object replace/change requests.
1090 * Object ``MOVE`` support and ``ignore_content_type`` parameter in both ``COPY`` and ``MOVE``.
1091 * Conditional object create/update operations, using ``If-Match`` and ``If-None-Match`` headers.
1092 * Time-variant account/container listings via the ``until`` parameter.
1093 * Object versions - parameter ``version`` in ``HEAD``/``GET`` (list versions with ``GET``), ``X-Object-Version-*`` meta in replies, ``X-Source-Version`` in ``PUT``/``COPY``.
1094 * Sharing/publishing with ``X-Object-Sharing``, ``X-Object-Public`` at the object level. Cross-user operations are allowed - controlled by sharing directives. Available actions in cross-user requests are reported with ``X-Object-Allowed-To``. Permissions may include groups defined with ``X-Account-Group-*`` at the account level. These apply to the object - not its versions.
1095 * Support for prefix-based inheritance when enforcing permissions. Parent object carrying the authorization directives is reported in ``X-Object-Shared-By``.
1096 * Copy and move between accounts with ``X-Source-Account`` and ``Destination-Account`` headers.
1097 * Large object support with ``X-Object-Manifest``.
1098 * Trace the user that created/modified an object with ``X-Object-Modified-By``.
1099 * Purge container/object history with the ``until`` parameter in ``DELETE``.
1101 Clarifications/suggestions:
1103 * All non-ASCII characters in headers should be URL-encoded.
1104 * Authentication is done by another system. The token is used in the same way, but it is obtained differently. The top level ``GET`` request is kept compatible with the OOS API and allows for guest/testing operations.
1105 * Some processing is done in the variable part of all ``X-*-Meta-*`` headers. If it includes underscores, they will be converted to dashes and the first letter of all intra-dash strings will be capitalized.
1106 * A ``GET`` reply for a level will include all headers of the corresponding ``HEAD`` request.
1107 * To avoid conflicts between objects and virtual directory markers in container listings, it is recommended that object names do not end with the delimiter used.
1108 * The ``Accept`` header may be used in requests instead of the ``format`` parameter to specify the desired request/reply format. The parameter overrides the header.
1109 * Container/object lists use a ``200`` return code if the reply is of type JSON/XML. The reply will include an empty JSON/XML.
1110 * In headers, dates are formatted according to RFC 1123. In extended information listings, the ``last_modified`` field is formatted according to ISO 8601 (for OOS API compatibility). All other fields (Pithos extensions) use integer tiemstamps.
1111 * The ``Last-Modified`` header value always reflects the actual latest change timestamp, regardless of time control parameters and version requests. Time precondition checks with ``If-Modified-Since`` and ``If-Unmodified-Since`` headers are applied to this value.
1112 * A copy/move using ``PUT``/``COPY``/``MOVE`` will always update metadata, keeping all old values except the ones redefined in the request headers.
1113 * A ``HEAD`` or ``GET`` for an ``X-Object-Manifest`` object, will include modified ``Content-Length`` and ``ETag`` headers, according to the characteristics of the objects under the specified prefix. The ``Etag`` will be the MD5 hash of the corresponding ETags concatenated. In extended container listings there is no metadata processing.
1121 Hopefully this API will allow for a multitude of client implementations, each supporting a different device or operating system. All clients will be able to manipulate containers and objects - even software only designed for OOS API compatibility. But a Pithos interface should not be only about showing containers and folders. There are some extra user interface elements and functionalities that should be common to all implementations.
1123 Upon entrance to the service, a user is presented with the following elements - which can be represented as folders or with other related icons:
1125 * The ``home`` element, which is used as the default entry point to the user's "files". Objects under ``home`` are represented in the usual hierarchical organization of folders and files.
1126 * The ``trash`` element, which contains files that have been marked for deletion, but can still be recovered.
1127 * The ``shared`` element, which contains all objects shared by the user to other users of the system.
1128 * The ``others`` element, which contains all objects that other users share with the user.
1129 * The ``groups`` element, which contains the names of groups the user has defined. Each group consists of a user list. Group creation, deletion, and manipulation is carried out by actions originating here.
1130 * The ``history`` element, which allows browsing past instances of ``home`` and - optionally - ``trash``.
1132 Objects in Pithos can be:
1134 * Moved to trash and then deleted.
1135 * Shared with specific permissions.
1136 * Made public (shared with non-Pithos users).
1137 * Restored from previous versions.
1139 Some of these functions are performed by the client software and some by the Pithos server.
1141 In the first version of Pithos, objects could also be assigned custom tags. This is no longer supported. Existing deployments can migrate tags into a specific metadata value, i.e. ``X-Object-Meta-Tags``.
1143 Implementation Guidelines
1144 ^^^^^^^^^^^^^^^^^^^^^^^^^
1146 Pithos clients should use the ``pithos`` and ``trash`` containers for active and inactive objects respectively. If any of these containers is not found, the client software should create it, without interrupting the user's workflow. The ``home`` element corresponds to ``pithos`` and the ``trash`` element to ``trash``. Use ``PUT`` with the ``X-Move-From`` header, or ``MOVE`` to transfer objects from one container to the other. Use ``DELETE`` to remove from ``pithos`` without trashing, or to remove from ``trash``. When moving objects, detect naming conflicts with the ``If-Match`` or ``If-None-Match`` headers. Such conflicts should be resolved by the user.
1148 Object names should use the ``/`` delimiter to impose a hierarchy of folders and files.
1150 The ``shared`` element should be implemented as a read-only view of the ``pithos`` container, using the ``shared`` parameter when listing objects. The ``others`` element, should start with a top-level ``GET`` to retrieve the list of accounts accessible to the user. It is suggested that the client software hides the next step of navigation - the container - if it only includes ``pithos`` and forwards the user directly to the objects.
1152 Public objects are not included in ``shared`` and ``others`` listings. It is suggested that they are marked in a visually distinctive way in ``pithos`` listings (for example using an icon overlay).
1154 A special application menu, or a section in application preferences, should be devoted to managing groups (the ``groups`` element). All group-related actions are implemented at the account level.
1156 Browsing past versions of objects should be available both at the object and the container level. At the object level, a list of past versions can be included in the screen showing details or more information on the object (metadata, permissions, etc.). At the container level, it is suggested that clients use a ``history`` element, which presents to the user a read-only, time-variable view of ``pithos`` contents. This can be accomplished via the ``until`` parameter in listings. Optionally, ``history`` may include ``trash``.
1158 Uploading and downloading data
1159 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1161 By using hashmaps to upload and download objects the corresponding operations can complete much faster.
1163 In the case of an upload, only the missing blocks will be submitted to the server:
1165 * Calculate the hash value for each block of the object to be uploaded. Use the hash algorithm and block size of the destination container.
1166 * Send a hashmap ``PUT`` request for the object.
1168 * Server responds with status ``201`` (Created):
1170 * Blocks are already on the server. The object has been created. Done.
1172 * Server responds with status ``409`` (Conflict):
1174 * Server's response body contains the hashes of the blocks that do not exist on the server.
1175 * For each hash value in the server's response (or all hashes together):
1177 * Send a ``POST`` request to the destination container with the corresponding data.
1179 * Repeat hashmap ``PUT``. Fail if the server's response is not ``201``.
1181 Consulting hashmaps when downloading allows for resuming partially transferred objects. The client should retrieve the hashmap from the server and compare it with the hashmap computed from the respective local file. Any missing parts can be downloaded with ``GET`` requests with the additional ``Range`` header.
1186 Consider the following algorithm for synchronizing a local folder with the server. The "state" is the complete object listing, with the corresponding attributes.
1190 L: local state (stored state from last sync with the server)
1191 C: current state (state computed right before sync)
1197 # No remote changes, nothing to do
1199 # Update local state to match that of the server
1202 # We have local changes
1204 # No remote changes, update the server
1208 # Both we and server have changes
1210 # We were lucky, we did the same change
1213 # We have conflicting changes
1218 * States represent file hashes (it is suggested to use Merkle). Deleted or non-existing files are assumed to have a magic hash (e.g. empty string).
1219 * Updating a state (either local or remote) implies downloading, uploading or deleting the appropriate file.
1221 Recommended Practices and Examples
1222 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1224 Assuming an authentication token is obtained, the following high-level operations are available - shown with ``curl``:
1226 * Get account information ::
1229 -H "X-Auth-Token: 0000" \
1230 https://pithos.dev.grnet.gr/v1/user
1232 * List available containers ::
1235 -H "X-Auth-Token: 0000" \
1236 https://pithos.dev.grnet.gr/v1/user
1238 * Get container information ::
1241 -H "X-Auth-Token: 0000" \
1242 https://pithos.dev.grnet.gr/v1/user/pithos
1244 * Add a new container ::
1247 -H "X-Auth-Token: 0000" \
1248 https://pithos.dev.grnet.gr/v1/user/test
1250 * Delete a container ::
1252 curl -X DELETE -D - \
1253 -H "X-Auth-Token: 0000" \
1254 https://pithos.dev.grnet.gr/v1/user/test
1256 * List objects in a container ::
1259 -H "X-Auth-Token: 0000" \
1260 https://pithos.dev.grnet.gr/v1/user/pithos
1262 * List objects in a container (extended reply) ::
1265 -H "X-Auth-Token: 0000" \
1266 https://pithos.dev.grnet.gr/v1/user/pithos?format=json
1268 It is recommended that extended replies are cached and subsequent requests utilize the ``If-Modified-Since`` header.
1270 * List metadata keys used by objects in a container
1272 Will be in the ``X-Container-Object-Meta`` reply header, included in container information or object list (``HEAD`` or ``GET``). (**TBD**)
1274 * List objects in a container having a specific meta defined ::
1277 -H "X-Auth-Token: 0000" \
1278 https://pithos.dev.grnet.gr/v1/user/pithos?meta=favorites
1280 * Retrieve an object ::
1283 -H "X-Auth-Token: 0000" \
1284 https://pithos.dev.grnet.gr/v1/user/pithos/README.txt
1286 * Retrieve an object (specific ranges of data) ::
1289 -H "X-Auth-Token: 0000" \
1290 -H "Range: bytes=0-9" \
1291 https://pithos.dev.grnet.gr/v1/user/pithos/README.txt
1293 This will return the first 10 bytes. To get the first 10, bytes 30-39 and the last 100 use ``Range: bytes=0-9,30-39,-100``.
1295 * Add a new object (folder type) (**TBD**) ::
1298 -H "X-Auth-Token: 0000" \
1299 -H "Content-Type: application/directory" \
1300 https://pithos.dev.grnet.gr/v1/user/pithos/folder
1302 * Add a new object ::
1305 -H "X-Auth-Token: 0000" \
1306 -H "Content-Type: text/plain" \
1308 https://pithos.dev.grnet.gr/v1/user/pithos/folder/EXAMPLE.txt
1310 * Update an object ::
1313 -H "X-Auth-Token: 0000" \
1314 -H "Content-Length: 10" \
1315 -H "Content-Type: application/octet-stream" \
1316 -H "Content-Range: bytes 10-19/*" \
1318 https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt
1320 This will update bytes 10-19 with the data specified.
1322 * Update an object (append) ::
1325 -H "X-Auth-Token: 0000" \
1326 -H "Content-Length: 10" \
1327 -H "Content-Type: application/octet-stream" \
1328 -H "Content-Range: bytes */*" \
1330 https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt
1332 * Update an object (truncate) ::
1335 -H "X-Auth-Token: 0000" \
1336 -H "X-Source-Object: /folder/EXAMPLE.txt" \
1337 -H "Content-Range: bytes 0-0/*" \
1338 -H "X-Object-Bytes: 0" \
1339 https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt
1341 This will truncate the object to 0 bytes.
1343 * Add object metadata ::
1346 -H "X-Auth-Token: 0000" \
1347 -H "X-Object-Meta-First: first_meta_value" \
1348 -H "X-Object-Meta-Second: second_meta_value" \
1349 https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt
1351 * Delete object metadata ::
1354 -H "X-Auth-Token: 0000" \
1355 -H "X-Object-Meta-First: first_meta_value" \
1356 https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt
1358 Metadata can only be "set". To delete ``X-Object-Meta-Second``, reset all metadata.
1360 * Delete an object ::
1362 curl -X DELETE -D - \
1363 -H "X-Auth-Token: 0000" \
1364 https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt
projects / pithos / blob