1 Pithos v2 Developer Guide
2 =========================
7 Pithos is a storage service implemented by GRNET (http://www.grnet.gr). Data is stored as objects, organized in containers, belonging to an account. This hierarchy of storage layers has been inspired by the OpenStack Object Storage (OOS) API and similar CloudFiles API by Rackspace. The Pithos API follows the OOS API as closely as possible. One of the design requirements has been to be able to use Pithos with clients built for the OOS, without changes.
9 This document's goals are:
11 * Define the Pithos ReST API that allows the storage and retrieval of data and metadata via HTTP calls
12 * Specify metadata semantics and user interface guidelines for a common experience across client software implementations
14 The present document is meant to be read alongside the OOS API documentation. Thus, it is suggested that the reader is familiar with associated technologies, the OOS API as well as the first version of the Pithos API. This document refers to the second version of Pithos. Information on the first version of the storage API can be found at http://code.google.com/p/gss.
16 Whatever marked as to be determined (**TBD**), should not be considered by implementors.
21 ========================= ================================
23 ========================= ================================
24 0.2 (May 25, 2011) Add object meta listing and filtering in containers.
25 \ Support for multi-range object GET requests.
26 0.1 (May 17, 2011) Initial release. Based on OpenStack Object Storage Developer Guide API v1 (Apr. 15, 2011).
27 ========================= ================================
32 The URI requests supported by Pithos follow one of the following forms:
34 * Top level: ``https://hostname/v1/``
35 * Account level: ``https://hostname/v1/<account>``
36 * Container level: ``https://hostname/v1/<account>/<container>``
37 * Object level: ``https://hostname/v1/<account>/<container>/<object>``
39 All requests must include an ``X-Auth-Token``, except from those that refer to publicly available files (**TBD**). The process of obtaining the token is still to be determined (**TBD**).
41 The allowable request operations and corresponding return codes per level are presented in the remainder of this chapter. Common to all requests are the following return codes.
43 ========================= ================================
44 Return Code Description
45 ========================= ================================
46 400 (Bad Request) The request is invalid
47 401 (Unauthorized) Request not allowed
48 404 (Not Found) The requested resource was not found
49 503 (Service Unavailable) The request cannot be completed because of an internal error
50 ========================= ================================
57 ========= ==================
59 ========= ==================
60 GET Authentication. This is kept for compatibility with the OOS API
61 ========= ==================
66 If the ``X-Auth-User`` and ``X-Auth-Key`` headers are given, a dummy ``X-Auth-Token`` and ``X-Storage-Url`` will be replied, which can be used as a guest token/namespace for testing Pithos.
68 ================ =====================
69 Return Code Description
70 ================ =====================
71 204 (No Content) The request succeeded
72 ================ =====================
80 ========= ==================
82 ========= ==================
83 HEAD Retrieve account metadata
85 POST Update account metadata
86 ========= ==================
91 No request parameters/headers.
93 ========================== =====================
94 Reply Header Name Value
95 ========================== =====================
96 X-Account-Container-Count The total number of containers
97 X-Account-Object-Count The total number of objects (**TBD**)
98 X-Account-Bytes-Used The total number of bytes stored
99 X-Account-Bytes-Remaining The total number of bytes remaining (**TBD**)
100 X-Account-Last-Login The last login (**TBD**)
101 X-Account-Meta-* Optional user defined metadata
102 Last-Modified The last object modification date
103 ========================== =====================
107 ================ =====================
108 Return Code Description
109 ================ =====================
110 204 (No Content) The request succeeded
111 ================ =====================
117 ==================== ===========================
118 Request Header Name Value
119 ==================== ===========================
120 If-Modified-Since Retrieve if account has changed since provided timestamp
121 If-Unmodified-Since Retrieve if account has not changed since provided timestamp
122 ==================== ===========================
126 ====================== =========================
127 Request Parameter Name Value
128 ====================== =========================
129 limit The amount of results requested (default is 10000)
130 marker Return containers with name lexicographically after marker
131 format Optional extended reply type (can be ``json`` or ``xml``)
132 ====================== =========================
134 The reply is a list of container names. Account headers (as in a ``HEAD`` request) will also be included.
135 If a ``format=xml`` or ``format=json`` argument is given, extended information on the containers will be returned, serialized in the chosen format.
136 For each container, the information will include all container metadata (names will be in lower case and with hyphens replaced with underscores):
138 =================== ============================
140 =================== ============================
141 name The name of the container
142 count The number of objects inside the container
143 bytes The total size of the objects inside the container
144 last_modified The last object modification date
145 x_container_meta_* Optional user defined metadata
146 =================== ============================
148 For examples of container details returned in JSON/XML formats refer to the OOS API documentation.
150 ================ =====================
151 Return Code Description
152 ================ =====================
153 200 (OK) The request succeeded
154 204 (No Content) The account has no containers (only for non-extended replies)
155 ================ =====================
157 Will use a ``200`` return code if the reply is of type json/xml.
163 ==================== ===========================
164 Request Header Name Value
165 ==================== ===========================
166 X-Account-Meta-* Optional user defined metadata
167 ==================== ===========================
169 No reply content/headers.
171 The update operation will overwrite all user defined metadata.
173 ================ ===============================
174 Return Code Description
175 ================ ===============================
176 202 (Accepted) The request has been accepted
177 ================ ===============================
185 ========= ============================
186 Operation Description
187 ========= ============================
188 HEAD Retrieve container metadata
190 PUT Create/update container
191 POST Update container metadata
192 DELETE Delete container
193 ========= ============================
199 No request parameters/headers.
201 ========================== ===============================
202 Reply Header Name Value
203 ========================== ===============================
204 X-Container-Object-Count The total number of objects in the container
205 X-Container-Bytes-Used The total number of bytes of all objects stored
206 X-Container-Meta-* Optional user defined metadata
207 X-Container-Object-Meta A list with all meta keys used by objects
208 Last-Modified The last object modification date
209 ========================== ===============================
211 The keys returned in ``X-Container-Object-Meta`` are all the unique strings after the ``X-Object-Meta-`` prefix.
213 ================ ===============================
214 Return Code Description
215 ================ ===============================
216 204 (No Content) The request succeeded
217 ================ ===============================
223 ==================== ===========================
224 Request Header Name Value
225 ==================== ===========================
226 If-Modified-Since Retrieve if container has changed since provided timestamp
227 If-Unmodified-Since Retrieve if container has not changed since provided timestamp
228 ==================== ===========================
232 ====================== ===================================
233 Request Parameter Name Value
234 ====================== ===================================
235 limit The amount of results requested (default is 10000)
236 marker Return containers with name lexicographically after marker
237 prefix Return objects starting with prefix
238 delimiter Return objects up to the delimiter (discussion follows)
239 path Assume ``prefix=path`` and ``delimiter=/``
240 format Optional extended reply type (can be ``json`` or ``xml``)
241 meta Return objects having the specified meta keys (can be a comma separated list)
242 ====================== ===================================
244 The ``path`` parameter overrides ``prefix`` and ``delimiter``. When using ``path``, results will include objects ending in ``delimiter``.
246 The keys given with ``meta`` will be matched with the strings after the ``X-Object-Meta-`` prefix.
248 The reply is a list of object names. Container headers (as in a ``HEAD`` request) will also be included.
249 If a ``format=xml`` or ``format=json`` argument is given, extended information on the objects will be returned, serialized in the chosen format.
250 For each object, the information will include all object metadata (names will be in lower case and with hyphens replaced with underscores):
252 =================== ======================================
254 =================== ======================================
255 name The name of the object
256 hash The MD5 hash of the object
257 bytes The size of the object
258 content_type The MIME content type of the object
259 content_encoding The encoding of the object (optional)
260 last_modified The last object modification date
261 x_object_manifest Large object support
262 x_object_meta_* Optional user defined metadata
263 =================== ======================================
265 Extended replies may also include virtual directory markers in separate sections of the ``json`` or ``xml`` results.
266 Virtual directory markers are only included when ``delimiter`` is explicitly set. They correspond to the substrings up to and including the first occurrence of the delimiter.
267 In JSON results they appear as dictionaries with only a ``"subdir"`` key. In XML results they appear interleaved with ``<object>`` tags as ``<subdir name="..." />``.
268 In case there is an object with the same name as a virtual directory marker, the object will be returned.
270 For examples of object details returned in JSON/XML formats refer to the OOS API documentation.
272 ================ ===============================
273 Return Code Description
274 ================ ===============================
275 200 (OK) The request succeeded
276 204 (No Content) The account has no containers (only for non-extended replies)
277 ================ ===============================
279 Will use a ``200`` return code if the reply is of type json/xml.
285 ==================== ================================
286 Request Header Name Value
287 ==================== ================================
288 X-Container-Meta-* Optional user defined metadata
289 ==================== ================================
291 No reply content/headers.
293 ================ ===============================
294 Return Code Description
295 ================ ===============================
296 201 (Created) The container has been created
297 202 (Accepted) The request has been accepted
298 ================ ===============================
304 ==================== ================================
305 Request Header Name Value
306 ==================== ================================
307 X-Container-Meta-* Optional user defined metadata
308 ==================== ================================
310 No reply content/headers.
312 The update operation will overwrite all user defined metadata.
314 ================ ===============================
315 Return Code Description
316 ================ ===============================
317 202 (Accepted) The request has been accepted
318 ================ ===============================
324 No request parameters/headers.
326 No reply content/headers.
328 ================ ===============================
329 Return Code Description
330 ================ ===============================
331 204 (No Content) The request succeeded
332 409 (Conflict) The container is not empty
333 ================ ===============================
341 ========= =================================
342 Operation Description
343 ========= =================================
344 HEAD Retrieve object metadata
346 PUT Write object data or copy/move object
349 POST Update object metadata
351 ========= =================================
357 No request parameters/headers.
359 ========================== ===============================
360 Reply Header Name Value
361 ========================== ===============================
362 ETag The MD5 hash of the object
363 Content-Length The size of the object
364 Content-Type The MIME content type of the object
365 Last-Modified The last object modification date
366 Content-Encoding The encoding of the object (optional)
367 Content-Disposition The presentation style of the object (optional)
368 X-Object-Manifest Large object support (optional)
369 X-Object-Meta-* Optional user defined metadata
370 ========================== ===============================
374 ================ ===============================
375 Return Code Description
376 ================ ===============================
377 204 (No Content) The request succeeded
378 ================ ===============================
384 ==================== ================================
385 Request Header Name Value
386 ==================== ================================
387 Range Optional range of data to retrieve
388 If-Match Retrieve if ETags match
389 If-None-Match Retrieve if ETags don't match
390 If-Modified-Since Retrieve if object has changed since provided timestamp
391 If-Unmodified-Since Retrieve if object has not changed since provided timestamp
392 ==================== ================================
394 The reply is the object's data (or part of it). Object headers (as in a ``HEAD`` request) will also be included.
396 The ``Range`` header may include multiple ranges, as outlined in RFC2616. Then the ``Content-Type`` of the reply will be ``multipart/byteranges`` and each part will include a ``Content-Range`` header.
398 ========================== ===============================
399 Reply Header Name Value
400 ========================== ===============================
401 ETag The MD5 hash of the object
402 Content-Length The size of the data returned
403 Content-Type The MIME content type of the object
404 Content-Range The range of data included (only on a single range request)
405 Last-Modified The last object modification date
406 Content-Encoding The encoding of the object (optional)
407 Content-Disposition The presentation style of the object (optional)
408 X-Object-Manifest Large object support (optional)
409 X-Object-Meta-* Optional user defined metadata
410 ========================== ===============================
414 =========================== ==============================
415 Return Code Description
416 =========================== ==============================
417 200 (OK) The request succeeded
418 206 (Partial Content) The range request succeeded
419 304 (Not Modified) The object has not been modified
420 412 (Precondition Failed) The condition set can not be satisfied
421 416 (Range Not Satisfiable) The requested range is out of limits
422 =========================== ==============================
428 ==================== ================================
429 Request Header Name Value
430 ==================== ================================
431 ETag The MD5 hash of the object (optional to check written data)
432 Content-Length The size of the data written
433 Content-Type The MIME content type of the object
434 Transfer-Encoding Set to ``chunked`` to specify incremental uploading (if used, ``Content-Length`` is ignored)
435 X-Copy-From The source path in the form ``/<container>/<object>``
436 X-Move-From The source path in the form ``/<container>/<object>``
437 Content-Encoding The encoding of the object (optional)
438 Content-Disposition The presentation style of the object (optional)
439 X-Object-Manifest Large object support (optional)
440 X-Object-Meta-* Optional user defined metadata
441 ==================== ================================
445 ========================== ===============================
446 Reply Header Name Value
447 ========================== ===============================
448 ETag The MD5 hash of the object (on create)
449 ========================== ===============================
453 =========================== ==============================
454 Return Code Description
455 =========================== ==============================
456 201 (Created) The object has been created
457 411 (Length Required) Missing ``Content-Length`` or ``Content-Type`` in the request
458 422 (Unprocessable Entity) The MD5 checksum of the data written to the storage system does not match the (optionally) supplied ETag value
459 =========================== ==============================
465 ==================== ================================
466 Request Header Name Value
467 ==================== ================================
468 Destination The destination path in the form ``/<container>/<object>``
469 Content-Type The MIME content type of the object (optional)
470 Content-Encoding The encoding of the object (optional)
471 Content-Disposition The presentation style of the object (optional)
472 X-Object-Manifest Large object support (optional)
473 X-Object-Meta-* Optional user defined metadata
474 ==================== ================================
476 No reply content/headers.
478 =========================== ==============================
479 Return Code Description
480 =========================== ==============================
481 201 (Created) The object has been created
482 =========================== ==============================
494 ==================== ================================
495 Request Header Name Value
496 ==================== ================================
497 Content-Encoding The encoding of the object (optional)
498 Content-Disposition The presentation style of the object (optional)
499 X-Object-Manifest Large object support (optional)
500 X-Object-Meta-* Optional user defined metadata
501 ==================== ================================
503 No reply content/headers.
505 The allowed headers are considered to be user defined metadata. The update operation will overwrite all previous values and remove any keys not supplied.
507 =========================== ==============================
508 Return Code Description
509 =========================== ==============================
510 202 (Accepted) The request has been accepted
511 =========================== ==============================
517 No request parameters/headers.
519 No reply content/headers.
521 =========================== ==============================
522 Return Code Description
523 =========================== ==============================
524 204 (No Content) The request succeeded
525 =========================== ==============================
531 List of differences from the OOS API:
533 * Support for ``X-Account-Meta-*`` style headers at the account level. Use ``POST`` to update.
534 * Support for ``X-Container-Meta-*`` style headers at the account level. Can be set when creating via ``PUT``. Use ``POST`` to update.
535 * Header ``X-Container-Object-Meta`` at the container level and parameter ``meta`` in container listings.
536 * All metadata replies, at all levels, include latest modification information.
537 * At all levels, a ``GET`` request may use ``If-Modified-Since`` and ``If-Unmodified-Since`` headers.
538 * Container/object lists include all associated metadata if the reply is of type json/xml. Some names are kept to their OOS API equivalents for compatibility.
539 * Object metadata allowed, in addition to ``X-Object-Meta-*``: ``Content-Encoding``, ``Content-Disposition``, ``X-Object-Manifest``. These are all replaced with every update operation.
540 * Multi-range object GET support as outlined in RFC2616.
541 * Object ``MOVE`` support.
543 Clarifications/suggestions:
545 * Authentication is done by another system. The token is used in the same way, but it is obtained differently. The top level ``GET`` request is kept compatible with the OOS API and allows for guest/testing operations.
546 * Some processing is done in the variable part of all ``X-*-Meta-*`` headers. If it includes underscores, they will be converted to dashes and the first letter of all intra-dash strings will be capitalized.
547 * A ``GET`` reply for a level will include all headers of the corresponding ``HEAD`` request.
548 * To avoid conflicts between objects and virtual directory markers in container listings, it is recommended that object names do not end with the delimiter used.
549 * The ``Accept`` header may be used in requests instead of the ``format`` parameter to specify the desired reply format. The parameter overrides the header.
550 * Container/object lists use a ``200`` return code if the reply is of type json/xml. The reply will include an empty json/xml.
551 * In headers, dates are formatted according to RFC 1123. In extended information listings, dates are formatted according to ISO 8601.
552 * While ``X-Object-Manifest`` can be set and unset, large object support is not yet implemented (**TBD**).
560 Hopefully this API will allow for a multitude of client implementations, each supporting a different device or operating system. All clients will be able to manipulate containers and objects - even software only designed for OOS API compatibility. But a Pithos interface should not be only about showing containers and folders. There are some extra user interface elements and functionalities that should be common to all implementations.
562 Upon entrance to the service, a user is presented with the following elements - which can be represented as folders or with other related icons:
564 * The ``home`` element, which is used as the default entry point to the user's "files". Objects under ``home`` are represented in the usual hierarchical organization of folders and files.
565 * The ``trash`` element, which contains files that have been marked for deletion, but can still be recovered.
566 * The ``shared`` element, which contains all objects shared by the user to other users of the system.
567 * The ``others`` element, which contains all objects that other users share with the user.
568 * The ``tags`` element, which lists the names of tags the user has defined. This can be an entry point to list all files that have been assigned a specific tag or manage tags in general (remove a tag completely, rename a tag etc.).
569 * The ``groups`` element, which contains the names of groups the user has defined. Each group consists of a user list. Group creation, deletion, and manipulation is carried out by actions originating here.
571 Objects in Pithos can be:
573 * Assigned custom tags.
574 * Moved to trash and then deleted.
575 * Shared with specific permissions.
576 * Made public (shared with non-Pithos users).
577 * Set to monitor changes via version tracking.
579 Some of these functions are performed by the client software and some by the Pithos server. Client-driven functionality is based on specific metadata that should be handled equally across implementations. These metadata names are discussed in the next chapter.
581 Conventions and Metadata Specification
582 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
584 Pithos clients should use the ``pithos`` container for all Pithos objects. Object names use the ``/`` delimiter to impose a hierarchy of folders and files.
586 At the object level, tags are implemented by managing metadata keys. The client software should allow the user to use any string as a tag (except ``trash``) and then set the corresponding ``X-Object-Meta-<tag>`` key at the server. The API extensions provided, allow for listing all tags in a container and filtering object listings based on one or more tags. The tag list is sufficient for implementing the ``tags`` element, either as a special, virtual folder (as done in the first version of Pithos), or as an application menu.
588 To manage the deletion of files use the same API and the ``X-Object-Meta-Trash`` key. The string ``trash`` can not be used as a tag. The ``trash`` element should be presented as a folder, although with no hierarchy.
590 The metadata specification is summarized in the following table.
592 =========================== ==============================
594 =========================== ==============================
595 X-Object-Meta-Trash Set to ``true`` if the object has been moved to the trash
596 X-Object-Meta-* Use for other tags that apply to the object
597 =========================== ==============================
projects / pithos / blob