Whatever marked as to be determined (**TBD**), should not be considered by implementors.
+More info about Pithos can be found here: https://code.grnet.gr/projects/pithos
+
Document Revisions
^^^^^^^^^^^^^^^^^^
========================= ================================
Revision Description
========================= ================================
-0.8 (Dec 2, 2011) Update allowed versioning values.
+0.8 (Jan 24, 2012) Update allowed versioning values.
\ Change policy/meta formatting in JSON/XML replies.
\ Document that all non-ASCII characters in headers should be URL-encoded.
+\ Support metadata-based queries when listing objects at the container level.
+\ Note Content-Type issue when using the internal django web server.
+\ Add object UUID field.
+\ Always reply with the MD5 in the ETag.
+\ Note that ``/login`` will only work if an external authentication system is defined.
+\ Include option to ignore Content-Type on ``COPY``/``MOVE``.
+\ Use format parameter for conflict (409) replies.
0.7 (Nov 21, 2011) Suggest upload/download methods using hashmaps.
\ Propose syncing algorithm.
\ Support cross-account object copy and move.
Pithos Users and Authentication
-------------------------------
-Pithos keeps separate databases for users and objects.
-
-Each user is uniquely identified by the ``Uniq`` field. This should be used as the user's account in the API. The API uses the ``Token`` field to authenticate a user, thus allowing cross-account requests. All API requests require a token.
+In Pithos, each user is uniquely identified by a token. All API requests require a token and each token is internally resolved to an account string. The API uses the account string to identify the user's own files, thus whether a request is local or cross-account.
-User entries can be modified/added via the management interface available at ``https://hostname/admin``.
+Pithos does not keep a user database. For development and testing purposes, user identifiers and their corresponding tokens can be defined in the settings file. However, Pithos is designed with an external authentication service in mind. This service must handle the details of validating user credentials and communicate with Pithos via a middleware software component that, given a token, fills in the internal request account variable.
-Pithos is also compatible with Shibboleth (http://shibboleth.internet2.edu/). The connection between Shibboleth and Pithos is done by ``https://hostname/login``. An application that wishes to connect to Pithos, but does not have a token, should redirect the user to the login URI.
+Client software using Pithos, if not already knowing a user's identifier and token, should forward to the ``/login`` URI. The Pithos server, depending on its configuration will redirect to the appropriate login page.
The login URI accepts the following parameters:
renew Force token renewal (no value parameter)
====================== =========================
-The login process starts by redirecting the user to an external URI (controlled by Shibboleth), where the actual authentication credentials are entered. Then, the user is redirected back to the login URI from Shibboleth, with various identification information in the request headers.
+When done with logging in, the service's login URI should redirect to the URI provided with ``next``, adding ``user`` and ``token`` parameters, which contain the account and token fields respectively.
-If the user does not exist in the database, Pithos adds the user and creates a random token. If the user exists, the token has not expired and ``renew`` is not set, the existing token is reused. Finally, the login URI redirects to the URI provided with ``next``, adding the ``user`` and ``token`` parameters, which contain the ``Uniq`` and ``Token`` fields respectively.
+A user management service that implements a login URI according to these conventions is Astakos (https://code.grnet.gr/projects/astakos), by GRNET.
The Pithos API
--------------
delimiter Return objects up to the delimiter (discussion follows)
path Assume ``prefix=path`` and ``delimiter=/``
format Optional extended reply type (can be ``json`` or ``xml``)
-meta Return objects having the specified meta keys (can be a comma separated list)
+meta Return objects that satisfy the key queries in the specified comma separated list (use ``<key>``, ``!<key>`` for existence queries, ``<key><op><value>`` for value queries, where ``<op>`` can be one of ``=``, ``!=``, ``<=``, ``>=``, ``<``, ``>``)
shared Show only shared objects (no value parameter)
until Optional timestamp
====================== ===================================
content-disposition The presentation style of the object (optional)
last_modified The last object modification date (regardless of version)
x_object_hash The Merkle hash
+x_object_uuid The object's UUID
x_object_version The object's version identifier
x_object_version_timestamp The object's version timestamp
x_object_modified_by The user that committed the object's version
x_object_meta_* Optional user defined metadata
========================== ======================================
+Sharing metadata will only be returned if there is no ``until`` parameter defined.
+
Extended replies may also include virtual directory markers in separate sections of the ``json`` or ``xml`` results.
Virtual directory markers are only included when ``delimiter`` is explicitly set. They correspond to the substrings up to and including the first occurrence of the delimiter.
-In JSON results they appear as dictionaries with only a ``"subdir"`` key. In XML results they appear interleaved with ``<object>`` tags as ``<subdir name="..." />``.
+In JSON results they appear as dictionaries with only a ``subdir`` key. In XML results they appear interleaved with ``<object>`` tags as ``<subdir name="..." />``.
In case there is an object with the same name as a virtual directory marker, the object will be returned.
Example ``format=json`` reply:
"last_modified": "2011-12-02T08:10:41.565891+00:00",
"x_object_meta": {"asdf": "qwerty"},
"x_object_hash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
+ "x_object_uuid": "8ed9af1b-c948-4bb6-82b0-48344f5c822c",
"x_object_version": 98,
"x_object_version_timestamp": "1322813441.565891",
"x_object_modified_by": "user"}, ...]
<key>asdf</key><value>qwerty</value>
</x_object_meta>
<x_object_hash>e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855</x_object_hash>
+ <x_object_uuid>8ed9af1b-c948-4bb6-82b0-48344f5c822c</x_object_uuid>
<x_object_version>98</x_object_version>
<x_object_version_timestamp>1322813441.565891</x_object_version_timestamp>
<x_object_modified_by>chazapis</x_object_modified_by>
Content-Encoding The encoding of the object (optional)
Content-Disposition The presentation style of the object (optional)
X-Object-Hash The Merkle hash
+X-Object-UUID The object's UUID
X-Object-Version The object's version identifier
X-Object-Version-Timestamp The object's version timestamp
X-Object-Modified-By The user that comitted the object's version
The reply is the object's data (or part of it), except if a hashmap is requested with ``hashmap``, or a version list with ``version=list`` (in both cases an extended reply format must be specified). Object headers (as in a ``HEAD`` request) are always included.
-Hashmaps expose the underlying storage format of the object. Note that each hash is computed after trimming trailing null bytes of the corresponding block.
+Hashmaps expose the underlying storage format of the object. Note that each hash is computed after trimming trailing null bytes of the corresponding block. The ``X-Object-Hash`` header reports the single Merkle hash of the object's hashmap (refer to http://bittorrent.org/beps/bep_0030.html for more information).
Example ``format=json`` reply:
Content-Encoding The encoding of the object (optional)
Content-Disposition The presentation style of the object (optional)
X-Object-Hash The Merkle hash
+X-Object-UUID The object's UUID
X-Object-Version The object's version identifier
X-Object-Version-Timestamp The object's version timestamp
X-Object-Modified-By The user that comitted the object's version
X-Object-Meta-* Optional user defined metadata
========================== ===============================
-|
+Sharing headers (``X-Object-Sharing``, ``X-Object-Shared-By`` and ``X-Object-Allowed-To``) are only included if the request is for the object's latest version (no specific ``version`` parameter is set).
=========================== ==============================
Return Code Description
====================== ===================================
Request Parameter Name Value
====================== ===================================
-format Optional extended request type (can be ``json`` or ``xml``)
+format Optional extended request/conflict response type (can be ``json`` or ``xml``)
hashmap Optional hashmap provided instead of data (no value parameter)
====================== ===================================
-The request is the object's data (or part of it), except if a hashmap is provided (using ``hashmap`` and ``format`` parameters). If using a hashmap and all different parts are stored in the server, the object is created, otherwise the server returns Conflict (409) with the list of the missing parts (in a simple text format, with one hash per line).
+The request is the object's data (or part of it), except if a hashmap is provided (using ``hashmap`` and ``format`` parameters). If using a hashmap and all different parts are stored in the server, the object is created. Otherwise the server returns Conflict (409) with the list of the missing parts (in simple text format, with one hash per line, or in JSON/XML - depending on the ``format`` parameter).
Hashmaps should be formatted as outlined in ``GET``.
========================== ===============================
Reply Header Name Value
========================== ===============================
-ETag The MD5 hash of the object (on create)
+ETag The MD5 hash of the object
X-Object-Version The object's new version
========================== ===============================
Return Code Description
============================== ==============================
201 (Created) The object has been created
-409 (Conflict) The object can not be created from the provided hashmap, or there are conflicting permissions (a list of missing hashes, or a list of conflicting sharing paths will be included in the reply - in simple text format)
+409 (Conflict) The object can not be created from the provided hashmap, or there are conflicting permissions (a list of missing hashes, or a list of conflicting sharing paths will be included in the reply)
411 (Length Required) Missing ``Content-Length`` or ``Content-Type`` in the request
413 (Request Entity Too Large) Insufficient quota to complete the request
422 (Unprocessable Entity) The MD5 checksum of the data written to the storage system does not match the (optionally) supplied ETag value
If-None-Match Proceed if ETags don't match with object
Destination The destination path in the form ``/<container>/<object>``
Destination-Account The destination account to copy to
-Content-Type The MIME content type of the object (optional)
+Content-Type The MIME content type of the object (optional :sup:`*`)
Content-Encoding The encoding of the object (optional)
Content-Disposition The presentation style of the object (optional)
X-Source-Version The source version to copy from
X-Object-Meta-* Optional user defined metadata
==================== ================================
+:sup:`*` *When using django locally with the supplied web server, use the ignore_content_type parameter, or do provide a valid Content-Type, as a type of text/plain is applied by default to all requests. Client software should always state ignore_content_type, except when a Content-Type is explicitly defined by the user.*
+
+====================== ===================================
+Request Parameter Name Value
+====================== ===================================
+format Optional conflict response type (can be ``json`` or ``xml``)
+ignore_content_type Ignore the supplied Content-Type
+====================== ===================================
+
Refer to ``PUT``/``POST`` for a description of request headers. Metadata is also copied, updated with any values defined. Sharing/publishing options are not copied.
========================== ===============================
Return Code Description
============================== ==============================
201 (Created) The object has been created
-409 (Conflict) There are conflicting permissions (a list of conflicting sharing paths will be included in the reply - in simple text format)
+409 (Conflict) There are conflicting permissions (a list of conflicting sharing paths will be included in the reply)
413 (Request Entity Too Large) Insufficient quota to complete the request
============================== ==============================
====================== ============================================
Request Parameter Name Value
====================== ============================================
+format Optional conflict response type (can be ``json`` or ``xml``)
update Do not replace metadata (no value parameter)
====================== ============================================
Optionally, truncate the updated object to the desired length with the ``X-Object-Bytes`` header.
-A data update will trigger an ETag change. Updated ETags correspond to the single Merkle hash of the object's hashmap (refer to http://bittorrent.org/beps/bep_0030.html for more information).
+A data update will trigger an ETag change. Updated ETags may happen asynchronously and appear at the server with a delay.
No reply content. No reply headers if only metadata is updated.
============================== ==============================
202 (Accepted) The request has been accepted (not a data update)
204 (No Content) The request succeeded (data updated)
-409 (Conflict) There are conflicting permissions (a list of conflicting sharing paths will be included in the reply - in simple text format)
+409 (Conflict) There are conflicting permissions (a list of conflicting sharing paths will be included in the reply)
411 (Length Required) Missing ``Content-Length`` in the request
413 (Request Entity Too Large) Insufficient quota to complete the request
416 (Range Not Satisfiable) The supplied range is invalid
* Object hashmap retrieval through ``GET`` and the ``format`` parameter.
* Object create via hashmap through ``PUT`` and the ``format`` parameter.
* The object's Merkle hash is always returned in the ``X-Object-Hash`` header.
+* The object's UUID is always returned in the ``X-Object-UUID`` header. The UUID remains unchanged, even when the object's data or metadata changes, or the object is moved to another path (is renamed). A new UUID is assigned when creating or copying an object.
* Object create using ``POST`` to support standard HTML forms.
-* Partial object updates through ``POST``, using the ``Content-Length``, ``Content-Type``, ``Content-Range`` and ``Transfer-Encoding`` headers. Use another object's data to update with ``X-Source-Object`` and ``X-Source-Version``. Truncate with ``X-Object-Bytes``. New ETag corresponds to the Merkle hash of the object's hashmap.
+* Partial object updates through ``POST``, using the ``Content-Length``, ``Content-Type``, ``Content-Range`` and ``Transfer-Encoding`` headers. Use another object's data to update with ``X-Source-Object`` and ``X-Source-Version``. Truncate with ``X-Object-Bytes``.
* Include new version identifier in replies for object replace/change requests.
-* Object ``MOVE`` support.
+* Object ``MOVE`` support and ``ignore_content_type`` parameter in both ``COPY`` and ``MOVE``.
* Conditional object create/update operations, using ``If-Match`` and ``If-None-Match`` headers.
* Time-variant account/container listings via the ``until`` parameter.
* Object versions - parameter ``version`` in ``HEAD``/``GET`` (list versions with ``GET``), ``X-Object-Version-*`` meta in replies, ``X-Source-Version`` in ``PUT``/``COPY``.
Notes:
-* States represent file hashes (either MD5 or Merkle). Deleted or non-existing files are assumed to have a magic hash (e.g. empty string).
+* States represent file hashes (it is suggested to use Merkle). Deleted or non-existing files are assumed to have a magic hash (e.g. empty string).
* Updating a state (either local or remote) implies downloading, uploading or deleting the appropriate file.
Recommended Practices and Examples
projects / pithos / blobdiff