========================= ================================
Revision Description
========================= ================================
-0.6 (July 29, 2011) Reply with Merkle hash as the ETag when updating objects.
+0.6 (Sept 13, 2011) Reply with Merkle hash as the ETag when updating objects.
+\ Include version id in object replace/change replies.
+\ Change conflict (409) replies format to text.
+\ Tags should be migrated to a meta value.
+\ Container ``PUT`` updates metadata/policy.
+\ Report allowed actions in shared object replies.
+\ Provide ``https://hostname/login`` for Shibboleth authentication.
+\ Use ``hashmap`` parameter in object ``GET``/``PUT`` to use hashmaps.
0.5 (July 22, 2011) Object update from another object's data.
\ Support object truncate.
\ Create object using a standard HTML form.
0.1 (May 17, 2011) Initial release. Based on OpenStack Object Storage Developer Guide API v1 (Apr. 15, 2011).
========================= ================================
+Pithos Users and Authentication
+-------------------------------
+
+Pithos keeps separate databases for users and objects.
+
+Each user is uniquely identified by the ``Uniq`` field. This should be used as the user's account in the API. The API uses the ``Token`` field to authenticate a user, thus allowing cross-account requests. All API requests require a token.
+
+User entries can be modified/added via the management interface available at ``https://hostname/admin``.
+
+Pithos is also compatible with Shibboleth (http://shibboleth.internet2.edu/). The connection between Shibboleth and Pithos is done by ``https://hostname/login``. An application that wishes to connect to Pithos, but does not have a token, should redirect the user to the login URI.
+
+The login URI accepts the following parameters:
+
+====================== =========================
+Request Parameter Name Value
+====================== =========================
+next The URI to redirect to when the process is finished
+renew Force token renewal (no value parameter)
+====================== =========================
+
+The login process starts by redirecting the user to an external URI (controlled by Shibboleth), where the actual authentication credentials are entered. Then, the user is redirected back to the login URI from Shibboleth, with various identification information in the request headers.
+
+If the user does not exist in the database, Pithos adds the user and creates a random token. If the user exists, the token has not expired and ``renew`` is not set, the existing token is reused. Finally, the login URI redirects to the URI provided with ``next``, adding the ``user`` and ``token`` parameters, which contain the ``Uniq`` and ``Token`` fields respectively.
+
The Pithos API
--------------
* Container level: ``https://hostname/v1/<account>/<container>``
* Object level: ``https://hostname/v1/<account>/<container>/<object>``
-All requests must include an ``X-Auth-Token`` - as a header, or a parameter. The process of obtaining the token is still to be determined (**TBD**).
+All requests must include an ``X-Auth-Token`` - as a header, or a parameter.
The allowable request operations and respective return codes per level are presented in the remainder of this chapter. Common to all requests are the following return codes.
X-Container-Block-Size The block size used by the storage backend
X-Container-Block-Hash The hash algorithm used for block identifiers in object hashmaps
X-Container-Until-Timestamp The last container modification date until the timestamp provided
-X-Container-Object-Meta A list with all meta keys used by objects
+X-Container-Object-Meta A list with all meta keys used by objects (**TBD**)
X-Container-Policy-* Container behavior and limits
X-Container-Meta-* Optional user defined metadata
Last-Modified The last container modification date (regardless of ``until``)
=========================== ===============================
-The keys returned in ``X-Container-Object-Meta`` are all the unique strings after the ``X-Object-Meta-`` prefix, formatted as a comma-separated list. See container ``PUT`` for a reference of policy directives.
+The keys returned in ``X-Container-Object-Meta`` are all the unique strings after the ``X-Object-Meta-`` prefix, formatted as a comma-separated list. See container ``PUT`` for a reference of policy directives. (**TBD**)
================ ===============================
Return Code Description
* ``versioning``: Set to ``auto``, ``manual`` or ``none`` (default is ``manual``)
* ``quota``: Size limit in KB (default is ``0`` - unlimited)
-
+
+If the container already exists, the operation is equal to a ``POST`` with ``update`` defined.
+
================ ===============================
Return Code Description
================ ===============================
X-Object-Manifest Object parts prefix in ``<container>/<object>`` form (optional)
X-Object-Sharing Object permissions (optional)
X-Object-Shared-By Object inheriting permissions (optional)
+X-Object-Allowed-To Allowed actions on object (optional)
X-Object-Public Object's publicly accessible URI (optional)
X-Object-Meta-* Optional user defined metadata
========================== ===============================
Request Parameter Name Value
====================== ===================================
format Optional extended reply type (can be ``json`` or ``xml``)
+hashmap Optional request for hashmap (no value parameter)
version Optional version identifier or ``list`` (specify a format if requesting a list)
====================== ===================================
-The reply is the object's data (or part of it), except if a hashmap is requested with the ``format`` parameter, or a version list with ``version=list`` (in which case an extended reply format must be specified). Object headers (as in a ``HEAD`` request) are always included.
+The reply is the object's data (or part of it), except if a hashmap is requested with ``hashmap``, or a version list with ``version=list`` (in both cases an extended reply format must be specified). Object headers (as in a ``HEAD`` request) are always included.
Hashmaps expose the underlying storage format of the object. Note that each hash is computed after trimming trailing null bytes of the corresponding block.
X-Object-Manifest Object parts prefix in ``<container>/<object>`` form (optional)
X-Object-Sharing Object permissions (optional)
X-Object-Shared-By Object inheriting permissions (optional)
+X-Object-Allowed-To Allowed actions on object (optional)
X-Object-Public Object's publicly accessible URI (optional)
X-Object-Meta-* Optional user defined metadata
========================== ===============================
====================== ===================================
Request Parameter Name Value
====================== ===================================
-format Optional extended request type (can be ``json``) to create the object by suppling its hashmap instead
+format Optional extended request type (can be ``json`` or ``xml``)
+hashmap Optional hashmap provided instead of data (no value parameter)
====================== ===================================
-The request is the object's data (or part of it), except if a hashmap is provided with the ``format`` parameter. If format is used and all different parts are stored in the server, the object is created, otherwise the server returns Conflict (409) with the list of the missing parts.
-
-Hashmaps expose the underlying storage format of the object.
+The request is the object's data (or part of it), except if a hashmap is provided (using ``hashmap`` and ``format`` parameters). If using a hashmap and all different parts are stored in the server, the object is created, otherwise the server returns Conflict (409) with the list of the missing parts (in a simple text format, with one hash per line).
-Example ``format=json`` request:
-
-::
-
- {"block_hash": "sha1", "hashes": ["7295c41da03d7f916440b98e32c4a2a39351546c", ...], "block_size": 131072, "bytes": 242}
-
-Example ``format=xml`` request:
-
-::
-
- <?xml version="1.0" encoding="UTF-8"?>
- <object name="file" bytes="24223726" block_size="131072" block_hash="sha1">
- <hash>7295c41da03d7f916440b98e32c4a2a39351546c</hash>
- <hash>...</hash>
- </object>
+Hashmaps should be formatted as outlined in ``GET``.
========================== ===============================
Reply Header Name Value
========================== ===============================
ETag The MD5 hash of the object (on create)
+X-Object-Version The object's new version
========================== ===============================
The ``X-Object-Sharing`` header may include either a ``read=...`` comma-separated user/group list, or a ``write=...`` comma-separated user/group list, or both separated by a semicolon (``;``). Groups are specified as ``<account>:<group>``. To publish the object, set ``X-Object-Public`` to ``true``. To unpublish, set to ``false``, or use an empty header value.
Return Code Description
=========================== ==============================
201 (Created) The object has been created
-409 (Conflict) The object can not be created from the provided hashmap, or there are conflicting permissions (a list of missing hashes, or a conflicting sharing path will be included in the reply - in JSON format)
+409 (Conflict) The object can not be created from the provided hashmap, or there are conflicting permissions (a list of missing hashes, or a list of conflicting sharing paths will be included in the reply - in simple text format)
411 (Length Required) Missing ``Content-Length`` or ``Content-Type`` in the request
422 (Unprocessable Entity) The MD5 checksum of the data written to the storage system does not match the (optionally) supplied ETag value
=========================== ==============================
Refer to ``PUT``/``POST`` for a description of request headers. Metadata is also copied, updated with any values defined. Sharing/publishing options are not copied.
-No reply content/headers.
+========================== ===============================
+Reply Header Name Value
+========================== ===============================
+X-Object-Version The object's new version
+========================== ===============================
+
+|
=========================== ==============================
Return Code Description
=========================== ==============================
201 (Created) The object has been created
-409 (Conflict) There are conflicting permissions (a conflicting sharing path will be included in the reply - in JSON format)
+409 (Conflict) There are conflicting permissions (a list of conflicting sharing paths will be included in the reply - in simple text format)
=========================== ==============================
Reply Header Name Value
========================== ===============================
ETag The new ETag of the object (data updated)
+X-Object-Version The object's new version
========================== ===============================
|
=========================== ==============================
202 (Accepted) The request has been accepted (not a data update)
204 (No Content) The request succeeded (data updated)
-409 (Conflict) There are conflicting permissions (a conflicting sharing path will be included in the reply - in JSON format)
+409 (Conflict) There are conflicting permissions (a list of conflicting sharing paths will be included in the reply - in simple text format)
411 (Length Required) Missing ``Content-Length`` in the request
416 (Range Not Satisfiable) The supplied range is invalid
=========================== ==============================
Reply Header Name Value
========================== ===============================
ETag The MD5 hash of the object
+X-Object-Version The object's new version
========================== ===============================
|
Read and write control in Pithos is managed by setting appropriate permissions with the ``X-Object-Sharing`` header. The permissions are applied using prefix-based inheritance. Thus, each set of authorization directives is applied to all objects sharing the same prefix with the object where the corresponding ``X-Object-Sharing`` header is defined. For simplicity, nested/overlapping permissions are not allowed. Setting ``X-Object-Sharing`` will fail, if the object is already "covered", or another object with a longer common-prefix name already has permissions. When retrieving an object, the ``X-Object-Shared-By`` header reports where it gets its permissions from. If not present, the object is the actual source of authorization directives.
-A user may ``GET`` another account or container. The result will include a limited reply, containing only the allowed containers or objects respectively. A top-level request with an authentication token, will return a list of allowed accounts, so the user can easily find out which other users share objects.
+A user may ``GET`` another account or container. The result will include a limited reply, containing only the allowed containers or objects respectively. A top-level request with an authentication token, will return a list of allowed accounts, so the user can easily find out which other users share objects. The ``X-Object-Allowed-To`` header lists the actions allowed on an object, if it does not belong to the requesting user.
Objects that are marked as public, via the ``X-Object-Public`` meta, are also available at the corresponding URI returned for ``HEAD`` or ``GET``. Requests for public objects do not need to include an ``X-Auth-Token``. Pithos will ignore request parameters and only include the following headers in the reply (all ``X-Object-*`` meta is hidden):
* Support for ``X-Account-Meta-*`` style headers at the account level. Use ``POST`` to update.
* Support for ``X-Container-Meta-*`` style headers at the container level. Can be set when creating via ``PUT``. Use ``POST`` to update.
-* Header ``X-Container-Object-Meta`` at the container level and parameter ``meta`` in container listings.
+* Header ``X-Container-Object-Meta`` at the container level and parameter ``meta`` in container listings. (**TBD**)
* Container policies to manage behavior and limits.
* Headers ``X-Container-Block-*`` at the container level, exposing the underlying storage characteristics.
* All metadata replies, at all levels, include latest modification information.
* Object create via hashmap through ``PUT`` and the ``format`` parameter.
* Object create using ``POST`` to support standard HTML forms.
* Partial object updates through ``POST``, using the ``Content-Length``, ``Content-Type``, ``Content-Range`` and ``Transfer-Encoding`` headers. Use another object's data to update with ``X-Source-Object`` and ``X-Source-Version``. Truncate with ``X-Object-Bytes``. New ETag corresponds to the Merkle hash of the object's hashmap.
+* Include new version identifier in replies for object replace/change requests.
* Object ``MOVE`` support.
* Conditional object create/update operations, using ``If-Match`` and ``If-None-Match`` headers.
* Time-variant account/container listings via the ``until`` parameter.
* Object versions - parameter ``version`` in ``HEAD``/``GET`` (list versions with ``GET``), ``X-Object-Version-*`` meta in replies, ``X-Source-Version`` in ``PUT``/``COPY``.
-* Sharing/publishing with ``X-Object-Sharing``, ``X-Object-Public`` at the object level. Cross-user operations are allowed - controlled by sharing directives. Permissions may include groups defined with ``X-Account-Group-*`` at the account level. These apply to the object - not its versions.
+* Sharing/publishing with ``X-Object-Sharing``, ``X-Object-Public`` at the object level. Cross-user operations are allowed - controlled by sharing directives. Available actions in cross-user requests are reported with ``X-Object-Allowed-To``. Permissions may include groups defined with ``X-Account-Group-*`` at the account level. These apply to the object - not its versions.
* Support for prefix-based inheritance when enforcing permissions. Parent object carrying the authorization directives is reported in ``X-Object-Shared-By``.
* Large object support with ``X-Object-Manifest``.
* Trace the user that created/modified an object with ``X-Object-Modified-By``.
* Some processing is done in the variable part of all ``X-*-Meta-*`` headers. If it includes underscores, they will be converted to dashes and the first letter of all intra-dash strings will be capitalized.
* A ``GET`` reply for a level will include all headers of the corresponding ``HEAD`` request.
* To avoid conflicts between objects and virtual directory markers in container listings, it is recommended that object names do not end with the delimiter used.
-* The ``Accept`` header may be used in requests instead of the ``format`` parameter to specify the desired reply format. The parameter overrides the header (**TBD**).
+* The ``Accept`` header may be used in requests instead of the ``format`` parameter to specify the desired request/reply format. The parameter overrides the header.
* Container/object lists use a ``200`` return code if the reply is of type json/xml. The reply will include an empty json/xml.
* In headers, dates are formatted according to RFC 1123. In extended information listings, the ``last_modified`` field is formatted according to ISO 8601 (for OOS API compatibility). All other fields (Pithos extensions) use integer tiemstamps.
* The ``Last-Modified`` header value always reflects the actual latest change timestamp, regardless of time control parameters and version requests. Time precondition checks with ``If-Modified-Since`` and ``If-Unmodified-Since`` headers are applied to this value.
* The ``trash`` element, which contains files that have been marked for deletion, but can still be recovered.
* The ``shared`` element, which contains all objects shared by the user to other users of the system.
* The ``others`` element, which contains all objects that other users share with the user.
-* The ``tags`` element, which lists the names of tags the user has defined. This can be an entry point to list all files that have been assigned a specific tag or manage tags in general (remove a tag completely, rename a tag etc.).
* The ``groups`` element, which contains the names of groups the user has defined. Each group consists of a user list. Group creation, deletion, and manipulation is carried out by actions originating here.
* The ``history`` element, which allows browsing past instances of ``home`` and - optionally - ``trash``.
Objects in Pithos can be:
-* Assigned custom tags.
* Moved to trash and then deleted.
* Shared with specific permissions.
* Made public (shared with non-Pithos users).
Some of these functions are performed by the client software and some by the Pithos server.
+In the first version of Pithos, objects could also be assigned custom tags. This is no longer supported. Existing deployments can migrate tags into a specific metadata value, i.e. ``X-Object-Meta-Tags``.
+
Implementation Guidelines
^^^^^^^^^^^^^^^^^^^^^^^^^
Public objects are not included in ``shared`` and ``others`` listings. It is suggested that they are marked in a visually distinctive way in ``pithos`` listings (for example using an icon overlay).
-At the object level, tags are implemented by managing metadata keys. The client software should allow the user to use any string as a tag and set the corresponding ``X-Object-Meta-<tag>`` key at the server. The API extensions provided, allow for listing all tags in a container and filtering object listings based on one or more tags. The tag list is sufficient for implementing the ``tags`` element, either as a special, virtual folder (as done in the first version of Pithos), or as an application menu.
-
A special application menu, or a section in application preferences, should be devoted to managing groups (the ``groups`` element). All group-related actions are implemented at the account level.
-Browsing past versions of objects should be available both at the object and the container level. At the object level, a list of past versions can be included in the screen showing details or more information on the object (metadata, tags, permissions, etc.). At the container level, it is suggested that clients use a ``history`` element, which presents to the user a read-only, time-variable view of ``pithos`` contents. This can be accomplished via the ``until`` parameter in listings. Optionally, ``history`` may include ``trash``.
+Browsing past versions of objects should be available both at the object and the container level. At the object level, a list of past versions can be included in the screen showing details or more information on the object (metadata, permissions, etc.). At the container level, it is suggested that clients use a ``history`` element, which presents to the user a read-only, time-variable view of ``pithos`` contents. This can be accomplished via the ``until`` parameter in listings. Optionally, ``history`` may include ``trash``.
Recommended Practices and Examples
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Assuming an authentication token is obtained (**TBD**), the following high-level operations are available - shown with ``curl``:
+Assuming an authentication token is obtained, the following high-level operations are available - shown with ``curl``:
* Get account information ::
* List metadata keys used by objects in a container
- Will be in the ``X-Container-Object-Meta`` reply header, included in container information or object list (``HEAD`` or ``GET``).
+ Will be in the ``X-Container-Object-Meta`` reply header, included in container information or object list (``HEAD`` or ``GET``). (**TBD**)
* List objects in a container having a specific meta defined ::
projects / pithos / blobdiff