========================= ================================
Revision Description
========================= ================================
-0.7 (Sept 29, 2011) Suggest upload/download methods using hashmaps.
+0.8 (Dec 2, 2011) Update allowed versioning values.
+\ Change policy/meta formatting in JSON/XML replies.
+\ Document that all non-ASCII characters in headers should be URL-encoded.
+0.7 (Nov 21, 2011) Suggest upload/download methods using hashmaps.
\ Propose syncing algorithm.
\ Support cross-account object copy and move.
\ Pass token as a request parameter when using ``POST`` via an HTML form.
+\ Optionally use source account to update object from another object.
+\ Use container ``POST`` to upload missing blocks of data.
+\ Report policy in account headers.
+\ Add insufficient quota reply.
+\ Use special meta to always report Merkle hash.
0.6 (Sept 13, 2011) Reply with Merkle hash as the ETag when updating objects.
\ Include version id in object replace/change replies.
\ Change conflict (409) replies format to text.
Return Code Description
========================= ================================
400 (Bad Request) The request is invalid
-401 (Unauthorized) Request not allowed
+401 (Unauthorized) Missing or invalid token
+403 (Forbidden) Request not allowed
404 (Not Found) The requested resource was not found
503 (Service Unavailable) The request cannot be completed because of an internal error
========================= ================================
::
- [{"name": "user", "last_modified": "2011-07-19T10:48:16"}, ...]
+ [{"name": "user", "last_modified": "2011-12-02T08:10:41.565891+00:00"}, ...]
Example ``format=xml`` reply:
<accounts>
<account>
<name>user</name>
- <last_modified>2011-07-19T10:48:16</last_modified>
+ <last_modified>2011-12-02T08:10:41.565891+00:00</last_modified>
</account>
<account>...</account>
</accounts>
204 (No Content) The user has no access to other accounts (only for non-extended replies)
=========================== =====================
-Will use a ``200`` return code if the reply is of type json/xml.
+Will use a ``200`` return code if the reply is of type JSON/XML.
Account Level
^^^^^^^^^^^^^
Reply Header Name Value
========================== =====================
X-Account-Container-Count The total number of containers
-X-Account-Object-Count The total number of objects (**TBD**)
X-Account-Bytes-Used The total number of bytes stored
-X-Account-Bytes-Remaining The total number of bytes remaining (**TBD**)
-X-Account-Last-Login The last login (**TBD**)
X-Account-Until-Timestamp The last account modification date until the timestamp provided
X-Account-Group-* Optional user defined groups
+X-Account-Policy-* Account behavior and limits
X-Account-Meta-* Optional user defined metadata
Last-Modified The last account modification date (regardless of ``until``)
========================== =====================
x_container_meta_* Optional user defined metadata
=========================== ============================
-For examples of container details returned in JSON/XML formats refer to the OOS API documentation.
+Example ``format=json`` reply:
+
+::
+
+ [{"name": "pithos",
+ "bytes": 62452,
+ "count": 8374,
+ "last_modified": "2011-12-02T08:10:41.565891+00:00",
+ "x_container_policy": {"quota": "53687091200", "versioning": "auto"},
+ "x_container_meta": {"a": "b", "1": "2"}}, ...]
+
+Example ``format=xml`` reply:
+
+::
+
+ <?xml version="1.0" encoding="UTF-8"?>
+ <account name="user">
+ <container>
+ <name>pithos</name>
+ <bytes>62452</bytes>
+ <count>8374</count>
+ <last_modified>2011-12-02T08:10:41.565891+00:00</last_modified>
+ <x_container_policy>
+ <key>quota</key><value>53687091200</value>
+ <key>versioning</key><value>auto</value>
+ </x_container_policy>
+ <x_container_meta>
+ <key>a</key><value>b</value>
+ <key>1</key><value>2</value>
+ </x_container_meta>
+ </container>
+ <container>...</container>
+ </account>
+
+For more examples of container details returned in JSON/XML formats refer to the OOS API documentation. In addition to the OOS API, Pithos returns all fields. Policy and metadata values are grouped and returned as key-value pairs.
=========================== =====================
Return Code Description
412 (Precondition Failed) The condition set can not be satisfied
=========================== =====================
-Will use a ``200`` return code if the reply is of type json/xml.
+Will use a ``200`` return code if the reply is of type JSON/XML.
POST
content_encoding The encoding of the object (optional)
content-disposition The presentation style of the object (optional)
last_modified The last object modification date (regardless of version)
+x_object_hash The Merkle hash
x_object_version The object's version identifier
x_object_version_timestamp The object's version timestamp
x_object_modified_by The user that committed the object's version
In JSON results they appear as dictionaries with only a ``"subdir"`` key. In XML results they appear interleaved with ``<object>`` tags as ``<subdir name="..." />``.
In case there is an object with the same name as a virtual directory marker, the object will be returned.
-For examples of object details returned in JSON/XML formats refer to the OOS API documentation.
+Example ``format=json`` reply:
+
+::
+
+ [{"name": "object",
+ "bytes": 0,
+ "hash": "d41d8cd98f00b204e9800998ecf8427e",
+ "content_type": "application/octet-stream",
+ "last_modified": "2011-12-02T08:10:41.565891+00:00",
+ "x_object_meta": {"asdf": "qwerty"},
+ "x_object_hash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
+ "x_object_version": 98,
+ "x_object_version_timestamp": "1322813441.565891",
+ "x_object_modified_by": "user"}, ...]
+
+Example ``format=xml`` reply:
+
+::
+
+ <?xml version="1.0" encoding="UTF-8"?>
+ <container name="pithos">
+ <object>
+ <name>object</name>
+ <bytes>0</bytes>
+ <hash>d41d8cd98f00b204e9800998ecf8427e</hash>
+ <content_type>application/octet-stream</content_type>
+ <last_modified>2011-12-02T08:10:41.565891+00:00</last_modified>
+ <x_object_meta>
+ <key>asdf</key><value>qwerty</value>
+ </x_object_meta>
+ <x_object_hash>e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855</x_object_hash>
+ <x_object_version>98</x_object_version>
+ <x_object_version_timestamp>1322813441.565891</x_object_version_timestamp>
+ <x_object_modified_by>chazapis</x_object_modified_by>
+ </object>
+ <object>...</object>
+ </container>
+
+For more examples of container details returned in JSON/XML formats refer to the OOS API documentation. In addition to the OOS API, Pithos returns all fields. Metadata values are grouped and returned as key-value pairs.
=========================== ===============================
Return Code Description
412 (Precondition Failed) The condition set can not be satisfied
=========================== ===============================
-Will use a ``200`` return code if the reply is of type json/xml.
+Will use a ``200`` return code if the reply is of type JSON/XML.
PUT
If no policy is defined, the container will be created with the default values.
Available policy directives:
-* ``versioning``: Set to ``auto``, ``manual`` or ``none`` (default is ``manual``)
+* ``versioning``: Set to ``auto`` or ``none`` (default is ``auto``)
* ``quota``: Size limit in KB (default is ``0`` - unlimited)
If the container already exists, the operation is equal to a ``POST`` with ``update`` defined.
==================== ================================
Request Header Name Value
==================== ================================
+Content-Length The size of the supplied data (optional, to upload)
+Content-Type The MIME content type of the supplied data (optional, to upload)
+Transfer-Encoding Set to ``chunked`` to specify incremental uploading (if used, ``Content-Length`` is ignored)
X-Container-Policy-* Container behavior and limits
X-Container-Meta-* Optional user defined metadata
==================== ================================
update Do not replace metadata/policy (no value parameter)
====================== ============================================
-No reply content/headers.
+No reply content/headers, except when uploading data, where the reply consists of a list of hashes for the blocks received (in a simple text format, with one hash per line).
The operation will overwrite all user defined metadata, except if ``update`` is defined.
To change policy, include an ``X-Container-Policy-*`` header with the name in the key. If no ``X-Container-Policy-*`` header is present, no changes will be applied to policy. The ``update`` parameter also applies to policy - deleted values will revert to defaults. To delete/revert a specific policy directive, use ``update`` and an empty header value. See container ``PUT`` for a reference of policy directives.
+To upload blocks of data to the container, set ``Content-Type`` to ``application/octet-stream`` and ``Content-Length`` to a valid value (except if using ``chunked`` as the ``Transfer-Encoding``).
+
================ ===============================
Return Code Description
================ ===============================
Last-Modified The last object modification date (regardless of version)
Content-Encoding The encoding of the object (optional)
Content-Disposition The presentation style of the object (optional)
+X-Object-Hash The Merkle hash
X-Object-Version The object's version identifier
X-Object-Version-Timestamp The object's version timestamp
X-Object-Modified-By The user that comitted the object's version
::
- {"versions": [[23, 1307700892], [28, 1307700898], ...]}
+ {"versions": [[85, "1322734861.248469"], [86, "1322734905.009272"], ...]}
Example ``format=xml`` reply:
<?xml version="1.0" encoding="UTF-8"?>
<object name="file">
- <version timestamp="1307700892">23</version>
- <version timestamp="1307700898">28</version>
+ <version timestamp="1322734861.248469">85</version>
+ <version timestamp="1322734905.009272">86</version>
<version timestamp="...">...</version>
</object>
Last-Modified The last object modification date (regardless of version)
Content-Encoding The encoding of the object (optional)
Content-Disposition The presentation style of the object (optional)
+X-Object-Hash The Merkle hash
X-Object-Version The object's version identifier
X-Object-Version-Timestamp The object's version timestamp
X-Object-Modified-By The user that comitted the object's version
The ``X-Object-Sharing`` header may include either a ``read=...`` comma-separated user/group list, or a ``write=...`` comma-separated user/group list, or both separated by a semicolon (``;``). Groups are specified as ``<account>:<group>``. To publish the object, set ``X-Object-Public`` to ``true``. To unpublish, set to ``false``, or use an empty header value.
-=========================== ==============================
-Return Code Description
-=========================== ==============================
-201 (Created) The object has been created
-409 (Conflict) The object can not be created from the provided hashmap, or there are conflicting permissions (a list of missing hashes, or a list of conflicting sharing paths will be included in the reply - in simple text format)
-411 (Length Required) Missing ``Content-Length`` or ``Content-Type`` in the request
-422 (Unprocessable Entity) The MD5 checksum of the data written to the storage system does not match the (optionally) supplied ETag value
-=========================== ==============================
+============================== ==============================
+Return Code Description
+============================== ==============================
+201 (Created) The object has been created
+409 (Conflict) The object can not be created from the provided hashmap, or there are conflicting permissions (a list of missing hashes, or a list of conflicting sharing paths will be included in the reply - in simple text format)
+411 (Length Required) Missing ``Content-Length`` or ``Content-Type`` in the request
+413 (Request Entity Too Large) Insufficient quota to complete the request
+422 (Unprocessable Entity) The MD5 checksum of the data written to the storage system does not match the (optionally) supplied ETag value
+============================== ==============================
COPY
|
-=========================== ==============================
-Return Code Description
-=========================== ==============================
-201 (Created) The object has been created
-409 (Conflict) There are conflicting permissions (a list of conflicting sharing paths will be included in the reply - in simple text format)
-=========================== ==============================
+============================== ==============================
+Return Code Description
+============================== ==============================
+201 (Created) The object has been created
+409 (Conflict) There are conflicting permissions (a list of conflicting sharing paths will be included in the reply - in simple text format)
+413 (Request Entity Too Large) Insufficient quota to complete the request
+============================== ==============================
MOVE
Content-Encoding The encoding of the object (optional)
Content-Disposition The presentation style of the object (optional)
X-Source-Object Update with data from the object at path ``/<container>/<object>`` (optional, to update)
+X-Source-Account The source account to update from
X-Source-Version The source version to update from (optional, to update)
X-Object-Bytes The updated object's final size (optional, when updating)
X-Object-Manifest Object parts prefix in ``<container>/<object>`` form (optional)
|
-=========================== ==============================
-Return Code Description
-=========================== ==============================
-202 (Accepted) The request has been accepted (not a data update)
-204 (No Content) The request succeeded (data updated)
-409 (Conflict) There are conflicting permissions (a list of conflicting sharing paths will be included in the reply - in simple text format)
-411 (Length Required) Missing ``Content-Length`` in the request
-416 (Range Not Satisfiable) The supplied range is invalid
-=========================== ==============================
+============================== ==============================
+Return Code Description
+============================== ==============================
+202 (Accepted) The request has been accepted (not a data update)
+204 (No Content) The request succeeded (data updated)
+409 (Conflict) There are conflicting permissions (a list of conflicting sharing paths will be included in the reply - in simple text format)
+411 (Length Required) Missing ``Content-Length`` in the request
+413 (Request Entity Too Large) Insufficient quota to complete the request
+416 (Range Not Satisfiable) The supplied range is invalid
+============================== ==============================
The ``POST`` method can also be used for creating an object via a standard HTML form. If the request ``Content-Type`` is ``multipart/form-data``, none of the above headers will be processed. The form should have an ``X-Object-Data`` field, as in the following example. The token is passed as a request parameter. ::
|
-=========================== ==============================
-Return Code Description
-=========================== ==============================
-201 (Created) The object has been created
-=========================== ==============================
+============================== ==============================
+Return Code Description
+============================== ==============================
+201 (Created) The object has been created
+413 (Request Entity Too Large) Insufficient quota to complete the request
+============================== ==============================
DELETE
* Support for ``X-Account-Meta-*`` style headers at the account level. Use ``POST`` to update.
* Support for ``X-Container-Meta-*`` style headers at the container level. Can be set when creating via ``PUT``. Use ``POST`` to update.
* Header ``X-Container-Object-Meta`` at the container level and parameter ``meta`` in container listings. (**TBD**)
-* Container policies to manage behavior and limits.
+* Account and container policies to manage behavior and limits. Container behavior overrides account settings. Account quota sets the maximum bytes limit, regardless of container values.
* Headers ``X-Container-Block-*`` at the container level, exposing the underlying storage characteristics.
* All metadata replies, at all levels, include latest modification information.
* At all levels, a ``HEAD`` or ``GET`` request may use ``If-Modified-Since`` and ``If-Unmodified-Since`` headers.
-* Container/object lists include all associated metadata if the reply is of type json/xml. Some names are kept to their OOS API equivalents for compatibility.
+* Container/object lists include all associated metadata if the reply is of type JSON/XML. Some names are kept to their OOS API equivalents for compatibility.
* Option to include only shared containers/objects in listings.
* Object metadata allowed, in addition to ``X-Object-Meta-*``: ``Content-Encoding``, ``Content-Disposition``, ``X-Object-Manifest``. These are all replaced with every update operation, except if using the ``update`` parameter (in which case individual keys can also be deleted). Deleting meta by providing empty values also works when copying/moving an object.
* Multi-range object ``GET`` support as outlined in RFC2616.
* Object hashmap retrieval through ``GET`` and the ``format`` parameter.
* Object create via hashmap through ``PUT`` and the ``format`` parameter.
+* The object's Merkle hash is always returned in the ``X-Object-Hash`` header.
* Object create using ``POST`` to support standard HTML forms.
* Partial object updates through ``POST``, using the ``Content-Length``, ``Content-Type``, ``Content-Range`` and ``Transfer-Encoding`` headers. Use another object's data to update with ``X-Source-Object`` and ``X-Source-Version``. Truncate with ``X-Object-Bytes``. New ETag corresponds to the Merkle hash of the object's hashmap.
* Include new version identifier in replies for object replace/change requests.
Clarifications/suggestions:
+* All non-ASCII characters in headers should be URL-encoded.
* Authentication is done by another system. The token is used in the same way, but it is obtained differently. The top level ``GET`` request is kept compatible with the OOS API and allows for guest/testing operations.
* Some processing is done in the variable part of all ``X-*-Meta-*`` headers. If it includes underscores, they will be converted to dashes and the first letter of all intra-dash strings will be capitalized.
* A ``GET`` reply for a level will include all headers of the corresponding ``HEAD`` request.
* To avoid conflicts between objects and virtual directory markers in container listings, it is recommended that object names do not end with the delimiter used.
* The ``Accept`` header may be used in requests instead of the ``format`` parameter to specify the desired request/reply format. The parameter overrides the header.
-* Container/object lists use a ``200`` return code if the reply is of type json/xml. The reply will include an empty json/xml.
+* Container/object lists use a ``200`` return code if the reply is of type JSON/XML. The reply will include an empty JSON/XML.
* In headers, dates are formatted according to RFC 1123. In extended information listings, the ``last_modified`` field is formatted according to ISO 8601 (for OOS API compatibility). All other fields (Pithos extensions) use integer tiemstamps.
* The ``Last-Modified`` header value always reflects the actual latest change timestamp, regardless of time control parameters and version requests. Time precondition checks with ``If-Modified-Since`` and ``If-Unmodified-Since`` headers are applied to this value.
* A copy/move using ``PUT``/``COPY``/``MOVE`` will always update metadata, keeping all old values except the ones redefined in the request headers.
* Server responds with status ``409`` (Conflict):
* Server's response body contains the hashes of the blocks that do not exist on the server.
- * For each one of the hash values in the server's response:
+ * For each hash value in the server's response (or all hashes together):
- * Send a ``PUT`` request to the server with the corresponding data block. Individual blocks are uploaded to a file named ``.upload``.
+ * Send a ``POST`` request to the destination container with the corresponding data.
* Repeat hashmap ``PUT``. Fail if the server's response is not ``201``.
curl -X PUT -D - \
-H "X-Auth-Token: 0000" \
- -H "Content-Type: application/folder" \
+ -H "Content-Type: application/directory" \
https://pithos.dev.grnet.gr/v1/user/pithos/folder
* Add a new object ::
projects / pithos / blobdiff