Statistics
| Branch: | Tag: | Revision:

root / docs / pithos-api-guide.rst @ 133e3fcf

History | View | Annotate | Download (69.6 kB)

1
Pithos API
2
==========
3

    
4
Introduction
5
------------
6

    
7
Pithos is a storage service implemented by GRNET (http://www.grnet.gr). Data is stored as objects, organized in containers, belonging to an account. This hierarchy of storage layers has been inspired by the OpenStack Object Storage (OOS) API and similar CloudFiles API by Rackspace. The Pithos API follows the OOS API as closely as possible. One of the design requirements has been to be able to use Pithos with clients built for the OOS, without changes.
8

    
9
However, to be able to take full advantage of the Pithos infrastructure, client software should be aware of the extensions that differentiate Pithos from OOS. Pithos objects can be updated, or appended to. Pithos will store sharing permissions per object and enforce corresponding authorization policies. Automatic version management, allows taking account and container listings back in time, as well as reading previous instances of objects.
10

    
11
The storage backend of Pithos is block oriented, permitting efficient, deduplicated data placement. The block structure of objects is exposed at the API layer, in order to encourage external software to implement advanced data management operations.
12

    
13
This document's goals are:
14

    
15
* Define the Pithos ReST API that allows the storage and retrieval of data and metadata via HTTP calls
16
* Specify metadata semantics and user interface guidelines for a common experience across client software implementations
17

    
18
The present document is meant to be read alongside the OOS API documentation. Thus, it is suggested that the reader is familiar with associated technologies, the OOS API as well as the first version of the Pithos API. This document refers to the second version of Pithos. Information on the first version of the storage API can be found at http://code.google.com/p/gss.
19

    
20
Whatever marked as to be determined (**TBD**), should not be considered by implementors.
21

    
22
More info about Pithos can be found here: https://code.grnet.gr/projects/pithos
23

    
24
Document Revisions
25
^^^^^^^^^^^^^^^^^^
26

    
27
=========================  ================================
28
Revision                   Description
29
=========================  ================================
30
0.14 (Jun 18, 2013)        Forbidden response for public listing by non path owners
31
0.14 (Apr 23, 2013)        Reply with Merkle hash in the ETag if MD5 is not computed.
32
0.13 (Mar 27, 2013)        Restrict public object listing only to the owner.
33
\                          Do not propagate public URL information in shared objects.
34
0.13 (Jan 21, 2013)        Proxy identity management services
35
\                          UUID to displayname translation
36
0.10 (Jul 18, 2012)        Support for bulk COPY/MOVE/DELETE
37
\                          Optionally include public objects in listings.
38
0.9 (Feb 17, 2012)         Change permissions model.
39
\                          Do not include user-defined metadata in account/container/object listings.
40
0.8 (Jan 24, 2012)         Update allowed versioning values.
41
\                          Change policy/meta formatting in JSON/XML replies.
42
\                          Document that all non-ASCII characters in headers should be URL-encoded.
43
\                          Support metadata-based queries when listing objects at the container level.
44
\                          Note Content-Type issue when using the internal django web server.
45
\                          Add object UUID field.
46
\                          Always reply with the MD5 in the ETag.
47
\                          Note that ``/login`` will only work if an external authentication system is defined.
48
\                          Include option to ignore Content-Type on ``COPY``/``MOVE``.
49
\                          Use format parameter for conflict (409) and uploaded hash list (container level) replies.
50
0.7 (Nov 21, 2011)         Suggest upload/download methods using hashmaps.
51
\                          Propose syncing algorithm.
52
\                          Support cross-account object copy and move.
53
\                          Pass token as a request parameter when using ``POST`` via an HTML form.
54
\                          Optionally use source account to update object from another object.
55
\                          Use container ``POST`` to upload missing blocks of data.
56
\                          Report policy in account headers.
57
\                          Add insufficient quota reply.
58
\                          Use special meta to always report Merkle hash.
59
0.6 (Sept 13, 2011)        Reply with Merkle hash as the ETag when updating objects.
60
\                          Include version id in object replace/change replies.
61
\                          Change conflict (409) replies format to text.
62
\                          Tags should be migrated to a meta value.
63
\                          Container ``PUT`` updates metadata/policy.
64
\                          Report allowed actions in shared object replies.
65
\                          Provide ``https://hostname/login`` for Shibboleth authentication.
66
\                          Use ``hashmap`` parameter in object ``GET``/``PUT`` to use hashmaps.
67
0.5 (July 22, 2011)        Object update from another object's data.
68
\                          Support object truncate.
69
\                          Create object using a standard HTML form.
70
\                          Purge container/object history.
71
\                          List other accounts that share objects with a user.
72
\                          List shared containers/objects.
73
\                          Update implementation guidelines.
74
\                          Check preconditions when creating/updating objects.
75
0.4 (July 01, 2011)        Object permissions and account groups.
76
\                          Control versioning behavior and container quotas with container policy directives.
77
\                          Support updating/deleting individual metadata with ``POST``.
78
\                          Create object using hashmap.
79
0.3 (June 14, 2011)        Large object support with ``X-Object-Manifest``.
80
\                          Allow for publicly available objects via ``https://hostname/public``.
81
\                          Support time-variant account/container listings.
82
\                          Add source version when duplicating with ``PUT``/``COPY``.
83
\                          Request version in object ``HEAD``/``GET`` requests (list versions with ``GET``).
84
0.2 (May 31, 2011)         Add object meta listing and filtering in containers.
85
\                          Include underlying storage characteristics in container meta.
86
\                          Support for partial object updates through ``POST``.
87
\                          Expose object hashmaps through ``GET``.
88
\                          Support for multi-range object ``GET`` requests.
89
0.1 (May 17, 2011)         Initial release. Based on OpenStack Object Storage Developer Guide API v1 (Apr. 15, 2011).
90
=========================  ================================
91

    
92
Pithos Users and Authentication
93
-------------------------------
94

    
95
In Pithos, each user is uniquely identified by a token. All API requests require a token and each token is internally resolved to an account string. The API uses the account string to identify the user's own files, thus whether a request is local or cross-account.
96

    
97
Pithos does not keep a user database. For development and testing purposes, user identifiers and their corresponding tokens can be defined in the settings file. However, Pithos is designed with an external authentication service in mind. This service must handle the details of validating user credentials and communicate with Pithos via a middleware software component that, given a token, fills in the internal request account variable.
98

    
99
Client software using Pithos, if not already knowing a user's identifier and token, should forward to the ``/login`` URI. The Pithos server, depending on its configuration will redirect to the appropriate login page.
100

    
101
The login URI accepts the following parameters:
102

    
103
======================  =========================
104
Request Parameter Name  Value
105
======================  =========================
106
next                    The URI to redirect to when the process is finished
107
renew                   Force token renewal (no value parameter)
108
force                   Force logout current user (no value parameter)
109
======================  =========================
110

    
111
When done with logging in, the service's login URI should redirect to the URI provided with ``next``, adding the ``token`` parameters which contains authentication token.
112

    
113
If ``next`` request parameter is missing the call fails with BadRequest (400) response status.
114

    
115
A user management service that implements a login URI according to these conventions is Astakos (https://code.grnet.gr/projects/astakos), by GRNET.
116

    
117
User feedback
118
-------------
119

    
120
Client software using Pithos, should forward to the ``/feedback`` URI. The Pithos service, depending on its configuration will delegate the request to the appropriate identity management URI.
121

    
122
============================ =========  ==================
123
Uri                          Method     Description
124
============================ =========  ==================
125
``/pithos/astakos/feedback`` POST       Send feedback
126
============================ =========  ==================
127

    
128
|
129

    
130
======================  =========================
131
Request Parameter Name  Value
132
======================  =========================
133
feedback_msg            Feedback message
134
feedback_data           Additional information about service client status
135
======================  =========================
136

    
137
|
138

    
139
====================  ===========================
140
Request Header Name   Value
141
====================  ===========================
142
X-Auth-Token          User authentication token
143
====================  ===========================
144

    
145
|
146

    
147
=========================== =====================
148
Return Code                 Description
149
=========================== =====================
150
200 (OK)                    The request succeeded
151
502 (Bad Gateway)           Send feedback failure
152
400 (Bad Request)           Method not allowed or invalid message data
153
401 (Unauthorized)          Missing or expired user token
154
500 (Internal Server Error) The request cannot be completed because of an internal error
155
=========================== =====================
156

    
157
User translation catalogs
158
-------------------------
159

    
160
Client software using Pithos, should forward to the ``/user_catalogs`` URI to get uuid to displayname translations and vice versa. The Pithos service, depending on its configuration will delegate the request to the appropriate identity management URI.
161

    
162
================================= =========  ==================
163
Uri                               Method     Description
164
================================= =========  ==================
165
``/pithos/astakos/user_catalogs`` POST       Get 2 catalogs containing uuid to displayname mapping and the opposite
166
================================= =========  ==================
167

    
168
|
169

    
170
====================  ===========================
171
Request Header Name   Value
172
====================  ===========================
173
X-Auth-Token          User authentication token
174
====================  ===========================
175

    
176
The request body is a json formatted dictionary containing a list with uuids and another list of displaynames to translate.
177

    
178
Example request content:
179

    
180
::
181

    
182
  {"displaynames": ["user1@example.com", "user2@example.com"],
183
   "uuids":["ff53baa9-c025-4d56-a6e3-963db0438830", "a9dc21d2-bcb2-4104-9a9e-402b7c70d6d8"]}
184

    
185
Example reply:
186

    
187
::
188

    
189
  {"displayname_catalog": {"user1@example.com": "a9dc21d2-bcb2-4104-9a9e-402b7c70d6d8",
190
                        "user2@example.com": "816351c7-7405-4f26-a968-6380cf47ba1f"},
191
  'uuid_catalog': {"a9dc21d2-bcb2-4104-9a9e-402b7c70d6d8": "user1@example.com",
192
                   "ff53baa9-c025-4d56-a6e3-963db0438830": "user2@example.com"}}
193

    
194

    
195
|
196

    
197
=========================== =====================
198
Return Code                 Description
199
=========================== =====================
200
200 (OK)                    The request succeeded
201
400 (Bad Request)           Method not allowed or request body is not json formatted
202
401 (Unauthorized)          Missing or expired or invalid user token
203
500 (Internal Server Error) The request cannot be completed because of an internal error
204
=========================== =====================
205

    
206
The Pithos API
207
--------------
208

    
209
The URI requests supported by the Pithos API follow one of the following forms:
210

    
211
* Top level: ``https://hostname/v1/``
212
* Account level: ``https://hostname/v1/<account>``
213
* Container level: ``https://hostname/v1/<account>/<container>``
214
* Object level: ``https://hostname/v1/<account>/<container>/<object>``
215

    
216
All requests must include an ``X-Auth-Token`` - as a header, or a parameter.
217

    
218
The allowable request operations and respective return codes per level are presented in the remainder of this chapter. Common to all requests are the following return codes.
219

    
220
==============================  ================================
221
Return Code                     Description
222
==============================  ================================
223
400 (Bad Request)               The request is invalid
224
401 (Unauthorized)              Missing or invalid token
225
403 (Forbidden)                 Request not allowed
226
404 (Not Found)                 The requested resource was not found
227
413 (Request Entity Too Large)  Insufficient quota to complete the request
228
503 (Service Unavailable)       The request cannot be completed because of an internal error
229
==============================  ================================
230

    
231
Top Level
232
^^^^^^^^^
233

    
234
List of operations:
235

    
236
=========  ==================
237
Operation  Description
238
=========  ==================
239
GET        Authentication (for compatibility with the OOS API) or list allowed accounts
240
=========  ==================
241

    
242
GET
243
"""
244

    
245
If the ``X-Auth-User`` and ``X-Auth-Key`` headers are given, a dummy ``X-Auth-Token`` and ``X-Storage-Url`` will be replied, which can be used as a guest token/namespace for testing Pithos.
246

    
247
================  =====================
248
Return Code       Description
249
================  =====================
250
204 (No Content)  The request succeeded
251
================  =====================
252

    
253
If an ``X-Auth-Token`` is already present, the operation will be interpreted as a request to list other accounts that share objects to the user.
254

    
255
======================  =========================
256
Request Parameter Name  Value
257
======================  =========================
258
limit                   The amount of results requested (default is 10000)
259
marker                  Return containers with name lexicographically after marker
260
format                  Optional extended reply type (can be ``json`` or ``xml``)
261
======================  =========================
262

    
263
The reply is a list of account names.
264
If a ``format=xml`` or ``format=json`` argument is given, extended information on the accounts will be returned, serialized in the chosen format.
265
For each account, the information will include the following (names will be in lower case and with hyphens replaced with underscores):
266

    
267
===========================  ============================
268
Name                         Description
269
===========================  ============================
270
name                         The name of the account
271
last_modified                The last account modification date (regardless of ``until``)
272
===========================  ============================
273

    
274
Example ``format=json`` reply:
275

    
276
::
277

    
278
  [{"name": "user-uuid", "last_modified": "2011-12-02T08:10:41.565891+00:00"}, ...]
279

    
280
Example ``format=xml`` reply:
281

    
282
::
283

    
284
  <?xml version="1.0" encoding="UTF-8"?>
285
  <accounts>
286
    <account>
287
      <name>user-uuid</name>
288
      <last_modified>2011-12-02T08:10:41.565891+00:00</last_modified>
289
    </account>
290
    <account>...</account>
291
  </accounts>
292

    
293
===========================  =====================
294
Return Code                  Description
295
===========================  =====================
296
200 (OK)                     The request succeeded
297
204 (No Content)             The user has no access to other accounts (only for non-extended replies)
298
===========================  =====================
299

    
300
Will use a ``200`` return code if the reply is of type JSON/XML.
301

    
302
Account Level
303
^^^^^^^^^^^^^
304

    
305
List of operations:
306

    
307
=========  ==================
308
Operation  Description
309
=========  ==================
310
HEAD       Retrieve account metadata
311
GET        List containers
312
POST       Update account metadata
313
=========  ==================
314

    
315
HEAD
316
""""
317

    
318
====================  ===========================
319
Request Header Name   Value
320
====================  ===========================
321
If-Modified-Since     Retrieve if account has changed since provided timestamp
322
If-Unmodified-Since   Retrieve if account has not changed since provided timestamp
323
====================  ===========================
324

    
325
|
326

    
327
======================  ===================================
328
Request Parameter Name  Value
329
======================  ===================================
330
until                   Optional timestamp
331
======================  ===================================
332

    
333
Cross-user requests are not allowed to use ``until`` and only include the account modification date in the reply.
334

    
335
==========================  =====================
336
Reply Header Name           Value
337
==========================  =====================
338
X-Account-Container-Count   The total number of containers
339
X-Account-Bytes-Used        The total number of bytes stored
340
X-Account-Until-Timestamp   The last account modification date until the timestamp provided
341
X-Account-Group-*           Optional user defined groups
342
X-Account-Policy-Quota      Account quota limit
343
X-Account-Meta-*            Optional user defined metadata
344
Last-Modified               The last account modification date (regardless of ``until``)
345
==========================  =====================
346

    
347
|
348

    
349
================  =====================
350
Return Code       Description
351
================  =====================
352
204 (No Content)  The request succeeded
353
================  =====================
354

    
355

    
356
GET
357
"""
358

    
359
====================  ===========================
360
Request Header Name   Value
361
====================  ===========================
362
If-Modified-Since     Retrieve if account has changed since provided timestamp
363
If-Unmodified-Since   Retrieve if account has not changed since provided timestamp
364
====================  ===========================
365

    
366
|
367

    
368
======================  =========================
369
Request Parameter Name  Value
370
======================  =========================
371
limit                   The amount of results requested (default is 10000)
372
marker                  Return containers with name lexicographically after marker
373
format                  Optional extended reply type (can be ``json`` or ``xml``)
374
shared                  Show only shared containers (no value parameter)
375
public                  Show only public containers (no value parameter / avalaible only for owner requests)
376
until                   Optional timestamp
377
======================  =========================
378

    
379
The reply is a list of container names. Account headers (as in a ``HEAD`` request) will also be included.
380
Cross-user requests are not allowed to use ``until`` and only include the account/container modification dates in the reply.
381

    
382
If a ``format=xml`` or ``format=json`` argument is given, extended information on the containers will be returned, serialized in the chosen format.
383
For each container, the information will include all container metadata, except user-defined (names will be in lower case and with hyphens replaced with underscores):
384

    
385
===========================  ============================
386
Name                         Description
387
===========================  ============================
388
name                         The name of the container
389
count                        The number of objects inside the container
390
bytes                        The total size of the objects inside the container
391
last_modified                The last container modification date (regardless of ``until``)
392
x_container_until_timestamp  The last container modification date until the timestamp provided
393
x_container_policy           Container behavior and limits
394
===========================  ============================
395

    
396
Example ``format=json`` reply:
397

    
398
::
399

    
400
  [{"name": "pithos",
401
    "bytes": 62452,
402
    "count": 8374,
403
    "last_modified": "2011-12-02T08:10:41.565891+00:00",
404
    "x_container_policy": {"quota": "53687091200", "versioning": "auto"}}, ...]
405

    
406
Example ``format=xml`` reply:
407

    
408
::
409

    
410
  <?xml version="1.0" encoding="UTF-8"?>
411
  <account name="user-uuid">
412
    <container>
413
      <name>pithos</name>
414
      <bytes>62452</bytes>
415
      <count>8374</count>
416
      <last_modified>2011-12-02T08:10:41.565891+00:00</last_modified>
417
      <x_container_policy>
418
        <key>quota</key><value>53687091200</value>
419
        <key>versioning</key><value>auto</value>
420
      </x_container_policy>
421
    </container>
422
    <container>...</container>
423
  </account>
424

    
425
For more examples of container details returned in JSON/XML formats refer to the OOS API documentation. In addition to the OOS API, Pithos returns policy fields, grouped as key-value pairs.
426

    
427
===========================  =====================
428
Return Code                  Description
429
===========================  =====================
430
200 (OK)                     The request succeeded
431
204 (No Content)             The account has no containers (only for non-extended replies)
432
304 (Not Modified)           The account has not been modified
433
403 (Forbidden)              Public is requested but the request user is not the path owner
434
412 (Precondition Failed)    The condition set can not be satisfied
435
===========================  =====================
436

    
437
Will use a ``200`` return code if the reply is of type JSON/XML.
438

    
439

    
440
POST
441
""""
442

    
443
====================  ===========================
444
Request Header Name   Value
445
====================  ===========================
446
X-Account-Group-*     Optional user defined groups
447
X-Account-Meta-*      Optional user defined metadata
448
====================  ===========================
449

    
450
|
451

    
452
======================  ============================================
453
Request Parameter Name  Value
454
======================  ============================================
455
update                  Do not replace metadata/groups (no value parameter)
456
======================  ============================================
457

    
458
No reply content/headers.
459

    
460
The operation will overwrite all user defined metadata, except if ``update`` is defined.
461
To create a group, include an ``X-Account-Group-*`` header with the name in the key and a comma separated list of user identifiers in the value. If no ``X-Account-Group-*`` header is present, no changes will be applied to groups. The ``update`` parameter also applies to groups. To delete a specific group, use ``update`` and an empty header value.
462

    
463
================  ===============================
464
Return Code       Description
465
================  ===============================
466
202 (Accepted)    The request has been accepted
467
================  ===============================
468

    
469

    
470
Container Level
471
^^^^^^^^^^^^^^^
472

    
473
List of operations:
474

    
475
=========  ============================
476
Operation  Description
477
=========  ============================
478
HEAD       Retrieve container metadata
479
GET        List objects
480
PUT        Create/update container
481
POST       Update container metadata
482
DELETE     Delete container
483
=========  ============================
484

    
485

    
486
HEAD
487
""""
488

    
489
====================  ===========================
490
Request Header Name   Value
491
====================  ===========================
492
If-Modified-Since     Retrieve if container has changed since provided timestamp
493
If-Unmodified-Since   Retrieve if container has not changed since provided timestamp
494
====================  ===========================
495

    
496
|
497

    
498
======================  ===================================
499
Request Parameter Name  Value
500
======================  ===================================
501
until                   Optional timestamp
502
======================  ===================================
503

    
504
Cross-user requests are not allowed to use ``until`` and only include the container modification date in the reply.
505

    
506
===========================  ===============================
507
Reply Header Name            Value
508
===========================  ===============================
509
X-Container-Object-Count     The total number of objects in the container
510
X-Container-Bytes-Used       The total number of bytes of all objects stored
511
X-Container-Block-Size       The block size used by the storage backend
512
X-Container-Block-Hash       The hash algorithm used for block identifiers in object hashmaps
513
X-Container-Until-Timestamp  The last container modification date until the timestamp provided
514
X-Container-Object-Meta      A list with all meta keys used by objects (**TBD**)
515
X-Container-Policy-*         Container behavior and limits
516
X-Container-Meta-*           Optional user defined metadata
517
Last-Modified                The last container modification date (regardless of ``until``)
518
===========================  ===============================
519

    
520
The keys returned in ``X-Container-Object-Meta`` are all the unique strings after the ``X-Object-Meta-`` prefix, formatted as a comma-separated list. See container ``PUT`` for a reference of policy directives. (**TBD**)
521

    
522
================  ===============================
523
Return Code       Description
524
================  ===============================
525
204 (No Content)  The request succeeded
526
================  ===============================
527

    
528

    
529
GET
530
"""
531

    
532
====================  ===========================
533
Request Header Name   Value
534
====================  ===========================
535
If-Modified-Since     Retrieve if container has changed since provided timestamp
536
If-Unmodified-Since   Retrieve if container has not changed since provided timestamp
537
====================  ===========================
538

    
539
|
540

    
541
======================  ===================================
542
Request Parameter Name  Value
543
======================  ===================================
544
limit                   The amount of results requested (default is 10000)
545
marker                  Return containers with name lexicographically after marker
546
prefix                  Return objects starting with prefix
547
delimiter               Return objects up to the delimiter (discussion follows)
548
path                    Assume ``prefix=path`` and ``delimiter=/``
549
format                  Optional extended reply type (can be ``json`` or ``xml``)
550
meta                    Return objects that satisfy the key queries in the specified comma separated list (use ``<key>``, ``!<key>`` for existence queries, ``<key><op><value>`` for value queries, where ``<op>`` can be one of ``=``, ``!=``, ``<=``, ``>=``, ``<``, ``>``)
551
shared                  Show only objects (no value parameter)
552
public                  Show only public objects (no value parameter / avalaible only for owner requests)
553
until                   Optional timestamp
554
======================  ===================================
555

    
556
The ``path`` parameter overrides ``prefix`` and ``delimiter``. When using ``path``, results will include objects ending in ``delimiter``.
557

    
558
The keys given with ``meta`` will be matched with the strings after the ``X-Object-Meta-`` prefix.
559

    
560
The reply is a list of object names. Container headers (as in a ``HEAD`` request) will also be included.
561
Cross-user requests are not allowed to use ``until`` and include the following limited set of headers in the reply:
562

    
563
===========================  ===============================
564
Reply Header Name            Value
565
===========================  ===============================
566
X-Container-Block-Size       The block size used by the storage backend
567
X-Container-Block-Hash       The hash algorithm used for block identifiers in object hashmaps
568
X-Container-Object-Meta      A list with all meta keys used by allowed objects (**TBD**)
569
Last-Modified                The last container modification date
570
===========================  ===============================
571

    
572
If a ``format=xml`` or ``format=json`` argument is given, extended information on the objects will be returned, serialized in the chosen format.
573
For each object, the information will include all object metadata, except user-defined (names will be in lower case and with hyphens replaced with underscores). User-defined metadata includes ``X-Object-Meta-*``, ``X-Object-Manifest``, ``Content-Disposition`` and ``Content-Encoding`` keys. Also, sharing directives will only be included with the actual shared objects (inherited permissions are not calculated):
574

    
575
==========================  ======================================
576
Name                        Description
577
==========================  ======================================
578
name                        The name of the object
579
hash                        The ETag of the object
580
bytes                       The size of the object
581
content_type                The MIME content type of the object
582
last_modified               The last object modification date (regardless of version)
583
x_object_hash               The Merkle hash
584
x_object_uuid               The object's UUID
585
x_object_version            The object's version identifier
586
x_object_version_timestamp  The object's version timestamp
587
x_object_modified_by        The user that committed the object's version
588
x_object_sharing            Object permissions (optional)
589
x_object_allowed_to         Allowed actions on object (optional)
590
x_object_public             Object's publicly accessible URI (optional: present if the object is public and the request user is the object owner)
591
==========================  ======================================
592

    
593
Sharing metadata and last modification timestamp will only be returned if there is no ``until`` parameter defined.
594

    
595
Extended replies may also include virtual directory markers in separate sections of the ``json`` or ``xml`` results.
596
Virtual directory markers are only included when ``delimiter`` is explicitly set. They correspond to the substrings up to and including the first occurrence of the delimiter.
597
In JSON results they appear as dictionaries with only a ``subdir`` key. In XML results they appear interleaved with ``<object>`` tags as ``<subdir name="..." />``.
598
In case there is an object with the same name as a virtual directory marker, the object will be returned.
599

    
600
Example ``format=json`` reply:
601

    
602
::
603

    
604
  [{"name": "object",
605
    "bytes": 0,
606
    "hash": "d41d8cd98f00b204e9800998ecf8427e",
607
    "content_type": "application/octet-stream",
608
    "last_modified": "2011-12-02T08:10:41.565891+00:00",
609
    "x_object_hash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
610
    "x_object_uuid": "8ed9af1b-c948-4bb6-82b0-48344f5c822c",
611
    "x_object_version": 98,
612
    "x_object_version_timestamp": "1322813441.565891",
613
    "x_object_modified_by": "user-uuid"}, ...]
614

    
615
Example ``format=xml`` reply:
616

    
617
::
618

    
619
  <?xml version="1.0" encoding="UTF-8"?>
620
  <container name="pithos">
621
    <object>
622
      <name>object</name>
623
      <bytes>0</bytes>
624
      <hash>d41d8cd98f00b204e9800998ecf8427e</hash>
625
      <content_type>application/octet-stream</content_type>
626
      <last_modified>2011-12-02T08:10:41.565891+00:00</last_modified>
627
      <x_object_hash>e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855</x_object_hash>
628
      <x_object_uuid>8ed9af1b-c948-4bb6-82b0-48344f5c822c</x_object_uuid>
629
      <x_object_version>98</x_object_version>
630
      <x_object_version_timestamp>1322813441.565891</x_object_version_timestamp>
631
      <x_object_modified_by>user-uuid</x_object_modified_by>
632
    </object>
633
    <object>...</object>
634
  </container>
635

    
636
For more examples of container details returned in JSON/XML formats refer to the OOS API documentation. In addition to the OOS API, Pithos returns more fields that should help with synchronization.
637

    
638
===========================  ===============================
639
Return Code                  Description
640
===========================  ===============================
641
200 (OK)                     The request succeeded
642
204 (No Content)             The container has no objects (only for non-extended replies)
643
304 (Not Modified)           The container has not been modified
644
403 (Forbidden)              Public is requested but the request user is not the path owner
645
412 (Precondition Failed)    The condition set can not be satisfied
646
===========================  ===============================
647

    
648
Will use a ``200`` return code if the reply is of type JSON/XML.
649

    
650

    
651
PUT
652
"""
653

    
654
====================  ================================
655
Request Header Name   Value
656
====================  ================================
657
X-Container-Policy-*  Container behavior and limits
658
X-Container-Meta-*    Optional user defined metadata
659
====================  ================================
660
 
661
No reply content/headers.
662

    
663
If no policy is defined, the container will be created with the default values.
664
Available policy directives:
665

    
666
* ``versioning``: Set to ``auto`` or ``none`` (default is ``auto``)
667
* ``quota``: Size limit in KB (default is ``0`` - unlimited)
668

    
669
If the container already exists, the operation is equal to a ``POST`` with ``update`` defined.
670

    
671
================  ===============================
672
Return Code       Description
673
================  ===============================
674
201 (Created)     The container has been created
675
202 (Accepted)    The request has been accepted
676
================  ===============================
677

    
678

    
679
POST
680
""""
681

    
682
====================  ================================
683
Request Header Name   Value
684
====================  ================================
685
Content-Length        The size of the supplied data (optional, to upload)
686
Content-Type          The MIME content type of the supplied data (optional, to upload)
687
Transfer-Encoding     Set to ``chunked`` to specify incremental uploading (if used, ``Content-Length`` is ignored)
688
X-Container-Policy-*  Container behavior and limits
689
X-Container-Meta-*    Optional user defined metadata
690
====================  ================================
691

    
692
|
693

    
694
======================  ============================================
695
Request Parameter Name  Value
696
======================  ============================================
697
format                  Optional hash list reply type (can be ``json`` or ``xml``)
698
update                  Do not replace metadata/policy (no value parameter)
699
======================  ============================================
700

    
701
No reply content/headers, except when uploading data, where the reply consists of a list of hashes for the blocks received (in the format specified).
702

    
703
The operation will overwrite all user defined metadata, except if ``update`` is defined.
704
To change policy, include an ``X-Container-Policy-*`` header with the name in the key. If no ``X-Container-Policy-*`` header is present, no changes will be applied to policy. The ``update`` parameter also applies to policy - deleted values will revert to defaults. To delete/revert a specific policy directive, use ``update`` and an empty header value. See container ``PUT`` for a reference of policy directives.
705

    
706
To upload blocks of data to the container, set ``Content-Type`` to ``application/octet-stream`` and ``Content-Length`` to a valid value (except if using ``chunked`` as the ``Transfer-Encoding``).
707

    
708
================  ===============================
709
Return Code       Description
710
================  ===============================
711
202 (Accepted)    The request has been accepted
712
================  ===============================
713

    
714

    
715
DELETE
716
""""""
717

    
718
======================  ===================================
719
Request Parameter Name  Value
720
======================  ===================================
721
until                   Optional timestamp
722
delimiter               Optional delete objects starting with container name and delimiter
723
======================  ===================================
724

    
725
If ``until`` is defined, the container is "purged" up to that time (the history of all objects up to then is deleted). If also ``delimiter`` is defined, purge is applied only on the container.
726

    
727
No reply content/headers.
728

    
729
================  ===============================
730
Return Code       Description
731
================  ===============================
732
204 (No Content)  The request succeeded
733
409 (Conflict)    The container is not empty
734
================  ===============================
735

    
736

    
737
Object Level
738
^^^^^^^^^^^^
739

    
740
List of operations:
741

    
742
=========  =================================
743
Operation  Description
744
=========  =================================
745
HEAD       Retrieve object metadata
746
GET        Read object data
747
PUT        Write object data or copy/move object
748
COPY       Copy object
749
MOVE       Move object
750
POST       Update object metadata/data
751
DELETE     Delete object
752
=========  =================================
753

    
754

    
755
HEAD
756
""""
757

    
758
====================  ================================
759
Request Header Name   Value
760
====================  ================================
761
If-Match              Retrieve if ETags match
762
If-None-Match         Retrieve if ETags don't match
763
If-Modified-Since     Retrieve if object has changed since provided timestamp
764
If-Unmodified-Since   Retrieve if object has not changed since provided timestamp
765
====================  ================================
766

    
767
|
768

    
769
======================  ===================================
770
Request Parameter Name  Value
771
======================  ===================================
772
version                 Optional version identifier
773
======================  ===================================
774

    
775
|
776

    
777
==========================  ===============================
778
Reply Header Name           Value
779
==========================  ===============================
780
ETag                        The ETag of the object
781
Content-Length              The size of the object
782
Content-Type                The MIME content type of the object
783
Last-Modified               The last object modification date (regardless of version)
784
Content-Encoding            The encoding of the object (optional)
785
Content-Disposition         The presentation style of the object (optional)
786
X-Object-Hash               The Merkle hash
787
X-Object-UUID               The object's UUID
788
X-Object-Version            The object's version identifier
789
X-Object-Version-Timestamp  The object's version timestamp
790
X-Object-Modified-By        The user that comitted the object's version
791
X-Object-Manifest           Object parts prefix in ``<container>/<object>`` form (optional)
792
X-Object-Sharing            Object permissions (optional)
793
X-Object-Shared-By          Object inheriting permissions (optional)
794
X-Object-Allowed-To         Allowed actions on object (optional)
795
X-Object-Public             Object's publicly accessible URI (optional: present if the object is public and the request user is the object owner)
796
X-Object-Meta-*             Optional user defined metadata
797
==========================  ===============================
798

    
799
|
800

    
801
================  ===============================
802
Return Code       Description
803
================  ===============================
804
200 (No Content)  The request succeeded
805
================  ===============================
806

    
807

    
808
GET
809
"""
810

    
811
====================  ================================
812
Request Header Name   Value
813
====================  ================================
814
Range                 Optional range of data to retrieve
815
If-Range              Retrieve the missing part if entity is unchanged; otherwise, retrieve the entire new entity (used together with Range header)
816
If-Match              Retrieve if ETags match
817
If-None-Match         Retrieve if ETags don't match
818
If-Modified-Since     Retrieve if object has changed since provided timestamp
819
If-Unmodified-Since   Retrieve if object has not changed since provided timestamp
820
====================  ================================
821

    
822
|
823

    
824
======================  ===================================
825
Request Parameter Name  Value
826
======================  ===================================
827
format                  Optional extended reply type (can be ``json`` or ``xml``)
828
hashmap                 Optional request for hashmap (no value parameter)
829
version                 Optional version identifier or ``list`` (specify a format if requesting a list)
830
======================  ===================================
831

    
832
The reply is the object's data (or part of it), except if a hashmap is requested with ``hashmap``, or a version list with ``version=list`` (in both cases an extended reply format must be specified). Object headers (as in a ``HEAD`` request) are always included.
833

    
834
Hashmaps expose the underlying storage format of the object. Note that each hash is computed after trimming trailing null bytes of the corresponding block. The ``X-Object-Hash`` header reports the single Merkle hash of the object's hashmap (refer to http://bittorrent.org/beps/bep_0030.html for more information).
835

    
836
Example ``format=json`` reply:
837

    
838
::
839

    
840
  {"block_hash": "sha1", "hashes": ["7295c41da03d7f916440b98e32c4a2a39351546c", ...], "block_size": 131072, "bytes": 242}
841

    
842
Example ``format=xml`` reply:
843

    
844
::
845

    
846
  <?xml version="1.0" encoding="UTF-8"?>
847
  <object name="file" bytes="24223726" block_size="131072" block_hash="sha1">
848
    <hash>7295c41da03d7f916440b98e32c4a2a39351546c</hash>
849
    <hash>...</hash>
850
  </object>
851

    
852
Version lists include the version identifier and timestamp for each available object version. Version identifiers can be arbitrary strings, so use the timestamp to find newer versions.
853

    
854
Example ``format=json`` reply:
855

    
856
::
857

    
858
  {"versions": [[85, "1322734861.248469"], [86, "1322734905.009272"], ...]}
859

    
860
Example ``format=xml`` reply:
861

    
862
::
863

    
864
  <?xml version="1.0" encoding="UTF-8"?>
865
  <object name="file">
866
    <version timestamp="1322734861.248469">85</version>
867
    <version timestamp="1322734905.009272">86</version>
868
    <version timestamp="...">...</version>
869
  </object>
870

    
871
The ``Range`` header may include multiple ranges, as outlined in RFC2616. Then the ``Content-Type`` of the reply will be ``multipart/byteranges`` and each part will include a ``Content-Range`` header.
872

    
873
==========================  ===============================
874
Reply Header Name           Value
875
==========================  ===============================
876
ETag                        The ETag of the object
877
Content-Length              The size of the data returned
878
Content-Type                The MIME content type of the object
879
Content-Range               The range of data included (only on a single range request)
880
Last-Modified               The last object modification date (regardless of version)
881
Content-Encoding            The encoding of the object (optional)
882
Content-Disposition         The presentation style of the object (optional)
883
X-Object-Hash               The Merkle hash
884
X-Object-UUID               The object's UUID
885
X-Object-Version            The object's version identifier
886
X-Object-Version-Timestamp  The object's version timestamp
887
X-Object-Modified-By        The user that comitted the object's version
888
X-Object-Manifest           Object parts prefix in ``<container>/<object>`` form (optional)
889
X-Object-Sharing            Object permissions (optional)
890
X-Object-Shared-By          Object inheriting permissions (optional)
891
X-Object-Allowed-To         Allowed actions on object (optional)
892
X-Object-Public             Object's publicly accessible URI (optional: present if the object is public and the request user is the object owner)
893
X-Object-Meta-*             Optional user defined metadata
894
==========================  ===============================
895

    
896
Sharing headers (``X-Object-Sharing``, ``X-Object-Shared-By`` and ``X-Object-Allowed-To``) are only included if the request is for the object's latest version (no specific ``version`` parameter is set).
897

    
898
===========================  ==============================
899
Return Code                  Description
900
===========================  ==============================
901
200 (OK)                     The request succeeded
902
206 (Partial Content)        The range request succeeded
903
304 (Not Modified)           The object has not been modified
904
412 (Precondition Failed)    The condition set can not be satisfied
905
416 (Range Not Satisfiable)  The requested range is out of limits
906
===========================  ==============================
907

    
908

    
909
PUT
910
"""
911

    
912
====================  ================================
913
Request Header Name   Value
914
====================  ================================
915
If-Match              Put if ETags match with current object
916
If-None-Match         Put if ETags don't match with current object
917
ETag                  The MD5 hash of the object (optional to check written data)
918
Content-Length        The size of the data written
919
Content-Type          The MIME content type of the object
920
Transfer-Encoding     Set to ``chunked`` to specify incremental uploading (if used, ``Content-Length`` is ignored)
921
X-Copy-From           The source path in the form ``/<container>/<object>``
922
X-Move-From           The source path in the form ``/<container>/<object>``
923
X-Source-Account      The source account to copy/move from
924
X-Source-Version      The source version to copy from
925
Content-Encoding      The encoding of the object (optional)
926
Content-Disposition   The presentation style of the object (optional)
927
X-Object-Manifest     Object parts prefix in ``<container>/<object>`` form (optional)
928
X-Object-Sharing      Object permissions (optional)
929
X-Object-Public       Object is publicly accessible (optional)
930
X-Object-Meta-*       Optional user defined metadata
931
====================  ================================
932

    
933
|
934

    
935
======================  ===================================
936
Request Parameter Name  Value
937
======================  ===================================
938
format                  Optional extended request/conflict response type (can be ``json`` or ``xml``)
939
hashmap                 Optional hashmap provided instead of data (no value parameter)
940
delimiter               Optional copy/move objects starting with object's path and delimiter (to be used with X-Copy-From/X-Move-From)
941
======================  ===================================
942

    
943
The request is the object's data (or part of it), except if a hashmap is provided (using ``hashmap`` and ``format`` parameters). If using a hashmap and all different parts are stored in the server, the object is created. Otherwise the server returns Conflict (409) with the list of the missing parts (in simple text format, with one hash per line, or in JSON/XML - depending on the ``format`` parameter).
944

    
945
Hashmaps should be formatted as outlined in ``GET``.
946

    
947
==========================  ===============================
948
Reply Header Name           Value
949
==========================  ===============================
950
ETag                        The MD5 (or the Merkle if MD5 is deactivated) hash of the object
951
X-Object-Version            The object's new version
952
==========================  ===============================
953

    
954
The ``X-Object-Sharing`` header may include either a ``read=...`` comma-separated user/group list, or a ``write=...`` comma-separated user/group list, or both separated by a semicolon (``;``). Groups are specified as ``<account>:<group>``. To publish the object, set ``X-Object-Public`` to ``true``. To unpublish, set to ``false``, or use an empty header value.
955

    
956
==============================  ==============================
957
Return Code                     Description
958
==============================  ==============================
959
201 (Created)                   The object has been created
960
409 (Conflict)                  The object can not be created from the provided hashmap (a list of missing hashes will be included in the reply)
961
411 (Length Required)           Missing ``Content-Length`` or ``Content-Type`` in the request
962
413 (Request Entity Too Large)  Insufficient quota to complete the request
963
422 (Unprocessable Entity)      The MD5 (or the Merkle if MD5 is deactivated) checksum of the data written to the storage system does not match the (optionally) supplied ETag value
964
==============================  ==============================
965

    
966

    
967
COPY
968
""""
969

    
970
====================  ================================
971
Request Header Name   Value
972
====================  ================================
973
If-Match              Proceed if ETags match with object
974
If-None-Match         Proceed if ETags don't match with object
975
Destination           The destination path in the form ``/<container>/<object>``
976
Destination-Account   The destination account to copy to
977
Content-Type          The MIME content type of the object (optional :sup:`*`)
978
Content-Encoding      The encoding of the object (optional)
979
Content-Disposition   The presentation style of the object (optional)
980
X-Source-Version      The source version to copy from
981
X-Object-Manifest     Object parts prefix in ``<container>/<object>`` form (optional)
982
X-Object-Sharing      Object permissions (optional)
983
X-Object-Public       Object is publicly accessible (optional)
984
X-Object-Meta-*       Optional user defined metadata
985
====================  ================================
986

    
987
:sup:`*` *When using django locally with the supplied web server, use the ignore_content_type parameter, or do provide a valid Content-Type, as a type of text/plain is applied by default to all requests. Client software should always state ignore_content_type, except when a Content-Type is explicitly defined by the user.*
988

    
989
======================  ===================================
990
Request Parameter Name  Value
991
======================  ===================================
992
format                  Optional conflict response type (can be ``json`` or ``xml``)
993
ignore_content_type     Ignore the supplied Content-Type
994
delimiter               Optional copy objects starting with object's path and delimiter
995
======================  ===================================
996

    
997
Refer to ``PUT``/``POST`` for a description of request headers. Metadata is also copied, updated with any values defined. Sharing/publishing options are not copied.
998

    
999
==========================  ===============================
1000
Reply Header Name           Value
1001
==========================  ===============================
1002
X-Object-Version            The object's new version
1003
==========================  ===============================
1004

    
1005
|
1006

    
1007
==============================  ==============================
1008
Return Code                     Description
1009
==============================  ==============================
1010
201 (Created)                   The object has been created
1011
413 (Request Entity Too Large)  Insufficient quota to complete the request
1012
==============================  ==============================
1013

    
1014

    
1015
MOVE
1016
""""
1017

    
1018
Same as ``COPY``, without the ``X-Source-Version`` request header. The ``MOVE`` operation is always applied on the latest version.
1019

    
1020

    
1021
POST
1022
""""
1023

    
1024
====================  ================================
1025
Request Header Name   Value
1026
====================  ================================
1027
If-Match              Proceed if ETags match with object
1028
If-None-Match         Proceed if ETags don't match with object
1029
Content-Length        The size of the data written (optional, to update)
1030
Content-Type          The MIME content type of the object (optional, to update)
1031
Content-Range         The range of data supplied (optional, to update)
1032
Transfer-Encoding     Set to ``chunked`` to specify incremental uploading (if used, ``Content-Length`` is ignored)
1033
Content-Encoding      The encoding of the object (optional)
1034
Content-Disposition   The presentation style of the object (optional)
1035
X-Source-Object       Update with data from the object at path ``/<container>/<object>`` (optional, to update)
1036
X-Source-Account      The source account to update from
1037
X-Source-Version      The source version to update from (optional, to update)
1038
X-Object-Bytes        The updated object's final size (optional, when updating)
1039
X-Object-Manifest     Object parts prefix in ``<container>/<object>`` form (optional)
1040
X-Object-Sharing      Object permissions (optional)
1041
X-Object-Public       Object is publicly accessible (optional)
1042
X-Object-Meta-*       Optional user defined metadata
1043
====================  ================================
1044

    
1045
|
1046

    
1047
======================  ============================================
1048
Request Parameter Name  Value
1049
======================  ============================================
1050
format                  Optional conflict response type (can be ``json`` or ``xml``)
1051
update                  Do not replace metadata (no value parameter)
1052
======================  ============================================
1053

    
1054
The ``Content-Encoding``, ``Content-Disposition``, ``X-Object-Manifest`` and ``X-Object-Meta-*`` headers are considered to be user defined metadata. An operation without the ``update`` parameter will overwrite all previous values and remove any keys not supplied. When using ``update`` any metadata with an empty value will be deleted.
1055

    
1056
To change permissions, include an ``X-Object-Sharing`` header (as defined in ``PUT``). To publish, include an ``X-Object-Public`` header, with a value of ``true``. If no such headers are defined, no changes will be applied to sharing/public. Use empty values to remove permissions/unpublish (unpublishing also works with ``false`` as a header value). Sharing options are applied to the object - not its versions.
1057

    
1058
To update an object's data:
1059

    
1060
* Either set ``Content-Type`` to ``application/octet-stream``, or provide an object with ``X-Source-Object``. If ``Content-Type`` has some other value, it will be ignored and only the metadata will be updated.
1061
* If the data is supplied in the request (using ``Content-Type`` instead of ``X-Source-Object``), a valid ``Content-Length`` header is required - except if using chunked transfers (set ``Transfer-Encoding`` to ``chunked``).
1062
* Set ``Content-Range`` as specified in RFC2616, with the following differences:
1063

    
1064
  * Client software MAY omit ``last-byte-pos`` of if the length of the range being transferred is unknown or difficult to determine.
1065
  * Client software SHOULD not specify the ``instance-length`` (use a ``*``), unless there is a reason for performing a size check at the server.
1066
* If ``Content-Range`` used has a ``byte-range-resp-spec = *``, data will be appended to the object.
1067

    
1068
Optionally, truncate the updated object to the desired length with the ``X-Object-Bytes`` header.
1069

    
1070
A data update will trigger an ETag change. Updated ETags may happen asynchronously and appear at the server with a delay.
1071

    
1072
No reply content. No reply headers if only metadata is updated.
1073

    
1074
==========================  ===============================
1075
Reply Header Name           Value
1076
==========================  ===============================
1077
ETag                        The new ETag of the object (data updated)
1078
X-Object-Version            The object's new version
1079
==========================  ===============================
1080

    
1081
|
1082

    
1083
==============================  ==============================
1084
Return Code                     Description
1085
==============================  ==============================
1086
202 (Accepted)                  The request has been accepted (not a data update)
1087
204 (No Content)                The request succeeded (data updated)
1088
411 (Length Required)           Missing ``Content-Length`` in the request
1089
413 (Request Entity Too Large)  Insufficient quota to complete the request
1090
416 (Range Not Satisfiable)     The supplied range is invalid
1091
==============================  ==============================
1092

    
1093
The ``POST`` method can also be used for creating an object via a standard HTML form. If the request ``Content-Type`` is ``multipart/form-data``, none of the above headers will be processed. The form should have an ``X-Object-Data`` field, as in the following example. The token is passed as a request parameter. ::
1094

    
1095
  <form method="post" action="https://storage.example.synnefo.org/pithos/object-store/v1.0/user-uuid/folder/EXAMPLE.txt?X-Auth-Token=0000" enctype="multipart/form-data">
1096
    <input type="file" name="X-Object-Data">
1097
    <input type="submit">
1098
  </form>
1099

    
1100
This will create/override the object with the given name, as if using ``PUT``. The ``Content-Type`` of the object will be set to the value of the corresponding header sent in the part of the request containing the data (usually, automatically handled by the browser). Metadata, sharing and other object attributes can not be set this way. The response will contain the object's ETag.
1101

    
1102
==========================  ===============================
1103
Reply Header Name           Value
1104
==========================  ===============================
1105
ETag                        The MD5 (or the Merkle if MD5 is deactivated) hash of the object
1106
X-Object-Version            The object's new version
1107
==========================  ===============================
1108

    
1109
|
1110

    
1111
==============================  ==============================
1112
Return Code                     Description
1113
==============================  ==============================
1114
201 (Created)                   The object has been created
1115
413 (Request Entity Too Large)  Insufficient quota to complete the request
1116
==============================  ==============================
1117

    
1118

    
1119
DELETE
1120
""""""
1121

    
1122
======================  ===================================
1123
Request Parameter Name  Value
1124
======================  ===================================
1125
until                   Optional timestamp
1126
delimiter               Optional delete also objects starting with object's path and delimiter
1127
======================  ===================================
1128

    
1129
If ``until`` is defined, the object is "purged" up to that time (the history up to then is deleted). If also ``delimiter`` is defined, purge is applied only on the object.
1130

    
1131
No reply content/headers.
1132

    
1133
===========================  ==============================
1134
Return Code                  Description
1135
===========================  ==============================
1136
204 (No Content)             The request succeeded
1137
===========================  ==============================
1138

    
1139
Sharing and Public Objects
1140
^^^^^^^^^^^^^^^^^^^^^^^^^^
1141

    
1142
Read and write control in Pithos is managed by setting appropriate permissions with the ``X-Object-Sharing`` header. The permissions are applied using directory-based inheritance. A directory is an object with the corresponding content type. The default delimiter is ``/``. Thus, each set of authorization directives is applied to all objects in the directory object where the corresponding ``X-Object-Sharing`` header is defined. If there are nested/overlapping permissions, the closest to the object is applied. When retrieving an object, the ``X-Object-Shared-By`` header reports where it gets its permissions from. If not present, the object is the actual source of authorization directives.
1143

    
1144
A user may ``GET`` another account or container. The result will include a limited reply, containing only the allowed containers or objects respectively. A top-level request with an authentication token, will return a list of allowed accounts, so the user can easily find out which other users share objects. The ``X-Object-Allowed-To`` header lists the actions allowed on an object, if it does not belong to the requesting user.
1145

    
1146
Shared objects that are also public do not expose the ``X-Object-Public`` meta information.
1147

    
1148
Objects that are marked as public, via the ``X-Object-Public`` meta, are also available at the corresponding URI returned for ``HEAD`` or ``GET``. Requests for public objects do not need to include an ``X-Auth-Token``. Pithos will ignore request parameters and only include the following headers in the reply (all ``X-Object-*`` meta is hidden):
1149

    
1150
==========================  ===============================
1151
Reply Header Name           Value
1152
==========================  ===============================
1153
ETag                        The ETag of the object
1154
Content-Length              The size of the data returned
1155
Content-Type                The MIME content type of the object
1156
Content-Range               The range of data included (only on a single range request)
1157
Last-Modified               The last object modification date (regardless of version)
1158
Content-Encoding            The encoding of the object (optional)
1159
Content-Disposition         The presentation style of the object (optional)
1160
==========================  ===============================
1161

    
1162
Public objects are not included and do not influence cross-user listings. They are, however, readable by all users.
1163

    
1164
Summary
1165
^^^^^^^
1166

    
1167
List of differences from the OOS API:
1168

    
1169
* Support for ``X-Account-Meta-*`` style headers at the account level. Use ``POST`` to update.
1170
* Support for ``X-Container-Meta-*`` style headers at the container level. Can be set when creating via ``PUT``. Use ``POST`` to update.
1171
* Header ``X-Container-Object-Meta`` at the container level and parameter ``meta`` in container listings. (**TBD**)
1172
* Account and container policies to manage behavior and limits. Container behavior overrides account settings. Account quota sets the maximum bytes limit, regardless of container values.
1173
* Headers ``X-Container-Block-*`` at the container level, exposing the underlying storage characteristics.
1174
* All metadata replies, at all levels, include latest modification information.
1175
* At all levels, a ``HEAD`` or ``GET`` request may use ``If-Modified-Since`` and ``If-Unmodified-Since`` headers.
1176
* Container/object lists include more fields if the reply is of type JSON/XML. Some names are kept to their OOS API equivalents for compatibility.
1177
* Option to include only shared containers/objects in listings.
1178
* Object metadata allowed, in addition to ``X-Object-Meta-*``: ``Content-Encoding``, ``Content-Disposition``, ``X-Object-Manifest``. These are all replaced with every update operation, except if using the ``update`` parameter (in which case individual keys can also be deleted). Deleting meta by providing empty values also works when copying/moving an object.
1179
* Multi-range object ``GET`` support as outlined in RFC2616.
1180
* Object hashmap retrieval through ``GET`` and the ``format`` parameter.
1181
* Object create via hashmap through ``PUT`` and the ``format`` parameter.
1182
* The object's Merkle hash is always returned in the ``X-Object-Hash`` header.
1183
* The object's UUID is always returned in the ``X-Object-UUID`` header. The UUID remains unchanged, even when the object's data or metadata changes, or the object is moved to another path (is renamed). A new UUID is assigned when creating or copying an object.
1184
* Object create using ``POST`` to support standard HTML forms.
1185
* Partial object updates through ``POST``, using the ``Content-Length``, ``Content-Type``, ``Content-Range`` and ``Transfer-Encoding`` headers. Use another object's data to update with ``X-Source-Object`` and ``X-Source-Version``. Truncate with ``X-Object-Bytes``.
1186
* Include new version identifier in replies for object replace/change requests.
1187
* Object ``MOVE`` support and ``ignore_content_type`` parameter in both ``COPY`` and ``MOVE``.
1188
* Conditional object create/update operations, using ``If-Match`` and ``If-None-Match`` headers.
1189
* Time-variant account/container listings via the ``until`` parameter.
1190
* Object versions - parameter ``version`` in ``HEAD``/``GET`` (list versions with ``GET``), ``X-Object-Version-*`` meta in replies, ``X-Source-Version`` in ``PUT``/``COPY``.
1191
* Sharing/publishing with ``X-Object-Sharing``, ``X-Object-Public`` at the object level. Cross-user operations are allowed - controlled by sharing directives. Available actions in cross-user requests are reported with ``X-Object-Allowed-To``. Permissions may include groups defined with ``X-Account-Group-*`` at the account level. These apply to the object - not its versions.
1192
* Support for directory-based inheritance when enforcing permissions. Parent object carrying the authorization directives is reported in ``X-Object-Shared-By``.
1193
* Copy and move between accounts with ``X-Source-Account`` and ``Destination-Account`` headers.
1194
* Large object support with ``X-Object-Manifest``.
1195
* Trace the user that created/modified an object with ``X-Object-Modified-By``.
1196
* Purge container/object history with the ``until`` parameter in ``DELETE``.
1197
* Bulk COPY/MOVE/DELETE objects starting with prefix
1198

    
1199
Clarifications/suggestions:
1200

    
1201
* All non-ASCII characters in headers should be URL-encoded.
1202
* Authentication is done by another system. The token is used in the same way, but it is obtained differently. The top level ``GET`` request is kept compatible with the OOS API and allows for guest/testing operations.
1203
* Some processing is done in the variable part of all ``X-*-Meta-*`` headers. If it includes underscores, they will be converted to dashes and the first letter of all intra-dash strings will be capitalized.
1204
* A ``GET`` reply for a level will include all headers of the corresponding ``HEAD`` request.
1205
* To avoid conflicts between objects and virtual directory markers in container listings, it is recommended that object names do not end with the delimiter used.
1206
* The ``Accept`` header may be used in requests instead of the ``format`` parameter to specify the desired request/reply format. The parameter overrides the header.
1207
* Container/object lists use a ``200`` return code if the reply is of type JSON/XML. The reply will include an empty JSON/XML.
1208
* In headers, dates are formatted according to RFC 1123. In extended information listings, the ``last_modified`` field is formatted according to ISO 8601 (for OOS API compatibility). All other fields (Pithos extensions) use integer tiemstamps.
1209
* The ``Last-Modified`` header value always reflects the actual latest change timestamp, regardless of time control parameters and version requests. Time precondition checks with ``If-Modified-Since`` and ``If-Unmodified-Since`` headers are applied to this value.
1210
* A copy/move using ``PUT``/``COPY``/``MOVE`` will always update metadata, keeping all old values except the ones redefined in the request headers.
1211
* A ``HEAD`` or ``GET`` for an ``X-Object-Manifest`` object, will include modified ``Content-Length`` and ``ETag`` headers, according to the characteristics of the objects under the specified prefix. The ``Etag`` will be the MD5 hash of the corresponding ETags concatenated. In extended container listings there is no metadata processing.
1212

    
1213
Recommended Practices and Examples
1214
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1215

    
1216
Assuming an authentication token is obtained, the following high-level operations are available - shown with ``curl``:
1217

    
1218
* Get account information ::
1219

    
1220
    curl -X HEAD -D - \
1221
         -H "X-Auth-Token: 0000" \
1222
         https://storage.example.synnefo.org/pithos/object-store/v1.0/user-uuid
1223

    
1224
* List available containers ::
1225

    
1226
    curl -X GET -D - \
1227
         -H "X-Auth-Token: 0000" \
1228
         https://storage.example.synnefo.org/pithos/object-store/v1.0/user-uuid
1229

    
1230
* Get container information ::
1231

    
1232
    curl -X HEAD -D - \
1233
         -H "X-Auth-Token: 0000" \
1234
         https://storage.example.synnefo.org/pithos/object-store/v1.0/user-uuid/pithos
1235

    
1236
* Add a new container ::
1237

    
1238
    curl -X PUT -D - \
1239
         -H "X-Auth-Token: 0000" \
1240
         https://storage.example.synnefo.org/pithos/object-store/v1.0/user-uuid/test
1241

    
1242
* Delete a container ::
1243

    
1244
    curl -X DELETE -D - \
1245
         -H "X-Auth-Token: 0000" \
1246
         https://storage.example.synnefo.org/pithos/object-store/v1.0/user-uuid/test
1247

    
1248
* List objects in a container ::
1249

    
1250
    curl -X GET -D - \
1251
         -H "X-Auth-Token: 0000" \
1252
         https://storage.example.synnefo.org/pithos/object-store/v1.0/user-uuid/pithos
1253

    
1254
* List objects in a container (extended reply) ::
1255

    
1256
    curl -X GET -D - \
1257
         -H "X-Auth-Token: 0000" \
1258
         https://storage.example.synnefo.org/pithos/object-store/v1.0/user-uuid/pithos?format=json
1259

    
1260
  It is recommended that extended replies are cached and subsequent requests utilize the ``If-Modified-Since`` header.
1261

    
1262
* List metadata keys used by objects in a container
1263

    
1264
  Will be in the ``X-Container-Object-Meta`` reply header, included in container information or object list (``HEAD`` or ``GET``). (**TBD**)
1265

    
1266
* List objects in a container having a specific meta defined ::
1267

    
1268
    curl -X GET -D - \
1269
         -H "X-Auth-Token: 0000" \
1270
         https://storage.example.synnefo.org/pithos/object-store/v1.0/user-uuid/pithos?meta=favorites
1271

    
1272
* Retrieve an object ::
1273

    
1274
    curl -X GET -D - \
1275
         -H "X-Auth-Token: 0000" \
1276
         https://storage.example.synnefo.org/pithos/object-store/v1.0/user-uuid/pithos/README.txt
1277

    
1278
* Retrieve an object (specific ranges of data) ::
1279

    
1280
    curl -X GET -D - \
1281
         -H "X-Auth-Token: 0000" \
1282
         -H "Range: bytes=0-9" \
1283
         https://storage.example.synnefo.org/pithos/object-store/v1.0/user-uuid/pithos/README.txt
1284

    
1285
  This will return the first 10 bytes. To get the first 10, bytes 30-39 and the last 100 use ``Range: bytes=0-9,30-39,-100``.
1286

    
1287
* Add a new object (folder type) (**TBD**) ::
1288

    
1289
    curl -X PUT -D - \
1290
         -H "X-Auth-Token: 0000" \
1291
         -H "Content-Type: application/directory" \
1292
         https://storage.example.synnefo.org/pithos/object-store/v1.0/user-uuid/pithos/folder
1293

    
1294
* Add a new object ::
1295

    
1296
    curl -X PUT -D - \
1297
         -H "X-Auth-Token: 0000" \
1298
         -H "Content-Type: text/plain" \
1299
         -T EXAMPLE.txt
1300
         https://storage.example.synnefo.org/pithos/object-store/v1.0/user-uuid/pithos/folder/EXAMPLE.txt
1301

    
1302
* Update an object ::
1303

    
1304
    curl -X POST -D - \
1305
         -H "X-Auth-Token: 0000" \
1306
         -H "Content-Length: 10" \
1307
         -H "Content-Type: application/octet-stream" \
1308
         -H "Content-Range: bytes 10-19/*" \
1309
         -d "0123456789" \
1310
         https://storage.example.synnefo.org/pithos/object-store/v1.0/user-uuid/folder/EXAMPLE.txt
1311

    
1312
  This will update bytes 10-19 with the data specified.
1313

    
1314
* Update an object (append) ::
1315

    
1316
    curl -X POST -D - \
1317
         -H "X-Auth-Token: 0000" \
1318
         -H "Content-Length: 10" \
1319
         -H "Content-Type: application/octet-stream" \
1320
         -H "Content-Range: bytes */*" \
1321
         -d "0123456789" \
1322
         https://storage.example.synnefo.org/pithos/object-store/v1.0/user-uuid/folder/EXAMPLE.txt
1323

    
1324
* Update an object (truncate) ::
1325

    
1326
    curl -X POST -D - \
1327
         -H "X-Auth-Token: 0000" \
1328
         -H "X-Source-Object: /folder/EXAMPLE.txt" \
1329
         -H "Content-Range: bytes 0-0/*" \
1330
         -H "X-Object-Bytes: 0" \
1331
         https://storage.example.synnefo.org/pithos/object-store/v1.0/user-uuid/folder/EXAMPLE.txt
1332

    
1333
  This will truncate the object to 0 bytes.
1334

    
1335
* Add object metadata ::
1336

    
1337
    curl -X POST -D - \
1338
         -H "X-Auth-Token: 0000" \
1339
         -H "X-Object-Meta-First: first_meta_value" \
1340
         -H "X-Object-Meta-Second: second_meta_value" \
1341
         https://storage.example.synnefo.org/pithos/object-store/v1.0/user-uuid/folder/EXAMPLE.txt
1342

    
1343
* Delete object metadata ::
1344

    
1345
    curl -X POST -D - \
1346
         -H "X-Auth-Token: 0000" \
1347
         -H "X-Object-Meta-First: first_meta_value" \
1348
         https://storage.example.synnefo.org/pithos/object-store/v1.0/user-uuid/folder/EXAMPLE.txt
1349

    
1350
  Metadata can only be "set". To delete ``X-Object-Meta-Second``, reset all metadata.
1351

    
1352
* Delete an object ::
1353

    
1354
    curl -X DELETE -D - \
1355
         -H "X-Auth-Token: 0000" \
1356
         https://storage.example.synnefo.org/pithos/object-store/v1.0/user-uuid/folder/EXAMPLE.txt