Statistics
| Branch: | Tag: | Revision:

root / docs / source / devguide.rst @ ed4725e4

History | View | Annotate | Download (69.7 kB)

1
Pithos v2 Developer Guide
2
=========================
3

    
4
Introduction
5
------------
6

    
7
Pithos is a storage service implemented by GRNET (http://www.grnet.gr). Data is stored as objects, organized in containers, belonging to an account. This hierarchy of storage layers has been inspired by the OpenStack Object Storage (OOS) API and similar CloudFiles API by Rackspace. The Pithos API follows the OOS API as closely as possible. One of the design requirements has been to be able to use Pithos with clients built for the OOS, without changes.
8

    
9
However, to be able to take full advantage of the Pithos infrastructure, client software should be aware of the extensions that differentiate Pithos from OOS. Pithos objects can be updated, or appended to. Pithos will store sharing permissions per object and enforce corresponding authorization policies. Automatic version management, allows taking account and container listings back in time, as well as reading previous instances of objects.
10

    
11
The storage backend of Pithos is block oriented, permitting efficient, deduplicated data placement. The block structure of objects is exposed at the API layer, in order to encourage external software to implement advanced data management operations.
12

    
13
This document's goals are:
14

    
15
* Define the Pithos ReST API that allows the storage and retrieval of data and metadata via HTTP calls
16
* Specify metadata semantics and user interface guidelines for a common experience across client software implementations
17

    
18
The present document is meant to be read alongside the OOS API documentation. Thus, it is suggested that the reader is familiar with associated technologies, the OOS API as well as the first version of the Pithos API. This document refers to the second version of Pithos. Information on the first version of the storage API can be found at http://code.google.com/p/gss.
19

    
20
Whatever marked as to be determined (**TBD**), should not be considered by implementors.
21

    
22
More info about Pithos can be found here: https://code.grnet.gr/projects/pithos
23

    
24
Document Revisions
25
^^^^^^^^^^^^^^^^^^
26

    
27
=========================  ================================
28
Revision                   Description
29
=========================  ================================
30
0.9 (Feb 17, 2012)         Change permissions model.
31
\                          Do not include user-defined metadata in account/container/object listings.
32
0.8 (Jan 24, 2012)         Update allowed versioning values.
33
\                          Change policy/meta formatting in JSON/XML replies.
34
\                          Document that all non-ASCII characters in headers should be URL-encoded.
35
\                          Support metadata-based queries when listing objects at the container level.
36
\                          Note Content-Type issue when using the internal django web server.
37
\                          Add object UUID field.
38
\                          Always reply with the MD5 in the ETag.
39
\                          Note that ``/login`` will only work if an external authentication system is defined.
40
\                          Include option to ignore Content-Type on ``COPY``/``MOVE``.
41
\                          Use format parameter for conflict (409) and uploaded hash list (container level) replies.
42
0.7 (Nov 21, 2011)         Suggest upload/download methods using hashmaps.
43
\                          Propose syncing algorithm.
44
\                          Support cross-account object copy and move.
45
\                          Pass token as a request parameter when using ``POST`` via an HTML form.
46
\                          Optionally use source account to update object from another object.
47
\                          Use container ``POST`` to upload missing blocks of data.
48
\                          Report policy in account headers.
49
\                          Add insufficient quota reply.
50
\                          Use special meta to always report Merkle hash.
51
0.6 (Sept 13, 2011)        Reply with Merkle hash as the ETag when updating objects.
52
\                          Include version id in object replace/change replies.
53
\                          Change conflict (409) replies format to text.
54
\                          Tags should be migrated to a meta value.
55
\                          Container ``PUT`` updates metadata/policy.
56
\                          Report allowed actions in shared object replies.
57
\                          Provide ``https://hostname/login`` for Shibboleth authentication.
58
\                          Use ``hashmap`` parameter in object ``GET``/``PUT`` to use hashmaps.
59
0.5 (July 22, 2011)        Object update from another object's data.
60
\                          Support object truncate.
61
\                          Create object using a standard HTML form.
62
\                          Purge container/object history.
63
\                          List other accounts that share objects with a user.
64
\                          List shared containers/objects.
65
\                          Update implementation guidelines.
66
\                          Check preconditions when creating/updating objects.
67
0.4 (July 01, 2011)        Object permissions and account groups.
68
\                          Control versioning behavior and container quotas with container policy directives.
69
\                          Support updating/deleting individual metadata with ``POST``.
70
\                          Create object using hashmap.
71
0.3 (June 14, 2011)        Large object support with ``X-Object-Manifest``.
72
\                          Allow for publicly available objects via ``https://hostname/public``.
73
\                          Support time-variant account/container listings. 
74
\                          Add source version when duplicating with ``PUT``/``COPY``.
75
\                          Request version in object ``HEAD``/``GET`` requests (list versions with ``GET``).
76
0.2 (May 31, 2011)         Add object meta listing and filtering in containers.
77
\                          Include underlying storage characteristics in container meta.
78
\                          Support for partial object updates through ``POST``.
79
\                          Expose object hashmaps through ``GET``.
80
\                          Support for multi-range object ``GET`` requests.
81
0.1 (May 17, 2011)         Initial release. Based on OpenStack Object Storage Developer Guide API v1 (Apr. 15, 2011).
82
=========================  ================================
83

    
84
Pithos Users and Authentication
85
-------------------------------
86

    
87
In Pithos, each user is uniquely identified by a token. All API requests require a token and each token is internally resolved to an account string. The API uses the account string to identify the user's own files, thus whether a request is local or cross-account.
88

    
89
Pithos does not keep a user database. For development and testing purposes, user identifiers and their corresponding tokens can be defined in the settings file. However, Pithos is designed with an external authentication service in mind. This service must handle the details of validating user credentials and communicate with Pithos via a middleware software component that, given a token, fills in the internal request account variable.
90

    
91
Client software using Pithos, if not already knowing a user's identifier and token, should forward to the ``/login`` URI. The Pithos server, depending on its configuration will redirect to the appropriate login page.
92

    
93
The login URI accepts the following parameters:
94

    
95
======================  =========================
96
Request Parameter Name  Value
97
======================  =========================
98
next                    The URI to redirect to when the process is finished
99
renew                   Force token renewal (no value parameter)
100
force                   Force logout current user (no value parameter)
101
======================  =========================
102

    
103
When done with logging in, the service's login URI should redirect to the URI provided with ``next``, adding ``user`` and ``token`` parameters, which contain the account and token fields respectively.
104

    
105
A user management service that implements a login URI according to these conventions is Astakos (https://code.grnet.gr/projects/astakos), by GRNET.
106

    
107
The Pithos API
108
--------------
109

    
110
The URI requests supported by the Pithos API follow one of the following forms:
111

    
112
* Top level: ``https://hostname/v1/``
113
* Account level: ``https://hostname/v1/<account>``
114
* Container level: ``https://hostname/v1/<account>/<container>``
115
* Object level: ``https://hostname/v1/<account>/<container>/<object>``
116

    
117
All requests must include an ``X-Auth-Token`` - as a header, or a parameter.
118

    
119
The allowable request operations and respective return codes per level are presented in the remainder of this chapter. Common to all requests are the following return codes.
120

    
121
=========================  ================================
122
Return Code                Description
123
=========================  ================================
124
400 (Bad Request)          The request is invalid
125
401 (Unauthorized)         Missing or invalid token
126
403 (Forbidden)            Request not allowed
127
404 (Not Found)            The requested resource was not found
128
503 (Service Unavailable)  The request cannot be completed because of an internal error
129
=========================  ================================
130

    
131
Top Level
132
^^^^^^^^^
133

    
134
List of operations:
135

    
136
=========  ==================
137
Operation  Description
138
=========  ==================
139
GET        Authentication (for compatibility with the OOS API) or list allowed accounts
140
=========  ==================
141

    
142
GET
143
"""
144

    
145
If the ``X-Auth-User`` and ``X-Auth-Key`` headers are given, a dummy ``X-Auth-Token`` and ``X-Storage-Url`` will be replied, which can be used as a guest token/namespace for testing Pithos.
146

    
147
================  =====================
148
Return Code       Description
149
================  =====================
150
204 (No Content)  The request succeeded
151
================  =====================
152

    
153
If an ``X-Auth-Token`` is already present, the operation will be interpreted as a request to list other accounts that share objects to the user.
154

    
155
======================  =========================
156
Request Parameter Name  Value
157
======================  =========================
158
limit                   The amount of results requested (default is 10000)
159
marker                  Return containers with name lexicographically after marker
160
format                  Optional extended reply type (can be ``json`` or ``xml``)
161
======================  =========================
162

    
163
The reply is a list of account names.
164
If a ``format=xml`` or ``format=json`` argument is given, extended information on the accounts will be returned, serialized in the chosen format.
165
For each account, the information will include the following (names will be in lower case and with hyphens replaced with underscores):
166

    
167
===========================  ============================
168
Name                         Description
169
===========================  ============================
170
name                         The name of the account
171
last_modified                The last account modification date (regardless of ``until``)
172
===========================  ============================
173

    
174
Example ``format=json`` reply:
175

    
176
::
177

    
178
  [{"name": "user", "last_modified": "2011-12-02T08:10:41.565891+00:00"}, ...]
179

    
180
Example ``format=xml`` reply:
181

    
182
::
183

    
184
  <?xml version="1.0" encoding="UTF-8"?>
185
  <accounts>
186
    <account>
187
      <name>user</name>
188
      <last_modified>2011-12-02T08:10:41.565891+00:00</last_modified>
189
    </account>
190
    <account>...</account>
191
  </accounts>
192

    
193
===========================  =====================
194
Return Code                  Description
195
===========================  =====================
196
200 (OK)                     The request succeeded
197
204 (No Content)             The user has no access to other accounts (only for non-extended replies)
198
===========================  =====================
199

    
200
Will use a ``200`` return code if the reply is of type JSON/XML.
201

    
202
Account Level
203
^^^^^^^^^^^^^
204

    
205
List of operations:
206

    
207
=========  ==================
208
Operation  Description
209
=========  ==================
210
HEAD       Retrieve account metadata
211
GET        List containers
212
POST       Update account metadata
213
=========  ==================
214

    
215
HEAD
216
""""
217

    
218
====================  ===========================
219
Request Header Name   Value
220
====================  ===========================
221
If-Modified-Since     Retrieve if account has changed since provided timestamp
222
If-Unmodified-Since   Retrieve if account has not changed since provided timestamp
223
====================  ===========================
224

    
225
|
226

    
227
======================  ===================================
228
Request Parameter Name  Value
229
======================  ===================================
230
until                   Optional timestamp
231
======================  ===================================
232

    
233
Cross-user requests are not allowed to use ``until`` and only include the account modification date in the reply.
234

    
235
==========================  =====================
236
Reply Header Name           Value
237
==========================  =====================
238
X-Account-Container-Count   The total number of containers
239
X-Account-Bytes-Used        The total number of bytes stored
240
X-Account-Until-Timestamp   The last account modification date until the timestamp provided
241
X-Account-Group-*           Optional user defined groups
242
X-Account-Policy-*          Account behavior and limits
243
X-Account-Meta-*            Optional user defined metadata
244
Last-Modified               The last account modification date (regardless of ``until``)
245
==========================  =====================
246

    
247
|
248

    
249
================  =====================
250
Return Code       Description
251
================  =====================
252
204 (No Content)  The request succeeded
253
================  =====================
254

    
255

    
256
GET
257
"""
258

    
259
====================  ===========================
260
Request Header Name   Value
261
====================  ===========================
262
If-Modified-Since     Retrieve if account has changed since provided timestamp
263
If-Unmodified-Since   Retrieve if account has not changed since provided timestamp
264
====================  ===========================
265

    
266
|
267

    
268
======================  =========================
269
Request Parameter Name  Value
270
======================  =========================
271
limit                   The amount of results requested (default is 10000)
272
marker                  Return containers with name lexicographically after marker
273
format                  Optional extended reply type (can be ``json`` or ``xml``)
274
shared                  Show only shared containers (no value parameter)
275
until                   Optional timestamp
276
======================  =========================
277

    
278
The reply is a list of container names. Account headers (as in a ``HEAD`` request) will also be included.
279
Cross-user requests are not allowed to use ``until`` and only include the account/container modification dates in the reply.
280

    
281
If a ``format=xml`` or ``format=json`` argument is given, extended information on the containers will be returned, serialized in the chosen format.
282
For each container, the information will include all container metadata, except user-defined (names will be in lower case and with hyphens replaced with underscores):
283

    
284
===========================  ============================
285
Name                         Description
286
===========================  ============================
287
name                         The name of the container
288
count                        The number of objects inside the container
289
bytes                        The total size of the objects inside the container
290
last_modified                The last container modification date (regardless of ``until``)
291
x_container_until_timestamp  The last container modification date until the timestamp provided
292
x_container_policy           Container behavior and limits
293
===========================  ============================
294

    
295
Example ``format=json`` reply:
296

    
297
::
298

    
299
  [{"name": "pithos",
300
    "bytes": 62452,
301
    "count": 8374,
302
    "last_modified": "2011-12-02T08:10:41.565891+00:00",
303
    "x_container_policy": {"quota": "53687091200", "versioning": "auto"}}, ...]
304

    
305
Example ``format=xml`` reply:
306

    
307
::
308

    
309
  <?xml version="1.0" encoding="UTF-8"?>
310
  <account name="user">
311
    <container>
312
      <name>pithos</name>
313
      <bytes>62452</bytes>
314
      <count>8374</count>
315
      <last_modified>2011-12-02T08:10:41.565891+00:00</last_modified>
316
      <x_container_policy>
317
        <key>quota</key><value>53687091200</value>
318
        <key>versioning</key><value>auto</value>
319
      </x_container_policy>
320
    </container>
321
    <container>...</container>
322
  </account>
323

    
324
For more examples of container details returned in JSON/XML formats refer to the OOS API documentation. In addition to the OOS API, Pithos returns policy fields, grouped as key-value pairs.
325

    
326
===========================  =====================
327
Return Code                  Description
328
===========================  =====================
329
200 (OK)                     The request succeeded
330
204 (No Content)             The account has no containers (only for non-extended replies)
331
304 (Not Modified)           The account has not been modified
332
412 (Precondition Failed)    The condition set can not be satisfied
333
===========================  =====================
334

    
335
Will use a ``200`` return code if the reply is of type JSON/XML.
336

    
337

    
338
POST
339
""""
340

    
341
====================  ===========================
342
Request Header Name   Value
343
====================  ===========================
344
X-Account-Group-*     Optional user defined groups
345
X-Account-Meta-*      Optional user defined metadata
346
====================  ===========================
347

    
348
|
349

    
350
======================  ============================================
351
Request Parameter Name  Value
352
======================  ============================================
353
update                  Do not replace metadata/groups (no value parameter)
354
======================  ============================================
355

    
356
No reply content/headers.
357

    
358
The operation will overwrite all user defined metadata, except if ``update`` is defined.
359
To create a group, include an ``X-Account-Group-*`` header with the name in the key and a comma separated list of user names in the value. If no ``X-Account-Group-*`` header is present, no changes will be applied to groups. The ``update`` parameter also applies to groups. To delete a specific group, use ``update`` and an empty header value.
360

    
361
================  ===============================
362
Return Code       Description
363
================  ===============================
364
202 (Accepted)    The request has been accepted
365
================  ===============================
366

    
367

    
368
Container Level
369
^^^^^^^^^^^^^^^
370

    
371
List of operations:
372

    
373
=========  ============================
374
Operation  Description
375
=========  ============================
376
HEAD       Retrieve container metadata
377
GET        List objects
378
PUT        Create/update container
379
POST       Update container metadata
380
DELETE     Delete container
381
=========  ============================
382

    
383

    
384
HEAD
385
""""
386

    
387
====================  ===========================
388
Request Header Name   Value
389
====================  ===========================
390
If-Modified-Since     Retrieve if container has changed since provided timestamp
391
If-Unmodified-Since   Retrieve if container has not changed since provided timestamp
392
====================  ===========================
393

    
394
|
395

    
396
======================  ===================================
397
Request Parameter Name  Value
398
======================  ===================================
399
until                   Optional timestamp
400
======================  ===================================
401

    
402
Cross-user requests are not allowed to use ``until`` and only include the container modification date in the reply.
403

    
404
===========================  ===============================
405
Reply Header Name            Value
406
===========================  ===============================
407
X-Container-Object-Count     The total number of objects in the container
408
X-Container-Bytes-Used       The total number of bytes of all objects stored
409
X-Container-Block-Size       The block size used by the storage backend
410
X-Container-Block-Hash       The hash algorithm used for block identifiers in object hashmaps
411
X-Container-Until-Timestamp  The last container modification date until the timestamp provided
412
X-Container-Object-Meta      A list with all meta keys used by objects (**TBD**)
413
X-Container-Policy-*         Container behavior and limits
414
X-Container-Meta-*           Optional user defined metadata
415
Last-Modified                The last container modification date (regardless of ``until``)
416
===========================  ===============================
417

    
418
The keys returned in ``X-Container-Object-Meta`` are all the unique strings after the ``X-Object-Meta-`` prefix, formatted as a comma-separated list. See container ``PUT`` for a reference of policy directives. (**TBD**)
419

    
420
================  ===============================
421
Return Code       Description
422
================  ===============================
423
204 (No Content)  The request succeeded
424
================  ===============================
425

    
426

    
427
GET
428
"""
429

    
430
====================  ===========================
431
Request Header Name   Value
432
====================  ===========================
433
If-Modified-Since     Retrieve if container has changed since provided timestamp
434
If-Unmodified-Since   Retrieve if container has not changed since provided timestamp
435
====================  ===========================
436

    
437
|
438

    
439
======================  ===================================
440
Request Parameter Name  Value
441
======================  ===================================
442
limit                   The amount of results requested (default is 10000)
443
marker                  Return containers with name lexicographically after marker
444
prefix                  Return objects starting with prefix
445
delimiter               Return objects up to the delimiter (discussion follows)
446
path                    Assume ``prefix=path`` and ``delimiter=/``
447
format                  Optional extended reply type (can be ``json`` or ``xml``)
448
meta                    Return objects that satisfy the key queries in the specified comma separated list (use ``<key>``, ``!<key>`` for existence queries, ``<key><op><value>`` for value queries, where ``<op>`` can be one of ``=``, ``!=``, ``<=``, ``>=``, ``<``, ``>``)
449
shared                  Show only shared objects (no value parameter)
450
until                   Optional timestamp
451
======================  ===================================
452

    
453
The ``path`` parameter overrides ``prefix`` and ``delimiter``. When using ``path``, results will include objects ending in ``delimiter``.
454

    
455
The keys given with ``meta`` will be matched with the strings after the ``X-Object-Meta-`` prefix.
456

    
457
The reply is a list of object names. Container headers (as in a ``HEAD`` request) will also be included.
458
Cross-user requests are not allowed to use ``until`` and include the following limited set of headers in the reply:
459

    
460
===========================  ===============================
461
Reply Header Name            Value
462
===========================  ===============================
463
X-Container-Block-Size       The block size used by the storage backend
464
X-Container-Block-Hash       The hash algorithm used for block identifiers in object hashmaps
465
X-Container-Object-Meta      A list with all meta keys used by allowed objects (**TBD**)
466
Last-Modified                The last container modification date
467
===========================  ===============================
468

    
469
If a ``format=xml`` or ``format=json`` argument is given, extended information on the objects will be returned, serialized in the chosen format.
470
For each object, the information will include all object metadata, except user-defined (names will be in lower case and with hyphens replaced with underscores). User-defined metadata includes ``X-Object-Meta-*``, ``X-Object-Manifest``, ``Content-Disposition`` and ``Content-Encoding`` keys. Also, sharing directives will only be included with the actual shared objects (inherited permissions are not calculated):
471

    
472
==========================  ======================================
473
Name                        Description
474
==========================  ======================================
475
name                        The name of the object
476
hash                        The ETag of the object
477
bytes                       The size of the object
478
content_type                The MIME content type of the object
479
last_modified               The last object modification date (regardless of version)
480
x_object_hash               The Merkle hash
481
x_object_uuid               The object's UUID
482
x_object_version            The object's version identifier
483
x_object_version_timestamp  The object's version timestamp
484
x_object_modified_by        The user that committed the object's version
485
x_object_sharing            Object permissions (optional)
486
x_object_allowed_to         Allowed actions on object (optional)
487
x_object_public             Object's publicly accessible URI (optional)
488
==========================  ======================================
489

    
490
Sharing metadata and last modification timestamp will only be returned if there is no ``until`` parameter defined.
491

    
492
Extended replies may also include virtual directory markers in separate sections of the ``json`` or ``xml`` results.
493
Virtual directory markers are only included when ``delimiter`` is explicitly set. They correspond to the substrings up to and including the first occurrence of the delimiter.
494
In JSON results they appear as dictionaries with only a ``subdir`` key. In XML results they appear interleaved with ``<object>`` tags as ``<subdir name="..." />``.
495
In case there is an object with the same name as a virtual directory marker, the object will be returned.
496

    
497
Example ``format=json`` reply:
498

    
499
::
500

    
501
  [{"name": "object",
502
    "bytes": 0,
503
    "hash": "d41d8cd98f00b204e9800998ecf8427e",
504
    "content_type": "application/octet-stream",
505
    "last_modified": "2011-12-02T08:10:41.565891+00:00",
506
    "x_object_hash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
507
    "x_object_uuid": "8ed9af1b-c948-4bb6-82b0-48344f5c822c",
508
    "x_object_version": 98,
509
    "x_object_version_timestamp": "1322813441.565891",
510
    "x_object_modified_by": "user"}, ...]
511

    
512
Example ``format=xml`` reply:
513

    
514
::
515

    
516
  <?xml version="1.0" encoding="UTF-8"?>
517
  <container name="pithos">
518
    <object>
519
      <name>object</name>
520
      <bytes>0</bytes>
521
      <hash>d41d8cd98f00b204e9800998ecf8427e</hash>
522
      <content_type>application/octet-stream</content_type>
523
      <last_modified>2011-12-02T08:10:41.565891+00:00</last_modified>
524
      <x_object_hash>e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855</x_object_hash>
525
      <x_object_uuid>8ed9af1b-c948-4bb6-82b0-48344f5c822c</x_object_uuid>
526
      <x_object_version>98</x_object_version>
527
      <x_object_version_timestamp>1322813441.565891</x_object_version_timestamp>
528
      <x_object_modified_by>chazapis</x_object_modified_by>
529
    </object>
530
    <object>...</object>
531
  </container>
532

    
533
For more examples of container details returned in JSON/XML formats refer to the OOS API documentation. In addition to the OOS API, Pithos returns more fields that should help with synchronization.
534

    
535
===========================  ===============================
536
Return Code                  Description
537
===========================  ===============================
538
200 (OK)                     The request succeeded
539
204 (No Content)             The account has no containers (only for non-extended replies)
540
304 (Not Modified)           The container has not been modified
541
412 (Precondition Failed)    The condition set can not be satisfied
542
===========================  ===============================
543

    
544
Will use a ``200`` return code if the reply is of type JSON/XML.
545

    
546

    
547
PUT
548
"""
549

    
550
====================  ================================
551
Request Header Name   Value
552
====================  ================================
553
X-Container-Policy-*  Container behavior and limits
554
X-Container-Meta-*    Optional user defined metadata
555
====================  ================================
556
 
557
No reply content/headers.
558

    
559
If no policy is defined, the container will be created with the default values.
560
Available policy directives:
561

    
562
* ``versioning``: Set to ``auto`` or ``none`` (default is ``auto``)
563
* ``quota``: Size limit in KB (default is ``0`` - unlimited)
564

    
565
If the container already exists, the operation is equal to a ``POST`` with ``update`` defined.
566

    
567
================  ===============================
568
Return Code       Description
569
================  ===============================
570
201 (Created)     The container has been created
571
202 (Accepted)    The request has been accepted
572
================  ===============================
573

    
574

    
575
POST
576
""""
577

    
578
====================  ================================
579
Request Header Name   Value
580
====================  ================================
581
Content-Length        The size of the supplied data (optional, to upload)
582
Content-Type          The MIME content type of the supplied data (optional, to upload)
583
Transfer-Encoding     Set to ``chunked`` to specify incremental uploading (if used, ``Content-Length`` is ignored)
584
X-Container-Policy-*  Container behavior and limits
585
X-Container-Meta-*    Optional user defined metadata
586
====================  ================================
587

    
588
|
589

    
590
======================  ============================================
591
Request Parameter Name  Value
592
======================  ============================================
593
format                  Optional hash list reply type (can be ``json`` or ``xml``)
594
update                  Do not replace metadata/policy (no value parameter)
595
======================  ============================================
596

    
597
No reply content/headers, except when uploading data, where the reply consists of a list of hashes for the blocks received (in the format specified).
598

    
599
The operation will overwrite all user defined metadata, except if ``update`` is defined.
600
To change policy, include an ``X-Container-Policy-*`` header with the name in the key. If no ``X-Container-Policy-*`` header is present, no changes will be applied to policy. The ``update`` parameter also applies to policy - deleted values will revert to defaults. To delete/revert a specific policy directive, use ``update`` and an empty header value. See container ``PUT`` for a reference of policy directives.
601

    
602
To upload blocks of data to the container, set ``Content-Type`` to ``application/octet-stream`` and ``Content-Length`` to a valid value (except if using ``chunked`` as the ``Transfer-Encoding``).
603

    
604
================  ===============================
605
Return Code       Description
606
================  ===============================
607
202 (Accepted)    The request has been accepted
608
================  ===============================
609

    
610

    
611
DELETE
612
""""""
613

    
614
======================  ===================================
615
Request Parameter Name  Value
616
======================  ===================================
617
until                   Optional timestamp
618
======================  ===================================
619

    
620
If ``until`` is defined, the container is "purged" up to that time (the history of all objects up to then is deleted).
621

    
622
No reply content/headers.
623

    
624
================  ===============================
625
Return Code       Description
626
================  ===============================
627
204 (No Content)  The request succeeded
628
409 (Conflict)    The container is not empty
629
================  ===============================
630

    
631

    
632
Object Level
633
^^^^^^^^^^^^
634

    
635
List of operations:
636

    
637
=========  =================================
638
Operation  Description
639
=========  =================================
640
HEAD       Retrieve object metadata
641
GET        Read object data
642
PUT        Write object data or copy/move object
643
COPY       Copy object
644
MOVE       Move object
645
POST       Update object metadata/data
646
DELETE     Delete object
647
=========  =================================
648

    
649

    
650
HEAD
651
""""
652

    
653
====================  ================================
654
Request Header Name   Value
655
====================  ================================
656
If-Match              Retrieve if ETags match
657
If-None-Match         Retrieve if ETags don't match
658
If-Modified-Since     Retrieve if object has changed since provided timestamp
659
If-Unmodified-Since   Retrieve if object has not changed since provided timestamp
660
====================  ================================
661

    
662
|
663

    
664
======================  ===================================
665
Request Parameter Name  Value
666
======================  ===================================
667
version                 Optional version identifier
668
======================  ===================================
669

    
670
|
671

    
672
==========================  ===============================
673
Reply Header Name           Value
674
==========================  ===============================
675
ETag                        The ETag of the object
676
Content-Length              The size of the object
677
Content-Type                The MIME content type of the object
678
Last-Modified               The last object modification date (regardless of version)
679
Content-Encoding            The encoding of the object (optional)
680
Content-Disposition         The presentation style of the object (optional)
681
X-Object-Hash               The Merkle hash
682
X-Object-UUID               The object's UUID
683
X-Object-Version            The object's version identifier
684
X-Object-Version-Timestamp  The object's version timestamp
685
X-Object-Modified-By        The user that comitted the object's version
686
X-Object-Manifest           Object parts prefix in ``<container>/<object>`` form (optional)
687
X-Object-Sharing            Object permissions (optional)
688
X-Object-Shared-By          Object inheriting permissions (optional)
689
X-Object-Allowed-To         Allowed actions on object (optional)
690
X-Object-Public             Object's publicly accessible URI (optional)
691
X-Object-Meta-*             Optional user defined metadata
692
==========================  ===============================
693

    
694
|
695

    
696
================  ===============================
697
Return Code       Description
698
================  ===============================
699
200 (No Content)  The request succeeded
700
================  ===============================
701

    
702

    
703
GET
704
"""
705

    
706
====================  ================================
707
Request Header Name   Value
708
====================  ================================
709
Range                 Optional range of data to retrieve
710
If-Range              Retrieve the missing part if entity is unchanged; otherwise, retrieve the entire new entity (used together with Range header)
711
If-Match              Retrieve if ETags match
712
If-None-Match         Retrieve if ETags don't match
713
If-Modified-Since     Retrieve if object has changed since provided timestamp
714
If-Unmodified-Since   Retrieve if object has not changed since provided timestamp
715
====================  ================================
716

    
717
|
718

    
719
======================  ===================================
720
Request Parameter Name  Value
721
======================  ===================================
722
format                  Optional extended reply type (can be ``json`` or ``xml``)
723
hashmap                 Optional request for hashmap (no value parameter)
724
version                 Optional version identifier or ``list`` (specify a format if requesting a list)
725
======================  ===================================
726

    
727
The reply is the object's data (or part of it), except if a hashmap is requested with ``hashmap``, or a version list with ``version=list`` (in both cases an extended reply format must be specified). Object headers (as in a ``HEAD`` request) are always included.
728

    
729
Hashmaps expose the underlying storage format of the object. Note that each hash is computed after trimming trailing null bytes of the corresponding block. The ``X-Object-Hash`` header reports the single Merkle hash of the object's hashmap (refer to http://bittorrent.org/beps/bep_0030.html for more information).
730

    
731
Example ``format=json`` reply:
732

    
733
::
734

    
735
  {"block_hash": "sha1", "hashes": ["7295c41da03d7f916440b98e32c4a2a39351546c", ...], "block_size": 131072, "bytes": 242}
736

    
737
Example ``format=xml`` reply:
738

    
739
::
740

    
741
  <?xml version="1.0" encoding="UTF-8"?>
742
  <object name="file" bytes="24223726" block_size="131072" block_hash="sha1">
743
    <hash>7295c41da03d7f916440b98e32c4a2a39351546c</hash>
744
    <hash>...</hash>
745
  </object>
746

    
747
Version lists include the version identifier and timestamp for each available object version. Version identifiers can be arbitrary strings, so use the timestamp to find newer versions.
748

    
749
Example ``format=json`` reply:
750

    
751
::
752

    
753
  {"versions": [[85, "1322734861.248469"], [86, "1322734905.009272"], ...]}
754

    
755
Example ``format=xml`` reply:
756

    
757
::
758

    
759
  <?xml version="1.0" encoding="UTF-8"?>
760
  <object name="file">
761
    <version timestamp="1322734861.248469">85</version>
762
    <version timestamp="1322734905.009272">86</version>
763
    <version timestamp="...">...</version>
764
  </object>
765

    
766
The ``Range`` header may include multiple ranges, as outlined in RFC2616. Then the ``Content-Type`` of the reply will be ``multipart/byteranges`` and each part will include a ``Content-Range`` header.
767

    
768
==========================  ===============================
769
Reply Header Name           Value
770
==========================  ===============================
771
ETag                        The ETag of the object
772
Content-Length              The size of the data returned
773
Content-Type                The MIME content type of the object
774
Content-Range               The range of data included (only on a single range request)
775
Last-Modified               The last object modification date (regardless of version)
776
Content-Encoding            The encoding of the object (optional)
777
Content-Disposition         The presentation style of the object (optional)
778
X-Object-Hash               The Merkle hash
779
X-Object-UUID               The object's UUID
780
X-Object-Version            The object's version identifier
781
X-Object-Version-Timestamp  The object's version timestamp
782
X-Object-Modified-By        The user that comitted the object's version
783
X-Object-Manifest           Object parts prefix in ``<container>/<object>`` form (optional)
784
X-Object-Sharing            Object permissions (optional)
785
X-Object-Shared-By          Object inheriting permissions (optional)
786
X-Object-Allowed-To         Allowed actions on object (optional)
787
X-Object-Public             Object's publicly accessible URI (optional)
788
X-Object-Meta-*             Optional user defined metadata
789
==========================  ===============================
790

    
791
Sharing headers (``X-Object-Sharing``, ``X-Object-Shared-By`` and ``X-Object-Allowed-To``) are only included if the request is for the object's latest version (no specific ``version`` parameter is set).
792

    
793
===========================  ==============================
794
Return Code                  Description
795
===========================  ==============================
796
200 (OK)                     The request succeeded
797
206 (Partial Content)        The range request succeeded
798
304 (Not Modified)           The object has not been modified
799
412 (Precondition Failed)    The condition set can not be satisfied
800
416 (Range Not Satisfiable)  The requested range is out of limits
801
===========================  ==============================
802

    
803

    
804
PUT
805
"""
806

    
807
====================  ================================
808
Request Header Name   Value
809
====================  ================================
810
If-Match              Put if ETags match with current object
811
If-None-Match         Put if ETags don't match with current object
812
ETag                  The MD5 hash of the object (optional to check written data)
813
Content-Length        The size of the data written
814
Content-Type          The MIME content type of the object
815
Transfer-Encoding     Set to ``chunked`` to specify incremental uploading (if used, ``Content-Length`` is ignored)
816
X-Copy-From           The source path in the form ``/<container>/<object>``
817
X-Move-From           The source path in the form ``/<container>/<object>``
818
X-Source-Account      The source account to copy/move from
819
X-Source-Version      The source version to copy from
820
Content-Encoding      The encoding of the object (optional)
821
Content-Disposition   The presentation style of the object (optional)
822
X-Object-Manifest     Object parts prefix in ``<container>/<object>`` form (optional)
823
X-Object-Sharing      Object permissions (optional)
824
X-Object-Public       Object is publicly accessible (optional)
825
X-Object-Meta-*       Optional user defined metadata
826
====================  ================================
827

    
828
|
829

    
830
======================  ===================================
831
Request Parameter Name  Value
832
======================  ===================================
833
format                  Optional extended request/conflict response type (can be ``json`` or ``xml``)
834
hashmap                 Optional hashmap provided instead of data (no value parameter)
835
======================  ===================================
836

    
837
The request is the object's data (or part of it), except if a hashmap is provided (using ``hashmap`` and ``format`` parameters). If using a hashmap and all different parts are stored in the server, the object is created. Otherwise the server returns Conflict (409) with the list of the missing parts (in simple text format, with one hash per line, or in JSON/XML - depending on the ``format`` parameter).
838

    
839
Hashmaps should be formatted as outlined in ``GET``.
840

    
841
==========================  ===============================
842
Reply Header Name           Value
843
==========================  ===============================
844
ETag                        The MD5 hash of the object
845
X-Object-Version            The object's new version
846
==========================  ===============================
847

    
848
The ``X-Object-Sharing`` header may include either a ``read=...`` comma-separated user/group list, or a ``write=...`` comma-separated user/group list, or both separated by a semicolon (``;``). Groups are specified as ``<account>:<group>``. To publish the object, set ``X-Object-Public`` to ``true``. To unpublish, set to ``false``, or use an empty header value.
849

    
850
==============================  ==============================
851
Return Code                     Description
852
==============================  ==============================
853
201 (Created)                   The object has been created
854
409 (Conflict)                  The object can not be created from the provided hashmap (a list of missing hashes will be included in the reply)
855
411 (Length Required)           Missing ``Content-Length`` or ``Content-Type`` in the request
856
413 (Request Entity Too Large)  Insufficient quota to complete the request
857
422 (Unprocessable Entity)      The MD5 checksum of the data written to the storage system does not match the (optionally) supplied ETag value
858
==============================  ==============================
859

    
860

    
861
COPY
862
""""
863

    
864
====================  ================================
865
Request Header Name   Value
866
====================  ================================
867
If-Match              Proceed if ETags match with object
868
If-None-Match         Proceed if ETags don't match with object
869
Destination           The destination path in the form ``/<container>/<object>``
870
Destination-Account   The destination account to copy to
871
Content-Type          The MIME content type of the object (optional :sup:`*`)
872
Content-Encoding      The encoding of the object (optional)
873
Content-Disposition   The presentation style of the object (optional)
874
X-Source-Version      The source version to copy from
875
X-Object-Manifest     Object parts prefix in ``<container>/<object>`` form (optional)
876
X-Object-Sharing      Object permissions (optional)
877
X-Object-Public       Object is publicly accessible (optional)
878
X-Object-Meta-*       Optional user defined metadata
879
====================  ================================
880

    
881
:sup:`*` *When using django locally with the supplied web server, use the ignore_content_type parameter, or do provide a valid Content-Type, as a type of text/plain is applied by default to all requests. Client software should always state ignore_content_type, except when a Content-Type is explicitly defined by the user.*
882

    
883
======================  ===================================
884
Request Parameter Name  Value
885
======================  ===================================
886
format                  Optional conflict response type (can be ``json`` or ``xml``)
887
ignore_content_type     Ignore the supplied Content-Type
888
======================  ===================================
889

    
890
Refer to ``PUT``/``POST`` for a description of request headers. Metadata is also copied, updated with any values defined. Sharing/publishing options are not copied.
891

    
892
==========================  ===============================
893
Reply Header Name           Value
894
==========================  ===============================
895
X-Object-Version            The object's new version
896
==========================  ===============================
897

    
898
|
899

    
900
==============================  ==============================
901
Return Code                     Description
902
==============================  ==============================
903
201 (Created)                   The object has been created
904
413 (Request Entity Too Large)  Insufficient quota to complete the request
905
==============================  ==============================
906

    
907

    
908
MOVE
909
""""
910

    
911
Same as ``COPY``, without the ``X-Source-Version`` request header. The ``MOVE`` operation is always applied on the latest version.
912

    
913

    
914
POST
915
""""
916

    
917
====================  ================================
918
Request Header Name   Value
919
====================  ================================
920
If-Match              Proceed if ETags match with object
921
If-None-Match         Proceed if ETags don't match with object
922
Content-Length        The size of the data written (optional, to update)
923
Content-Type          The MIME content type of the object (optional, to update)
924
Content-Range         The range of data supplied (optional, to update)
925
Transfer-Encoding     Set to ``chunked`` to specify incremental uploading (if used, ``Content-Length`` is ignored)
926
Content-Encoding      The encoding of the object (optional)
927
Content-Disposition   The presentation style of the object (optional)
928
X-Source-Object       Update with data from the object at path ``/<container>/<object>`` (optional, to update)
929
X-Source-Account      The source account to update from
930
X-Source-Version      The source version to update from (optional, to update)
931
X-Object-Bytes        The updated object's final size (optional, when updating)
932
X-Object-Manifest     Object parts prefix in ``<container>/<object>`` form (optional)
933
X-Object-Sharing      Object permissions (optional)
934
X-Object-Public       Object is publicly accessible (optional)
935
X-Object-Meta-*       Optional user defined metadata
936
====================  ================================
937

    
938
|
939

    
940
======================  ============================================
941
Request Parameter Name  Value
942
======================  ============================================
943
format                  Optional conflict response type (can be ``json`` or ``xml``)
944
update                  Do not replace metadata (no value parameter)
945
======================  ============================================
946

    
947
The ``Content-Encoding``, ``Content-Disposition``, ``X-Object-Manifest`` and ``X-Object-Meta-*`` headers are considered to be user defined metadata. An operation without the ``update`` parameter will overwrite all previous values and remove any keys not supplied. When using ``update`` any metadata with an empty value will be deleted.
948

    
949
To change permissions, include an ``X-Object-Sharing`` header (as defined in ``PUT``). To publish, include an ``X-Object-Public`` header, with a value of ``true``. If no such headers are defined, no changes will be applied to sharing/public. Use empty values to remove permissions/unpublish (unpublishing also works with ``false`` as a header value). Sharing options are applied to the object - not its versions.
950

    
951
To update an object's data:
952

    
953
* Either set ``Content-Type`` to ``application/octet-stream``, or provide an object with ``X-Source-Object``. If ``Content-Type`` has some other value, it will be ignored and only the metadata will be updated.
954
* If the data is supplied in the request (using ``Content-Type`` instead of ``X-Source-Object``), a valid ``Content-Length`` header is required - except if using chunked transfers (set ``Transfer-Encoding`` to ``chunked``).
955
* Set ``Content-Range`` as specified in RFC2616, with the following differences:
956

    
957
  * Client software MAY omit ``last-byte-pos`` of if the length of the range being transferred is unknown or difficult to determine.
958
  * Client software SHOULD not specify the ``instance-length`` (use a ``*``), unless there is a reason for performing a size check at the server.
959
* If ``Content-Range`` used has a ``byte-range-resp-spec = *``, data will be appended to the object.
960

    
961
Optionally, truncate the updated object to the desired length with the ``X-Object-Bytes`` header.
962

    
963
A data update will trigger an ETag change. Updated ETags may happen asynchronously and appear at the server with a delay.
964

    
965
No reply content. No reply headers if only metadata is updated.
966

    
967
==========================  ===============================
968
Reply Header Name           Value
969
==========================  ===============================
970
ETag                        The new ETag of the object (data updated)
971
X-Object-Version            The object's new version
972
==========================  ===============================
973

    
974
|
975

    
976
==============================  ==============================
977
Return Code                     Description
978
==============================  ==============================
979
202 (Accepted)                  The request has been accepted (not a data update)
980
204 (No Content)                The request succeeded (data updated)
981
411 (Length Required)           Missing ``Content-Length`` in the request
982
413 (Request Entity Too Large)  Insufficient quota to complete the request
983
416 (Range Not Satisfiable)     The supplied range is invalid
984
==============================  ==============================
985

    
986
The ``POST`` method can also be used for creating an object via a standard HTML form. If the request ``Content-Type`` is ``multipart/form-data``, none of the above headers will be processed. The form should have an ``X-Object-Data`` field, as in the following example. The token is passed as a request parameter. ::
987

    
988
  <form method="post" action="https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt?X-Auth-Token=0000" enctype="multipart/form-data">
989
    <input type="file" name="X-Object-Data">
990
    <input type="submit">
991
  </form>
992

    
993
This will create/override the object with the given name, as if using ``PUT``. The ``Content-Type`` of the object will be set to the value of the corresponding header sent in the part of the request containing the data (usually, automatically handled by the browser). Metadata, sharing and other object attributes can not be set this way. The response will contain the object's ETag.
994

    
995
==========================  ===============================
996
Reply Header Name           Value
997
==========================  ===============================
998
ETag                        The MD5 hash of the object
999
X-Object-Version            The object's new version
1000
==========================  ===============================
1001

    
1002
|
1003

    
1004
==============================  ==============================
1005
Return Code                     Description
1006
==============================  ==============================
1007
201 (Created)                   The object has been created
1008
413 (Request Entity Too Large)  Insufficient quota to complete the request
1009
==============================  ==============================
1010

    
1011

    
1012
DELETE
1013
""""""
1014

    
1015
======================  ===================================
1016
Request Parameter Name  Value
1017
======================  ===================================
1018
until                   Optional timestamp
1019
======================  ===================================
1020

    
1021
If ``until`` is defined, the object is "purged" up to that time (the history up to then is deleted).
1022

    
1023
No reply content/headers.
1024

    
1025
===========================  ==============================
1026
Return Code                  Description
1027
===========================  ==============================
1028
204 (No Content)             The request succeeded
1029
===========================  ==============================
1030

    
1031
Sharing and Public Objects
1032
^^^^^^^^^^^^^^^^^^^^^^^^^^
1033

    
1034
Read and write control in Pithos is managed by setting appropriate permissions with the ``X-Object-Sharing`` header. The permissions are applied using directory-based inheritance. A directory is an object with the corresponding content type. The default delimiter is ``/``. Thus, each set of authorization directives is applied to all objects in the directory object where the corresponding ``X-Object-Sharing`` header is defined. If there are nested/overlapping permissions, the closest to the object is applied. When retrieving an object, the ``X-Object-Shared-By`` header reports where it gets its permissions from. If not present, the object is the actual source of authorization directives.
1035

    
1036
A user may ``GET`` another account or container. The result will include a limited reply, containing only the allowed containers or objects respectively. A top-level request with an authentication token, will return a list of allowed accounts, so the user can easily find out which other users share objects. The ``X-Object-Allowed-To`` header lists the actions allowed on an object, if it does not belong to the requesting user.
1037

    
1038
Objects that are marked as public, via the ``X-Object-Public`` meta, are also available at the corresponding URI returned for ``HEAD`` or ``GET``. Requests for public objects do not need to include an ``X-Auth-Token``. Pithos will ignore request parameters and only include the following headers in the reply (all ``X-Object-*`` meta is hidden):
1039

    
1040
==========================  ===============================
1041
Reply Header Name           Value
1042
==========================  ===============================
1043
ETag                        The ETag of the object
1044
Content-Length              The size of the data returned
1045
Content-Type                The MIME content type of the object
1046
Content-Range               The range of data included (only on a single range request)
1047
Last-Modified               The last object modification date (regardless of version)
1048
Content-Encoding            The encoding of the object (optional)
1049
Content-Disposition         The presentation style of the object (optional)
1050
==========================  ===============================
1051

    
1052
Public objects are not included and do not influence cross-user listings. They are, however, readable by all users.
1053

    
1054
Summary
1055
^^^^^^^
1056

    
1057
List of differences from the OOS API:
1058

    
1059
* Support for ``X-Account-Meta-*`` style headers at the account level. Use ``POST`` to update.
1060
* Support for ``X-Container-Meta-*`` style headers at the container level. Can be set when creating via ``PUT``. Use ``POST`` to update.
1061
* Header ``X-Container-Object-Meta`` at the container level and parameter ``meta`` in container listings. (**TBD**)
1062
* Account and container policies to manage behavior and limits. Container behavior overrides account settings. Account quota sets the maximum bytes limit, regardless of container values.
1063
* Headers ``X-Container-Block-*`` at the container level, exposing the underlying storage characteristics.
1064
* All metadata replies, at all levels, include latest modification information.
1065
* At all levels, a ``HEAD`` or ``GET`` request may use ``If-Modified-Since`` and ``If-Unmodified-Since`` headers.
1066
* Container/object lists include more fields if the reply is of type JSON/XML. Some names are kept to their OOS API equivalents for compatibility.
1067
* Option to include only shared containers/objects in listings.
1068
* Object metadata allowed, in addition to ``X-Object-Meta-*``: ``Content-Encoding``, ``Content-Disposition``, ``X-Object-Manifest``. These are all replaced with every update operation, except if using the ``update`` parameter (in which case individual keys can also be deleted). Deleting meta by providing empty values also works when copying/moving an object.
1069
* Multi-range object ``GET`` support as outlined in RFC2616.
1070
* Object hashmap retrieval through ``GET`` and the ``format`` parameter.
1071
* Object create via hashmap through ``PUT`` and the ``format`` parameter.
1072
* The object's Merkle hash is always returned in the ``X-Object-Hash`` header.
1073
* The object's UUID is always returned in the ``X-Object-UUID`` header. The UUID remains unchanged, even when the object's data or metadata changes, or the object is moved to another path (is renamed). A new UUID is assigned when creating or copying an object.
1074
* Object create using ``POST`` to support standard HTML forms.
1075
* Partial object updates through ``POST``, using the ``Content-Length``, ``Content-Type``, ``Content-Range`` and ``Transfer-Encoding`` headers. Use another object's data to update with ``X-Source-Object`` and ``X-Source-Version``. Truncate with ``X-Object-Bytes``.
1076
* Include new version identifier in replies for object replace/change requests.
1077
* Object ``MOVE`` support and ``ignore_content_type`` parameter in both ``COPY`` and ``MOVE``.
1078
* Conditional object create/update operations, using ``If-Match`` and ``If-None-Match`` headers.
1079
* Time-variant account/container listings via the ``until`` parameter.
1080
* Object versions - parameter ``version`` in ``HEAD``/``GET`` (list versions with ``GET``), ``X-Object-Version-*`` meta in replies, ``X-Source-Version`` in ``PUT``/``COPY``.
1081
* Sharing/publishing with ``X-Object-Sharing``, ``X-Object-Public`` at the object level. Cross-user operations are allowed - controlled by sharing directives. Available actions in cross-user requests are reported with ``X-Object-Allowed-To``. Permissions may include groups defined with ``X-Account-Group-*`` at the account level. These apply to the object - not its versions.
1082
* Support for directory-based inheritance when enforcing permissions. Parent object carrying the authorization directives is reported in ``X-Object-Shared-By``.
1083
* Copy and move between accounts with ``X-Source-Account`` and ``Destination-Account`` headers.
1084
* Large object support with ``X-Object-Manifest``.
1085
* Trace the user that created/modified an object with ``X-Object-Modified-By``.
1086
* Purge container/object history with the ``until`` parameter in ``DELETE``.
1087

    
1088
Clarifications/suggestions:
1089

    
1090
* All non-ASCII characters in headers should be URL-encoded.
1091
* Authentication is done by another system. The token is used in the same way, but it is obtained differently. The top level ``GET`` request is kept compatible with the OOS API and allows for guest/testing operations.
1092
* Some processing is done in the variable part of all ``X-*-Meta-*`` headers. If it includes underscores, they will be converted to dashes and the first letter of all intra-dash strings will be capitalized.
1093
* A ``GET`` reply for a level will include all headers of the corresponding ``HEAD`` request.
1094
* To avoid conflicts between objects and virtual directory markers in container listings, it is recommended that object names do not end with the delimiter used.
1095
* The ``Accept`` header may be used in requests instead of the ``format`` parameter to specify the desired request/reply format. The parameter overrides the header.
1096
* Container/object lists use a ``200`` return code if the reply is of type JSON/XML. The reply will include an empty JSON/XML.
1097
* In headers, dates are formatted according to RFC 1123. In extended information listings, the ``last_modified`` field is formatted according to ISO 8601 (for OOS API compatibility). All other fields (Pithos extensions) use integer tiemstamps.
1098
* The ``Last-Modified`` header value always reflects the actual latest change timestamp, regardless of time control parameters and version requests. Time precondition checks with ``If-Modified-Since`` and ``If-Unmodified-Since`` headers are applied to this value.
1099
* A copy/move using ``PUT``/``COPY``/``MOVE`` will always update metadata, keeping all old values except the ones redefined in the request headers.
1100
* A ``HEAD`` or ``GET`` for an ``X-Object-Manifest`` object, will include modified ``Content-Length`` and ``ETag`` headers, according to the characteristics of the objects under the specified prefix. The ``Etag`` will be the MD5 hash of the corresponding ETags concatenated. In extended container listings there is no metadata processing.
1101

    
1102
The Pithos Client
1103
-----------------
1104

    
1105
User Experience
1106
^^^^^^^^^^^^^^^
1107

    
1108
Hopefully this API will allow for a multitude of client implementations, each supporting a different device or operating system. All clients will be able to manipulate containers and objects - even software only designed for OOS API compatibility. But a Pithos interface should not be only about showing containers and folders. There are some extra user interface elements and functionalities that should be common to all implementations.
1109

    
1110
Upon entrance to the service, a user is presented with the following elements - which can be represented as folders or with other related icons:
1111

    
1112
* The ``home`` element, which is used as the default entry point to the user's "files". Objects under ``home`` are represented in the usual hierarchical organization of folders and files.
1113
* The ``trash`` element, which contains files that have been marked for deletion, but can still be recovered.
1114
* The ``shared`` element, which contains all objects shared by the user to other users of the system.
1115
* The ``others`` element, which contains all objects that other users share with the user.
1116
* The ``groups`` element, which contains the names of groups the user has defined. Each group consists of a user list. Group creation, deletion, and manipulation is carried out by actions originating here.
1117
* The ``history`` element, which allows browsing past instances of ``home`` and - optionally - ``trash``.
1118

    
1119
Objects in Pithos can be:
1120

    
1121
* Moved to trash and then deleted.
1122
* Shared with specific permissions.
1123
* Made public (shared with non-Pithos users).
1124
* Restored from previous versions.
1125

    
1126
Some of these functions are performed by the client software and some by the Pithos server.
1127

    
1128
In the first version of Pithos, objects could also be assigned custom tags. This is no longer supported. Existing deployments can migrate tags into a specific metadata value, i.e. ``X-Object-Meta-Tags``.
1129

    
1130
Implementation Guidelines
1131
^^^^^^^^^^^^^^^^^^^^^^^^^
1132

    
1133
Pithos clients should use the ``pithos`` and ``trash`` containers for active and inactive objects respectively. If any of these containers is not found, the client software should create it, without interrupting the user's workflow. The ``home`` element corresponds to ``pithos`` and the ``trash`` element to ``trash``. Use ``PUT`` with the ``X-Move-From`` header, or ``MOVE`` to transfer objects from one container to the other. Use ``DELETE`` to remove from ``pithos`` without trashing, or to remove from ``trash``. When moving objects, detect naming conflicts with the ``If-Match`` or ``If-None-Match`` headers. Such conflicts should be resolved by the user.
1134

    
1135
Object names should use the ``/`` delimiter to impose a hierarchy of folders and files.
1136

    
1137
The ``shared`` element should be implemented as a read-only view of the ``pithos`` container, using the ``shared`` parameter when listing objects. The ``others`` element, should start with a top-level ``GET`` to retrieve the list of accounts accessible to the user. It is suggested that the client software hides the next step of navigation - the container - if it only includes ``pithos`` and forwards the user directly to the objects.
1138

    
1139
Public objects are not included in ``shared`` and ``others`` listings. It is suggested that they are marked in a visually distinctive way in ``pithos`` listings (for example using an icon overlay).
1140

    
1141
A special application menu, or a section in application preferences, should be devoted to managing groups (the ``groups`` element). All group-related actions are implemented at the account level.
1142

    
1143
Browsing past versions of objects should be available both at the object and the container level. At the object level, a list of past versions can be included in the screen showing details or more information on the object (metadata, permissions, etc.). At the container level, it is suggested that clients use a ``history`` element, which presents to the user a read-only, time-variable view of ``pithos`` contents. This can be accomplished via the ``until`` parameter in listings. Optionally, ``history`` may include ``trash``.
1144

    
1145
Uploading and downloading data
1146
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1147

    
1148
By using hashmaps to upload and download objects the corresponding operations can complete much faster.
1149

    
1150
In the case of an upload, only the missing blocks will be submitted to the server:
1151

    
1152
* Calculate the hash value for each block of the object to be uploaded. Use the hash algorithm and block size of the destination container.
1153
* Send a hashmap ``PUT`` request for the object.
1154

    
1155
  * Server responds with status ``201`` (Created):
1156

    
1157
    * Blocks are already on the server. The object has been created. Done.
1158

    
1159
  * Server responds with status ``409`` (Conflict):
1160

    
1161
    * Server's response body contains the hashes of the blocks that do not exist on the server.
1162
    * For each hash value in the server's response (or all hashes together):
1163

    
1164
      * Send a ``POST`` request to the destination container with the corresponding data.
1165

    
1166
* Repeat hashmap ``PUT``. Fail if the server's response is not ``201``.
1167

    
1168
Consulting hashmaps when downloading allows for resuming partially transferred objects. The client should retrieve the hashmap from the server and compare it with the hashmap computed from the respective local file. Any missing parts can be downloaded with ``GET`` requests with the additional ``Range`` header.
1169

    
1170
Syncing
1171
^^^^^^^
1172

    
1173
Consider the following algorithm for synchronizing a local folder with the server. The "state" is the complete object listing, with the corresponding attributes.
1174
 
1175
::
1176

    
1177
  L: local state (stored state from last sync with the server)
1178
  C: current state (state computed right before sync)
1179
  S: server state
1180

    
1181
  if C == L:
1182
      # No local changes
1183
      if S == L:
1184
          # No remote changes, nothing to do
1185
      else:
1186
          # Update local state to match that of the server
1187
         L = S
1188
  else:
1189
      # We have local changes
1190
      if S == L:
1191
          # No remote changes, update the server
1192
          S = C
1193
          L = S
1194
      else:
1195
          # Both we and server have changes
1196
          if C == S:
1197
              # We were lucky, we did the same change
1198
              L = S
1199
          else:
1200
              # We have conflicting changes
1201
              resolve conflict
1202

    
1203
Notes:
1204

    
1205
* States represent file hashes (it is suggested to use Merkle). Deleted or non-existing files are assumed to have a magic hash (e.g. empty string).
1206
* Updating a state (either local or remote) implies downloading, uploading or deleting the appropriate file.
1207

    
1208
Recommended Practices and Examples
1209
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1210

    
1211
Assuming an authentication token is obtained, the following high-level operations are available - shown with ``curl``:
1212

    
1213
* Get account information ::
1214

    
1215
    curl -X HEAD -D - \
1216
         -H "X-Auth-Token: 0000" \
1217
         https://pithos.dev.grnet.gr/v1/user
1218

    
1219
* List available containers ::
1220

    
1221
    curl -X GET -D - \
1222
         -H "X-Auth-Token: 0000" \
1223
         https://pithos.dev.grnet.gr/v1/user
1224

    
1225
* Get container information ::
1226

    
1227
    curl -X HEAD -D - \
1228
         -H "X-Auth-Token: 0000" \
1229
         https://pithos.dev.grnet.gr/v1/user/pithos
1230

    
1231
* Add a new container ::
1232

    
1233
    curl -X PUT -D - \
1234
         -H "X-Auth-Token: 0000" \
1235
         https://pithos.dev.grnet.gr/v1/user/test
1236

    
1237
* Delete a container ::
1238

    
1239
    curl -X DELETE -D - \
1240
         -H "X-Auth-Token: 0000" \
1241
         https://pithos.dev.grnet.gr/v1/user/test
1242

    
1243
* List objects in a container ::
1244

    
1245
    curl -X GET -D - \
1246
         -H "X-Auth-Token: 0000" \
1247
         https://pithos.dev.grnet.gr/v1/user/pithos
1248

    
1249
* List objects in a container (extended reply) ::
1250

    
1251
    curl -X GET -D - \
1252
         -H "X-Auth-Token: 0000" \
1253
         https://pithos.dev.grnet.gr/v1/user/pithos?format=json
1254

    
1255
  It is recommended that extended replies are cached and subsequent requests utilize the ``If-Modified-Since`` header.
1256

    
1257
* List metadata keys used by objects in a container
1258

    
1259
  Will be in the ``X-Container-Object-Meta`` reply header, included in container information or object list (``HEAD`` or ``GET``). (**TBD**)
1260

    
1261
* List objects in a container having a specific meta defined ::
1262

    
1263
    curl -X GET -D - \
1264
         -H "X-Auth-Token: 0000" \
1265
         https://pithos.dev.grnet.gr/v1/user/pithos?meta=favorites
1266

    
1267
* Retrieve an object ::
1268

    
1269
    curl -X GET -D - \
1270
         -H "X-Auth-Token: 0000" \
1271
         https://pithos.dev.grnet.gr/v1/user/pithos/README.txt
1272

    
1273
* Retrieve an object (specific ranges of data) ::
1274

    
1275
    curl -X GET -D - \
1276
         -H "X-Auth-Token: 0000" \
1277
         -H "Range: bytes=0-9" \
1278
         https://pithos.dev.grnet.gr/v1/user/pithos/README.txt
1279

    
1280
  This will return the first 10 bytes. To get the first 10, bytes 30-39 and the last 100 use ``Range: bytes=0-9,30-39,-100``.
1281

    
1282
* Add a new object (folder type) (**TBD**) ::
1283

    
1284
    curl -X PUT -D - \
1285
         -H "X-Auth-Token: 0000" \
1286
         -H "Content-Type: application/directory" \
1287
         https://pithos.dev.grnet.gr/v1/user/pithos/folder
1288

    
1289
* Add a new object ::
1290

    
1291
    curl -X PUT -D - \
1292
         -H "X-Auth-Token: 0000" \
1293
         -H "Content-Type: text/plain" \
1294
         -T EXAMPLE.txt
1295
         https://pithos.dev.grnet.gr/v1/user/pithos/folder/EXAMPLE.txt
1296

    
1297
* Update an object ::
1298

    
1299
    curl -X POST -D - \
1300
         -H "X-Auth-Token: 0000" \
1301
         -H "Content-Length: 10" \
1302
         -H "Content-Type: application/octet-stream" \
1303
         -H "Content-Range: bytes 10-19/*" \
1304
         -d "0123456789" \
1305
         https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt
1306

    
1307
  This will update bytes 10-19 with the data specified.
1308

    
1309
* Update an object (append) ::
1310

    
1311
    curl -X POST -D - \
1312
         -H "X-Auth-Token: 0000" \
1313
         -H "Content-Length: 10" \
1314
         -H "Content-Type: application/octet-stream" \
1315
         -H "Content-Range: bytes */*" \
1316
         -d "0123456789" \
1317
         https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt
1318

    
1319
* Update an object (truncate) ::
1320

    
1321
    curl -X POST -D - \
1322
         -H "X-Auth-Token: 0000" \
1323
         -H "X-Source-Object: /folder/EXAMPLE.txt" \
1324
         -H "Content-Range: bytes 0-0/*" \
1325
         -H "X-Object-Bytes: 0" \
1326
         https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt
1327

    
1328
  This will truncate the object to 0 bytes.
1329

    
1330
* Add object metadata ::
1331

    
1332
    curl -X POST -D - \
1333
         -H "X-Auth-Token: 0000" \
1334
         -H "X-Object-Meta-First: first_meta_value" \
1335
         -H "X-Object-Meta-Second: second_meta_value" \
1336
         https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt
1337

    
1338
* Delete object metadata ::
1339

    
1340
    curl -X POST -D - \
1341
         -H "X-Auth-Token: 0000" \
1342
         -H "X-Object-Meta-First: first_meta_value" \
1343
         https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt
1344

    
1345
  Metadata can only be "set". To delete ``X-Object-Meta-Second``, reset all metadata.
1346

    
1347
* Delete an object ::
1348

    
1349
    curl -X DELETE -D - \
1350
         -H "X-Auth-Token: 0000" \
1351
         https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt