Statistics
| Branch: | Tag: | Revision:

root / docs / source / devguide.rst @ 4e5673a0

History | View | Annotate | Download (68.1 kB)

1
Pithos v2 Developer Guide
2
=========================
3

    
4
Introduction
5
------------
6

    
7
Pithos is a storage service implemented by GRNET (http://www.grnet.gr). Data is stored as objects, organized in containers, belonging to an account. This hierarchy of storage layers has been inspired by the OpenStack Object Storage (OOS) API and similar CloudFiles API by Rackspace. The Pithos API follows the OOS API as closely as possible. One of the design requirements has been to be able to use Pithos with clients built for the OOS, without changes.
8

    
9
However, to be able to take full advantage of the Pithos infrastructure, client software should be aware of the extensions that differentiate Pithos from OOS. Pithos objects can be updated, or appended to. Pithos will store sharing permissions per object and enforce corresponding authorization policies. Automatic version management, allows taking account and container listings back in time, as well as reading previous instances of objects.
10

    
11
The storage backend of Pithos is block oriented, permitting efficient, deduplicated data placement. The block structure of objects is exposed at the API layer, in order to encourage external software to implement advanced data management operations.
12

    
13
This document's goals are:
14

    
15
* Define the Pithos ReST API that allows the storage and retrieval of data and metadata via HTTP calls
16
* Specify metadata semantics and user interface guidelines for a common experience across client software implementations
17

    
18
The present document is meant to be read alongside the OOS API documentation. Thus, it is suggested that the reader is familiar with associated technologies, the OOS API as well as the first version of the Pithos API. This document refers to the second version of Pithos. Information on the first version of the storage API can be found at http://code.google.com/p/gss.
19

    
20
Whatever marked as to be determined (**TBD**), should not be considered by implementors.
21

    
22
Document Revisions
23
^^^^^^^^^^^^^^^^^^
24

    
25
=========================  ================================
26
Revision                   Description
27
=========================  ================================
28
0.8 (Dec 15, 2011)         Update allowed versioning values.
29
\                          Change policy/meta formatting in JSON/XML replies.
30
\                          Document that all non-ASCII characters in headers should be URL-encoded.
31
\                          Support metadata-based queries when listing objects at the container level.
32
0.7 (Nov 21, 2011)         Suggest upload/download methods using hashmaps.
33
\                          Propose syncing algorithm.
34
\                          Support cross-account object copy and move.
35
\                          Pass token as a request parameter when using ``POST`` via an HTML form.
36
\                          Optionally use source account to update object from another object.
37
\                          Use container ``POST`` to upload missing blocks of data.
38
\                          Report policy in account headers.
39
\                          Add insufficient quota reply.
40
\                          Use special meta to always report Merkle hash.
41
0.6 (Sept 13, 2011)        Reply with Merkle hash as the ETag when updating objects.
42
\                          Include version id in object replace/change replies.
43
\                          Change conflict (409) replies format to text.
44
\                          Tags should be migrated to a meta value.
45
\                          Container ``PUT`` updates metadata/policy.
46
\                          Report allowed actions in shared object replies.
47
\                          Provide ``https://hostname/login`` for Shibboleth authentication.
48
\                          Use ``hashmap`` parameter in object ``GET``/``PUT`` to use hashmaps.
49
0.5 (July 22, 2011)        Object update from another object's data.
50
\                          Support object truncate.
51
\                          Create object using a standard HTML form.
52
\                          Purge container/object history.
53
\                          List other accounts that share objects with a user.
54
\                          List shared containers/objects.
55
\                          Update implementation guidelines.
56
\                          Check preconditions when creating/updating objects.
57
0.4 (July 01, 2011)        Object permissions and account groups.
58
\                          Control versioning behavior and container quotas with container policy directives.
59
\                          Support updating/deleting individual metadata with ``POST``.
60
\                          Create object using hashmap.
61
0.3 (June 14, 2011)        Large object support with ``X-Object-Manifest``.
62
\                          Allow for publicly available objects via ``https://hostname/public``.
63
\                          Support time-variant account/container listings. 
64
\                          Add source version when duplicating with ``PUT``/``COPY``.
65
\                          Request version in object ``HEAD``/``GET`` requests (list versions with ``GET``).
66
0.2 (May 31, 2011)         Add object meta listing and filtering in containers.
67
\                          Include underlying storage characteristics in container meta.
68
\                          Support for partial object updates through ``POST``.
69
\                          Expose object hashmaps through ``GET``.
70
\                          Support for multi-range object ``GET`` requests.
71
0.1 (May 17, 2011)         Initial release. Based on OpenStack Object Storage Developer Guide API v1 (Apr. 15, 2011).
72
=========================  ================================
73

    
74
Pithos Users and Authentication
75
-------------------------------
76

    
77
Pithos keeps separate databases for users and objects.
78

    
79
Each user is uniquely identified by the ``Uniq`` field. This should be used as the user's account in the API. The API uses the ``Token`` field to authenticate a user, thus allowing cross-account requests. All API requests require a token.
80

    
81
User entries can be modified/added via the management interface available at ``https://hostname/admin``.
82

    
83
Pithos is also compatible with Shibboleth (http://shibboleth.internet2.edu/). The connection between Shibboleth and Pithos is done by ``https://hostname/login``. An application that wishes to connect to Pithos, but does not have a token, should redirect the user to the login URI.
84

    
85
The login URI accepts the following parameters:
86

    
87
======================  =========================
88
Request Parameter Name  Value
89
======================  =========================
90
next                    The URI to redirect to when the process is finished
91
renew                   Force token renewal (no value parameter)
92
======================  =========================
93

    
94
The login process starts by redirecting the user to an external URI (controlled by Shibboleth), where the actual authentication credentials are entered. Then, the user is redirected back to the login URI from Shibboleth, with various identification information in the request headers.
95

    
96
If the user does not exist in the database, Pithos adds the user and creates a random token. If the user exists, the token has not expired and ``renew`` is not set, the existing token is reused. Finally, the login URI redirects to the URI provided with ``next``, adding the ``user`` and ``token`` parameters, which contain the ``Uniq`` and ``Token`` fields respectively. 
97

    
98
The Pithos API
99
--------------
100

    
101
The URI requests supported by the Pithos API follow one of the following forms:
102

    
103
* Top level: ``https://hostname/v1/``
104
* Account level: ``https://hostname/v1/<account>``
105
* Container level: ``https://hostname/v1/<account>/<container>``
106
* Object level: ``https://hostname/v1/<account>/<container>/<object>``
107

    
108
All requests must include an ``X-Auth-Token`` - as a header, or a parameter.
109

    
110
The allowable request operations and respective return codes per level are presented in the remainder of this chapter. Common to all requests are the following return codes.
111

    
112
=========================  ================================
113
Return Code                Description
114
=========================  ================================
115
400 (Bad Request)          The request is invalid
116
401 (Unauthorized)         Missing or invalid token
117
403 (Forbidden)            Request not allowed
118
404 (Not Found)            The requested resource was not found
119
503 (Service Unavailable)  The request cannot be completed because of an internal error
120
=========================  ================================
121

    
122
Top Level
123
^^^^^^^^^
124

    
125
List of operations:
126

    
127
=========  ==================
128
Operation  Description
129
=========  ==================
130
GET        Authentication (for compatibility with the OOS API) or list allowed accounts
131
=========  ==================
132

    
133
GET
134
"""
135

    
136
If the ``X-Auth-User`` and ``X-Auth-Key`` headers are given, a dummy ``X-Auth-Token`` and ``X-Storage-Url`` will be replied, which can be used as a guest token/namespace for testing Pithos.
137

    
138
================  =====================
139
Return Code       Description
140
================  =====================
141
204 (No Content)  The request succeeded
142
================  =====================
143

    
144
If an ``X-Auth-Token`` is already present, the operation will be interpreted as a request to list other accounts that share objects to the user.
145

    
146
======================  =========================
147
Request Parameter Name  Value
148
======================  =========================
149
limit                   The amount of results requested (default is 10000)
150
marker                  Return containers with name lexicographically after marker
151
format                  Optional extended reply type (can be ``json`` or ``xml``)
152
======================  =========================
153

    
154
The reply is a list of account names.
155
If a ``format=xml`` or ``format=json`` argument is given, extended information on the accounts will be returned, serialized in the chosen format.
156
For each account, the information will include the following (names will be in lower case and with hyphens replaced with underscores):
157

    
158
===========================  ============================
159
Name                         Description
160
===========================  ============================
161
name                         The name of the account
162
last_modified                The last account modification date (regardless of ``until``)
163
===========================  ============================
164

    
165
Example ``format=json`` reply:
166

    
167
::
168

    
169
  [{"name": "user", "last_modified": "2011-12-02T08:10:41.565891+00:00"}, ...]
170

    
171
Example ``format=xml`` reply:
172

    
173
::
174

    
175
  <?xml version="1.0" encoding="UTF-8"?>
176
  <accounts>
177
    <account>
178
      <name>user</name>
179
      <last_modified>2011-12-02T08:10:41.565891+00:00</last_modified>
180
    </account>
181
    <account>...</account>
182
  </accounts>
183

    
184
===========================  =====================
185
Return Code                  Description
186
===========================  =====================
187
200 (OK)                     The request succeeded
188
204 (No Content)             The user has no access to other accounts (only for non-extended replies)
189
===========================  =====================
190

    
191
Will use a ``200`` return code if the reply is of type JSON/XML.
192

    
193
Account Level
194
^^^^^^^^^^^^^
195

    
196
List of operations:
197

    
198
=========  ==================
199
Operation  Description
200
=========  ==================
201
HEAD       Retrieve account metadata
202
GET        List containers
203
POST       Update account metadata
204
=========  ==================
205

    
206
HEAD
207
""""
208

    
209
====================  ===========================
210
Request Header Name   Value
211
====================  ===========================
212
If-Modified-Since     Retrieve if account has changed since provided timestamp
213
If-Unmodified-Since   Retrieve if account has not changed since provided timestamp
214
====================  ===========================
215

    
216
|
217

    
218
======================  ===================================
219
Request Parameter Name  Value
220
======================  ===================================
221
until                   Optional timestamp
222
======================  ===================================
223

    
224
Cross-user requests are not allowed to use ``until`` and only include the account modification date in the reply.
225

    
226
==========================  =====================
227
Reply Header Name           Value
228
==========================  =====================
229
X-Account-Container-Count   The total number of containers
230
X-Account-Bytes-Used        The total number of bytes stored
231
X-Account-Until-Timestamp   The last account modification date until the timestamp provided
232
X-Account-Group-*           Optional user defined groups
233
X-Account-Policy-*          Account behavior and limits
234
X-Account-Meta-*            Optional user defined metadata
235
Last-Modified               The last account modification date (regardless of ``until``)
236
==========================  =====================
237

    
238
|
239

    
240
================  =====================
241
Return Code       Description
242
================  =====================
243
204 (No Content)  The request succeeded
244
================  =====================
245

    
246

    
247
GET
248
"""
249

    
250
====================  ===========================
251
Request Header Name   Value
252
====================  ===========================
253
If-Modified-Since     Retrieve if account has changed since provided timestamp
254
If-Unmodified-Since   Retrieve if account has not changed since provided timestamp
255
====================  ===========================
256

    
257
|
258

    
259
======================  =========================
260
Request Parameter Name  Value
261
======================  =========================
262
limit                   The amount of results requested (default is 10000)
263
marker                  Return containers with name lexicographically after marker
264
format                  Optional extended reply type (can be ``json`` or ``xml``)
265
shared                  Show only shared containers (no value parameter)
266
until                   Optional timestamp
267
======================  =========================
268

    
269
The reply is a list of container names. Account headers (as in a ``HEAD`` request) will also be included.
270
Cross-user requests are not allowed to use ``until`` and only include the account/container modification dates in the reply.
271

    
272
If a ``format=xml`` or ``format=json`` argument is given, extended information on the containers will be returned, serialized in the chosen format.
273
For each container, the information will include all container metadata (names will be in lower case and with hyphens replaced with underscores):
274

    
275
===========================  ============================
276
Name                         Description
277
===========================  ============================
278
name                         The name of the container
279
count                        The number of objects inside the container
280
bytes                        The total size of the objects inside the container
281
last_modified                The last container modification date (regardless of ``until``)
282
x_container_until_timestamp  The last container modification date until the timestamp provided
283
x_container_policy_*         Container behavior and limits
284
x_container_meta_*           Optional user defined metadata
285
===========================  ============================
286

    
287
Example ``format=json`` reply:
288

    
289
::
290

    
291
  [{"name": "pithos",
292
    "bytes": 62452,
293
    "count": 8374,
294
    "last_modified": "2011-12-02T08:10:41.565891+00:00",
295
    "x_container_policy": {"quota": "53687091200", "versioning": "auto"},
296
    "x_container_meta": {"a": "b", "1": "2"}}, ...]
297

    
298
Example ``format=xml`` reply:
299

    
300
::
301

    
302
  <?xml version="1.0" encoding="UTF-8"?>
303
  <account name="user">
304
    <container>
305
      <name>pithos</name>
306
      <bytes>62452</bytes>
307
      <count>8374</count>
308
      <last_modified>2011-12-02T08:10:41.565891+00:00</last_modified>
309
      <x_container_policy>
310
        <key>quota</key><value>53687091200</value>
311
        <key>versioning</key><value>auto</value>
312
      </x_container_policy>
313
      <x_container_meta>
314
        <key>a</key><value>b</value>
315
        <key>1</key><value>2</value>
316
      </x_container_meta>
317
    </container>
318
    <container>...</container>
319
  </account>
320

    
321
For more examples of container details returned in JSON/XML formats refer to the OOS API documentation. In addition to the OOS API, Pithos returns all fields. Policy and metadata values are grouped and returned as key-value pairs.
322

    
323
===========================  =====================
324
Return Code                  Description
325
===========================  =====================
326
200 (OK)                     The request succeeded
327
204 (No Content)             The account has no containers (only for non-extended replies)
328
304 (Not Modified)           The account has not been modified
329
412 (Precondition Failed)    The condition set can not be satisfied
330
===========================  =====================
331

    
332
Will use a ``200`` return code if the reply is of type JSON/XML.
333

    
334

    
335
POST
336
""""
337

    
338
====================  ===========================
339
Request Header Name   Value
340
====================  ===========================
341
X-Account-Group-*     Optional user defined groups
342
X-Account-Meta-*      Optional user defined metadata
343
====================  ===========================
344

    
345
|
346

    
347
======================  ============================================
348
Request Parameter Name  Value
349
======================  ============================================
350
update                  Do not replace metadata/groups (no value parameter)
351
======================  ============================================
352

    
353
No reply content/headers.
354

    
355
The operation will overwrite all user defined metadata, except if ``update`` is defined.
356
To create a group, include an ``X-Account-Group-*`` header with the name in the key and a comma separated list of user names in the value. If no ``X-Account-Group-*`` header is present, no changes will be applied to groups. The ``update`` parameter also applies to groups. To delete a specific group, use ``update`` and an empty header value.
357

    
358
================  ===============================
359
Return Code       Description
360
================  ===============================
361
202 (Accepted)    The request has been accepted
362
================  ===============================
363

    
364

    
365
Container Level
366
^^^^^^^^^^^^^^^
367

    
368
List of operations:
369

    
370
=========  ============================
371
Operation  Description
372
=========  ============================
373
HEAD       Retrieve container metadata
374
GET        List objects
375
PUT        Create/update container
376
POST       Update container metadata
377
DELETE     Delete container
378
=========  ============================
379

    
380

    
381
HEAD
382
""""
383

    
384
====================  ===========================
385
Request Header Name   Value
386
====================  ===========================
387
If-Modified-Since     Retrieve if container has changed since provided timestamp
388
If-Unmodified-Since   Retrieve if container has not changed since provided timestamp
389
====================  ===========================
390

    
391
|
392

    
393
======================  ===================================
394
Request Parameter Name  Value
395
======================  ===================================
396
until                   Optional timestamp
397
======================  ===================================
398

    
399
Cross-user requests are not allowed to use ``until`` and only include the container modification date in the reply.
400

    
401
===========================  ===============================
402
Reply Header Name            Value
403
===========================  ===============================
404
X-Container-Object-Count     The total number of objects in the container
405
X-Container-Bytes-Used       The total number of bytes of all objects stored
406
X-Container-Block-Size       The block size used by the storage backend
407
X-Container-Block-Hash       The hash algorithm used for block identifiers in object hashmaps
408
X-Container-Until-Timestamp  The last container modification date until the timestamp provided
409
X-Container-Object-Meta      A list with all meta keys used by objects (**TBD**)
410
X-Container-Policy-*         Container behavior and limits
411
X-Container-Meta-*           Optional user defined metadata
412
Last-Modified                The last container modification date (regardless of ``until``)
413
===========================  ===============================
414

    
415
The keys returned in ``X-Container-Object-Meta`` are all the unique strings after the ``X-Object-Meta-`` prefix, formatted as a comma-separated list. See container ``PUT`` for a reference of policy directives. (**TBD**)
416

    
417
================  ===============================
418
Return Code       Description
419
================  ===============================
420
204 (No Content)  The request succeeded
421
================  ===============================
422

    
423

    
424
GET
425
"""
426

    
427
====================  ===========================
428
Request Header Name   Value
429
====================  ===========================
430
If-Modified-Since     Retrieve if container has changed since provided timestamp
431
If-Unmodified-Since   Retrieve if container has not changed since provided timestamp
432
====================  ===========================
433

    
434
|
435

    
436
======================  ===================================
437
Request Parameter Name  Value
438
======================  ===================================
439
limit                   The amount of results requested (default is 10000)
440
marker                  Return containers with name lexicographically after marker
441
prefix                  Return objects starting with prefix
442
delimiter               Return objects up to the delimiter (discussion follows)
443
path                    Assume ``prefix=path`` and ``delimiter=/``
444
format                  Optional extended reply type (can be ``json`` or ``xml``)
445
meta                    Return objects that satisfy the key queries in the specified comma separated list (use ``<key>``, ``!<key>`` for existence queries, ``<key><op><value>`` for value queries, where ``<op>`` can be one of ``=``, ``!=``, ``<=``, ``>=``, ``<``, ``>``)
446
shared                  Show only shared objects (no value parameter)
447
until                   Optional timestamp
448
======================  ===================================
449

    
450
The ``path`` parameter overrides ``prefix`` and ``delimiter``. When using ``path``, results will include objects ending in ``delimiter``.
451

    
452
The keys given with ``meta`` will be matched with the strings after the ``X-Object-Meta-`` prefix.
453

    
454
The reply is a list of object names. Container headers (as in a ``HEAD`` request) will also be included.
455
Cross-user requests are not allowed to use ``until`` and include the following limited set of headers in the reply:
456

    
457
===========================  ===============================
458
Reply Header Name            Value
459
===========================  ===============================
460
X-Container-Block-Size       The block size used by the storage backend
461
X-Container-Block-Hash       The hash algorithm used for block identifiers in object hashmaps
462
X-Container-Object-Meta      A list with all meta keys used by allowed objects (**TBD**)
463
Last-Modified                The last container modification date
464
===========================  ===============================
465

    
466
If a ``format=xml`` or ``format=json`` argument is given, extended information on the objects will be returned, serialized in the chosen format.
467
For each object, the information will include all object metadata (names will be in lower case and with hyphens replaced with underscores):
468

    
469
==========================  ======================================
470
Name                        Description
471
==========================  ======================================
472
name                        The name of the object
473
hash                        The ETag of the object
474
bytes                       The size of the object
475
content_type                The MIME content type of the object
476
content_encoding            The encoding of the object (optional)
477
content-disposition         The presentation style of the object (optional)
478
last_modified               The last object modification date (regardless of version)
479
x_object_hash               The Merkle hash
480
x_object_version            The object's version identifier
481
x_object_version_timestamp  The object's version timestamp
482
x_object_modified_by        The user that committed the object's version
483
x_object_manifest           Object parts prefix in ``<container>/<object>`` form (optional)
484
x_object_sharing            Object permissions (optional)
485
x_object_shared_by          Object inheriting permissions (optional)
486
x_object_allowed_to         Allowed actions on object (optional)
487
x_object_public             Object's publicly accessible URI (optional)
488
x_object_meta_*             Optional user defined metadata
489
==========================  ======================================
490

    
491
Extended replies may also include virtual directory markers in separate sections of the ``json`` or ``xml`` results.
492
Virtual directory markers are only included when ``delimiter`` is explicitly set. They correspond to the substrings up to and including the first occurrence of the delimiter.
493
In JSON results they appear as dictionaries with only a ``subdir`` key. In XML results they appear interleaved with ``<object>`` tags as ``<subdir name="..." />``.
494
In case there is an object with the same name as a virtual directory marker, the object will be returned.
495

    
496
Example ``format=json`` reply:
497

    
498
::
499

    
500
  [{"name": "object",
501
    "bytes": 0,
502
    "hash": "d41d8cd98f00b204e9800998ecf8427e",
503
    "content_type": "application/octet-stream",
504
    "last_modified": "2011-12-02T08:10:41.565891+00:00",
505
    "x_object_meta": {"asdf": "qwerty"},
506
    "x_object_hash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
507
    "x_object_version": 98,
508
    "x_object_version_timestamp": "1322813441.565891",
509
    "x_object_modified_by": "user"}, ...]
510

    
511
Example ``format=xml`` reply:
512

    
513
::
514

    
515
  <?xml version="1.0" encoding="UTF-8"?>
516
  <container name="pithos">
517
    <object>
518
      <name>object</name>
519
      <bytes>0</bytes>
520
      <hash>d41d8cd98f00b204e9800998ecf8427e</hash>
521
      <content_type>application/octet-stream</content_type>
522
      <last_modified>2011-12-02T08:10:41.565891+00:00</last_modified>
523
      <x_object_meta>
524
        <key>asdf</key><value>qwerty</value>
525
      </x_object_meta>
526
      <x_object_hash>e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855</x_object_hash>
527
      <x_object_version>98</x_object_version>
528
      <x_object_version_timestamp>1322813441.565891</x_object_version_timestamp>
529
      <x_object_modified_by>chazapis</x_object_modified_by>
530
    </object>
531
    <object>...</object>
532
  </container>
533

    
534
For more examples of container details returned in JSON/XML formats refer to the OOS API documentation. In addition to the OOS API, Pithos returns all fields. Metadata values are grouped and returned as key-value pairs.
535

    
536
===========================  ===============================
537
Return Code                  Description
538
===========================  ===============================
539
200 (OK)                     The request succeeded
540
204 (No Content)             The account has no containers (only for non-extended replies)
541
304 (Not Modified)           The container has not been modified
542
412 (Precondition Failed)    The condition set can not be satisfied
543
===========================  ===============================
544

    
545
Will use a ``200`` return code if the reply is of type JSON/XML.
546

    
547

    
548
PUT
549
"""
550

    
551
====================  ================================
552
Request Header Name   Value
553
====================  ================================
554
X-Container-Policy-*  Container behavior and limits
555
X-Container-Meta-*    Optional user defined metadata
556
====================  ================================
557
 
558
No reply content/headers.
559

    
560
If no policy is defined, the container will be created with the default values.
561
Available policy directives:
562

    
563
* ``versioning``: Set to ``auto`` or ``none`` (default is ``auto``)
564
* ``quota``: Size limit in KB (default is ``0`` - unlimited)
565

    
566
If the container already exists, the operation is equal to a ``POST`` with ``update`` defined.
567

    
568
================  ===============================
569
Return Code       Description
570
================  ===============================
571
201 (Created)     The container has been created
572
202 (Accepted)    The request has been accepted
573
================  ===============================
574

    
575

    
576
POST
577
""""
578

    
579
====================  ================================
580
Request Header Name   Value
581
====================  ================================
582
Content-Length        The size of the supplied data (optional, to upload)
583
Content-Type          The MIME content type of the supplied data (optional, to upload)
584
Transfer-Encoding     Set to ``chunked`` to specify incremental uploading (if used, ``Content-Length`` is ignored)
585
X-Container-Policy-*  Container behavior and limits
586
X-Container-Meta-*    Optional user defined metadata
587
====================  ================================
588

    
589
|
590

    
591
======================  ============================================
592
Request Parameter Name  Value
593
======================  ============================================
594
update                  Do not replace metadata/policy (no value parameter)
595
======================  ============================================
596

    
597
No reply content/headers, except when uploading data, where the reply consists of a list of hashes for the blocks received (in a simple text format, with one hash per line).
598

    
599
The operation will overwrite all user defined metadata, except if ``update`` is defined.
600
To change policy, include an ``X-Container-Policy-*`` header with the name in the key. If no ``X-Container-Policy-*`` header is present, no changes will be applied to policy. The ``update`` parameter also applies to policy - deleted values will revert to defaults. To delete/revert a specific policy directive, use ``update`` and an empty header value. See container ``PUT`` for a reference of policy directives.
601

    
602
To upload blocks of data to the container, set ``Content-Type`` to ``application/octet-stream`` and ``Content-Length`` to a valid value (except if using ``chunked`` as the ``Transfer-Encoding``).
603

    
604
================  ===============================
605
Return Code       Description
606
================  ===============================
607
202 (Accepted)    The request has been accepted
608
================  ===============================
609

    
610

    
611
DELETE
612
""""""
613

    
614
======================  ===================================
615
Request Parameter Name  Value
616
======================  ===================================
617
until                   Optional timestamp
618
======================  ===================================
619

    
620
If ``until`` is defined, the container is "purged" up to that time (the history of all objects up to then is deleted).
621

    
622
No reply content/headers.
623

    
624
================  ===============================
625
Return Code       Description
626
================  ===============================
627
204 (No Content)  The request succeeded
628
409 (Conflict)    The container is not empty
629
================  ===============================
630

    
631

    
632
Object Level
633
^^^^^^^^^^^^
634

    
635
List of operations:
636

    
637
=========  =================================
638
Operation  Description
639
=========  =================================
640
HEAD       Retrieve object metadata
641
GET        Read object data
642
PUT        Write object data or copy/move object
643
COPY       Copy object
644
MOVE       Move object
645
POST       Update object metadata/data
646
DELETE     Delete object
647
=========  =================================
648

    
649

    
650
HEAD
651
""""
652

    
653
====================  ================================
654
Request Header Name   Value
655
====================  ================================
656
If-Match              Retrieve if ETags match
657
If-None-Match         Retrieve if ETags don't match
658
If-Modified-Since     Retrieve if object has changed since provided timestamp
659
If-Unmodified-Since   Retrieve if object has not changed since provided timestamp
660
====================  ================================
661

    
662
|
663

    
664
======================  ===================================
665
Request Parameter Name  Value
666
======================  ===================================
667
version                 Optional version identifier
668
======================  ===================================
669

    
670
|
671

    
672
==========================  ===============================
673
Reply Header Name           Value
674
==========================  ===============================
675
ETag                        The ETag of the object
676
Content-Length              The size of the object
677
Content-Type                The MIME content type of the object
678
Last-Modified               The last object modification date (regardless of version)
679
Content-Encoding            The encoding of the object (optional)
680
Content-Disposition         The presentation style of the object (optional)
681
X-Object-Hash               The Merkle hash
682
X-Object-Version            The object's version identifier
683
X-Object-Version-Timestamp  The object's version timestamp
684
X-Object-Modified-By        The user that comitted the object's version
685
X-Object-Manifest           Object parts prefix in ``<container>/<object>`` form (optional)
686
X-Object-Sharing            Object permissions (optional)
687
X-Object-Shared-By          Object inheriting permissions (optional)
688
X-Object-Allowed-To         Allowed actions on object (optional)
689
X-Object-Public             Object's publicly accessible URI (optional)
690
X-Object-Meta-*             Optional user defined metadata
691
==========================  ===============================
692

    
693
|
694

    
695
================  ===============================
696
Return Code       Description
697
================  ===============================
698
200 (No Content)  The request succeeded
699
================  ===============================
700

    
701

    
702
GET
703
"""
704

    
705
====================  ================================
706
Request Header Name   Value
707
====================  ================================
708
Range                 Optional range of data to retrieve
709
If-Range              Retrieve the missing part if entity is unchanged; otherwise, retrieve the entire new entity (used together with Range header)
710
If-Match              Retrieve if ETags match
711
If-None-Match         Retrieve if ETags don't match
712
If-Modified-Since     Retrieve if object has changed since provided timestamp
713
If-Unmodified-Since   Retrieve if object has not changed since provided timestamp
714
====================  ================================
715

    
716
|
717

    
718
======================  ===================================
719
Request Parameter Name  Value
720
======================  ===================================
721
format                  Optional extended reply type (can be ``json`` or ``xml``)
722
hashmap                 Optional request for hashmap (no value parameter)
723
version                 Optional version identifier or ``list`` (specify a format if requesting a list)
724
======================  ===================================
725

    
726
The reply is the object's data (or part of it), except if a hashmap is requested with ``hashmap``, or a version list with ``version=list`` (in both cases an extended reply format must be specified). Object headers (as in a ``HEAD`` request) are always included.
727

    
728
Hashmaps expose the underlying storage format of the object. Note that each hash is computed after trimming trailing null bytes of the corresponding block.
729

    
730
Example ``format=json`` reply:
731

    
732
::
733

    
734
  {"block_hash": "sha1", "hashes": ["7295c41da03d7f916440b98e32c4a2a39351546c", ...], "block_size": 131072, "bytes": 242}
735

    
736
Example ``format=xml`` reply:
737

    
738
::
739

    
740
  <?xml version="1.0" encoding="UTF-8"?>
741
  <object name="file" bytes="24223726" block_size="131072" block_hash="sha1">
742
    <hash>7295c41da03d7f916440b98e32c4a2a39351546c</hash>
743
    <hash>...</hash>
744
  </object>
745

    
746
Version lists include the version identifier and timestamp for each available object version. Version identifiers can be arbitrary strings, so use the timestamp to find newer versions.
747

    
748
Example ``format=json`` reply:
749

    
750
::
751

    
752
  {"versions": [[85, "1322734861.248469"], [86, "1322734905.009272"], ...]}
753

    
754
Example ``format=xml`` reply:
755

    
756
::
757

    
758
  <?xml version="1.0" encoding="UTF-8"?>
759
  <object name="file">
760
    <version timestamp="1322734861.248469">85</version>
761
    <version timestamp="1322734905.009272">86</version>
762
    <version timestamp="...">...</version>
763
  </object>
764

    
765
The ``Range`` header may include multiple ranges, as outlined in RFC2616. Then the ``Content-Type`` of the reply will be ``multipart/byteranges`` and each part will include a ``Content-Range`` header.
766

    
767
==========================  ===============================
768
Reply Header Name           Value
769
==========================  ===============================
770
ETag                        The ETag of the object
771
Content-Length              The size of the data returned
772
Content-Type                The MIME content type of the object
773
Content-Range               The range of data included (only on a single range request)
774
Last-Modified               The last object modification date (regardless of version)
775
Content-Encoding            The encoding of the object (optional)
776
Content-Disposition         The presentation style of the object (optional)
777
X-Object-Hash               The Merkle hash
778
X-Object-Version            The object's version identifier
779
X-Object-Version-Timestamp  The object's version timestamp
780
X-Object-Modified-By        The user that comitted the object's version
781
X-Object-Manifest           Object parts prefix in ``<container>/<object>`` form (optional)
782
X-Object-Sharing            Object permissions (optional)
783
X-Object-Shared-By          Object inheriting permissions (optional)
784
X-Object-Allowed-To         Allowed actions on object (optional)
785
X-Object-Public             Object's publicly accessible URI (optional)
786
X-Object-Meta-*             Optional user defined metadata
787
==========================  ===============================
788

    
789
|
790

    
791
===========================  ==============================
792
Return Code                  Description
793
===========================  ==============================
794
200 (OK)                     The request succeeded
795
206 (Partial Content)        The range request succeeded
796
304 (Not Modified)           The object has not been modified
797
412 (Precondition Failed)    The condition set can not be satisfied
798
416 (Range Not Satisfiable)  The requested range is out of limits
799
===========================  ==============================
800

    
801

    
802
PUT
803
"""
804

    
805
====================  ================================
806
Request Header Name   Value
807
====================  ================================
808
If-Match              Put if ETags match with current object
809
If-None-Match         Put if ETags don't match with current object
810
ETag                  The MD5 hash of the object (optional to check written data)
811
Content-Length        The size of the data written
812
Content-Type          The MIME content type of the object
813
Transfer-Encoding     Set to ``chunked`` to specify incremental uploading (if used, ``Content-Length`` is ignored)
814
X-Copy-From           The source path in the form ``/<container>/<object>``
815
X-Move-From           The source path in the form ``/<container>/<object>``
816
X-Source-Account      The source account to copy/move from
817
X-Source-Version      The source version to copy from
818
Content-Encoding      The encoding of the object (optional)
819
Content-Disposition   The presentation style of the object (optional)
820
X-Object-Manifest     Object parts prefix in ``<container>/<object>`` form (optional)
821
X-Object-Sharing      Object permissions (optional)
822
X-Object-Public       Object is publicly accessible (optional)
823
X-Object-Meta-*       Optional user defined metadata
824
====================  ================================
825

    
826
|
827

    
828
======================  ===================================
829
Request Parameter Name  Value
830
======================  ===================================
831
format                  Optional extended request type (can be ``json`` or ``xml``)
832
hashmap                 Optional hashmap provided instead of data (no value parameter)
833
======================  ===================================
834

    
835
The request is the object's data (or part of it), except if a hashmap is provided (using ``hashmap`` and ``format`` parameters). If using a hashmap and all different parts are stored in the server, the object is created, otherwise the server returns Conflict (409) with the list of the missing parts (in a simple text format, with one hash per line).
836

    
837
Hashmaps should be formatted as outlined in ``GET``.
838

    
839
==========================  ===============================
840
Reply Header Name           Value
841
==========================  ===============================
842
ETag                        The MD5 hash of the object (on create)
843
X-Object-Version            The object's new version
844
==========================  ===============================
845

    
846
The ``X-Object-Sharing`` header may include either a ``read=...`` comma-separated user/group list, or a ``write=...`` comma-separated user/group list, or both separated by a semicolon (``;``). Groups are specified as ``<account>:<group>``. To publish the object, set ``X-Object-Public`` to ``true``. To unpublish, set to ``false``, or use an empty header value.
847

    
848
==============================  ==============================
849
Return Code                     Description
850
==============================  ==============================
851
201 (Created)                   The object has been created
852
409 (Conflict)                  The object can not be created from the provided hashmap, or there are conflicting permissions (a list of missing hashes, or a list of conflicting sharing paths will be included in the reply - in simple text format)
853
411 (Length Required)           Missing ``Content-Length`` or ``Content-Type`` in the request
854
413 (Request Entity Too Large)  Insufficient quota to complete the request
855
422 (Unprocessable Entity)      The MD5 checksum of the data written to the storage system does not match the (optionally) supplied ETag value
856
==============================  ==============================
857

    
858

    
859
COPY
860
""""
861

    
862
====================  ================================
863
Request Header Name   Value
864
====================  ================================
865
If-Match              Proceed if ETags match with object
866
If-None-Match         Proceed if ETags don't match with object
867
Destination           The destination path in the form ``/<container>/<object>``
868
Destination-Account   The destination account to copy to
869
Content-Type          The MIME content type of the object (optional)
870
Content-Encoding      The encoding of the object (optional)
871
Content-Disposition   The presentation style of the object (optional)
872
X-Source-Version      The source version to copy from
873
X-Object-Manifest     Object parts prefix in ``<container>/<object>`` form (optional)
874
X-Object-Sharing      Object permissions (optional)
875
X-Object-Public       Object is publicly accessible (optional)
876
X-Object-Meta-*       Optional user defined metadata
877
====================  ================================
878

    
879
Refer to ``PUT``/``POST`` for a description of request headers. Metadata is also copied, updated with any values defined. Sharing/publishing options are not copied.
880

    
881
==========================  ===============================
882
Reply Header Name           Value
883
==========================  ===============================
884
X-Object-Version            The object's new version
885
==========================  ===============================
886

    
887
|
888

    
889
==============================  ==============================
890
Return Code                     Description
891
==============================  ==============================
892
201 (Created)                   The object has been created
893
409 (Conflict)                  There are conflicting permissions (a list of conflicting sharing paths will be included in the reply - in simple text format)
894
413 (Request Entity Too Large)  Insufficient quota to complete the request
895
==============================  ==============================
896

    
897

    
898
MOVE
899
""""
900

    
901
Same as ``COPY``, without the ``X-Source-Version`` request header. The ``MOVE`` operation is always applied on the latest version.
902

    
903

    
904
POST
905
""""
906

    
907
====================  ================================
908
Request Header Name   Value
909
====================  ================================
910
If-Match              Proceed if ETags match with object
911
If-None-Match         Proceed if ETags don't match with object
912
Content-Length        The size of the data written (optional, to update)
913
Content-Type          The MIME content type of the object (optional, to update)
914
Content-Range         The range of data supplied (optional, to update)
915
Transfer-Encoding     Set to ``chunked`` to specify incremental uploading (if used, ``Content-Length`` is ignored)
916
Content-Encoding      The encoding of the object (optional)
917
Content-Disposition   The presentation style of the object (optional)
918
X-Source-Object       Update with data from the object at path ``/<container>/<object>`` (optional, to update)
919
X-Source-Account      The source account to update from
920
X-Source-Version      The source version to update from (optional, to update)
921
X-Object-Bytes        The updated object's final size (optional, when updating)
922
X-Object-Manifest     Object parts prefix in ``<container>/<object>`` form (optional)
923
X-Object-Sharing      Object permissions (optional)
924
X-Object-Public       Object is publicly accessible (optional)
925
X-Object-Meta-*       Optional user defined metadata
926
====================  ================================
927

    
928
|
929

    
930
======================  ============================================
931
Request Parameter Name  Value
932
======================  ============================================
933
update                  Do not replace metadata (no value parameter)
934
======================  ============================================
935

    
936
The ``Content-Encoding``, ``Content-Disposition``, ``X-Object-Manifest`` and ``X-Object-Meta-*`` headers are considered to be user defined metadata. An operation without the ``update`` parameter will overwrite all previous values and remove any keys not supplied. When using ``update`` any metadata with an empty value will be deleted.
937

    
938
To change permissions, include an ``X-Object-Sharing`` header (as defined in ``PUT``). To publish, include an ``X-Object-Public`` header, with a value of ``true``. If no such headers are defined, no changes will be applied to sharing/public. Use empty values to remove permissions/unpublish (unpublishing also works with ``false`` as a header value). Sharing options are applied to the object - not its versions.
939

    
940
To update an object's data:
941

    
942
* Either set ``Content-Type`` to ``application/octet-stream``, or provide an object with ``X-Source-Object``. If ``Content-Type`` has some other value, it will be ignored and only the metadata will be updated.
943
* If the data is supplied in the request (using ``Content-Type`` instead of ``X-Source-Object``), a valid ``Content-Length`` header is required - except if using chunked transfers (set ``Transfer-Encoding`` to ``chunked``).
944
* Set ``Content-Range`` as specified in RFC2616, with the following differences:
945

    
946
  * Client software MAY omit ``last-byte-pos`` of if the length of the range being transferred is unknown or difficult to determine.
947
  * Client software SHOULD not specify the ``instance-length`` (use a ``*``), unless there is a reason for performing a size check at the server.
948
* If ``Content-Range`` used has a ``byte-range-resp-spec = *``, data will be appended to the object.
949

    
950
Optionally, truncate the updated object to the desired length with the ``X-Object-Bytes`` header.
951

    
952
A data update will trigger an ETag change. Updated ETags correspond to the single Merkle hash of the object's hashmap (refer to http://bittorrent.org/beps/bep_0030.html for more information).
953

    
954
No reply content. No reply headers if only metadata is updated.
955

    
956
==========================  ===============================
957
Reply Header Name           Value
958
==========================  ===============================
959
ETag                        The new ETag of the object (data updated)
960
X-Object-Version            The object's new version
961
==========================  ===============================
962

    
963
|
964

    
965
==============================  ==============================
966
Return Code                     Description
967
==============================  ==============================
968
202 (Accepted)                  The request has been accepted (not a data update)
969
204 (No Content)                The request succeeded (data updated)
970
409 (Conflict)                  There are conflicting permissions (a list of conflicting sharing paths will be included in the reply - in simple text format)
971
411 (Length Required)           Missing ``Content-Length`` in the request
972
413 (Request Entity Too Large)  Insufficient quota to complete the request
973
416 (Range Not Satisfiable)     The supplied range is invalid
974
==============================  ==============================
975

    
976
The ``POST`` method can also be used for creating an object via a standard HTML form. If the request ``Content-Type`` is ``multipart/form-data``, none of the above headers will be processed. The form should have an ``X-Object-Data`` field, as in the following example. The token is passed as a request parameter. ::
977

    
978
  <form method="post" action="https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt?X-Auth-Token=0000" enctype="multipart/form-data">
979
    <input type="file" name="X-Object-Data">
980
    <input type="submit">
981
  </form>
982

    
983
This will create/override the object with the given name, as if using ``PUT``. The ``Content-Type`` of the object will be set to the value of the corresponding header sent in the part of the request containing the data (usually, automatically handled by the browser). Metadata, sharing and other object attributes can not be set this way.
984

    
985
==========================  ===============================
986
Reply Header Name           Value
987
==========================  ===============================
988
ETag                        The MD5 hash of the object
989
X-Object-Version            The object's new version
990
==========================  ===============================
991

    
992
|
993

    
994
==============================  ==============================
995
Return Code                     Description
996
==============================  ==============================
997
201 (Created)                   The object has been created
998
413 (Request Entity Too Large)  Insufficient quota to complete the request
999
==============================  ==============================
1000

    
1001

    
1002
DELETE
1003
""""""
1004

    
1005
======================  ===================================
1006
Request Parameter Name  Value
1007
======================  ===================================
1008
until                   Optional timestamp
1009
======================  ===================================
1010

    
1011
If ``until`` is defined, the object is "purged" up to that time (the history up to then is deleted).
1012

    
1013
No reply content/headers.
1014

    
1015
===========================  ==============================
1016
Return Code                  Description
1017
===========================  ==============================
1018
204 (No Content)             The request succeeded
1019
===========================  ==============================
1020

    
1021
Sharing and Public Objects
1022
^^^^^^^^^^^^^^^^^^^^^^^^^^
1023

    
1024
Read and write control in Pithos is managed by setting appropriate permissions with the ``X-Object-Sharing`` header. The permissions are applied using prefix-based inheritance. Thus, each set of authorization directives is applied to all objects sharing the same prefix with the object where the corresponding ``X-Object-Sharing`` header is defined. For simplicity, nested/overlapping permissions are not allowed. Setting ``X-Object-Sharing`` will fail, if the object is already "covered", or another object with a longer common-prefix name already has permissions. When retrieving an object, the ``X-Object-Shared-By`` header reports where it gets its permissions from. If not present, the object is the actual source of authorization directives.
1025

    
1026
A user may ``GET`` another account or container. The result will include a limited reply, containing only the allowed containers or objects respectively. A top-level request with an authentication token, will return a list of allowed accounts, so the user can easily find out which other users share objects. The ``X-Object-Allowed-To`` header lists the actions allowed on an object, if it does not belong to the requesting user.
1027

    
1028
Objects that are marked as public, via the ``X-Object-Public`` meta, are also available at the corresponding URI returned for ``HEAD`` or ``GET``. Requests for public objects do not need to include an ``X-Auth-Token``. Pithos will ignore request parameters and only include the following headers in the reply (all ``X-Object-*`` meta is hidden):
1029

    
1030
==========================  ===============================
1031
Reply Header Name           Value
1032
==========================  ===============================
1033
ETag                        The ETag of the object
1034
Content-Length              The size of the data returned
1035
Content-Type                The MIME content type of the object
1036
Content-Range               The range of data included (only on a single range request)
1037
Last-Modified               The last object modification date (regardless of version)
1038
Content-Encoding            The encoding of the object (optional)
1039
Content-Disposition         The presentation style of the object (optional)
1040
==========================  ===============================
1041

    
1042
Public objects are not included and do not influence cross-user listings. They are, however, readable by all users.
1043

    
1044
Summary
1045
^^^^^^^
1046

    
1047
List of differences from the OOS API:
1048

    
1049
* Support for ``X-Account-Meta-*`` style headers at the account level. Use ``POST`` to update.
1050
* Support for ``X-Container-Meta-*`` style headers at the container level. Can be set when creating via ``PUT``. Use ``POST`` to update.
1051
* Header ``X-Container-Object-Meta`` at the container level and parameter ``meta`` in container listings. (**TBD**)
1052
* Account and container policies to manage behavior and limits. Container behavior overrides account settings. Account quota sets the maximum bytes limit, regardless of container values.
1053
* Headers ``X-Container-Block-*`` at the container level, exposing the underlying storage characteristics.
1054
* All metadata replies, at all levels, include latest modification information.
1055
* At all levels, a ``HEAD`` or ``GET`` request may use ``If-Modified-Since`` and ``If-Unmodified-Since`` headers.
1056
* Container/object lists include all associated metadata if the reply is of type JSON/XML. Some names are kept to their OOS API equivalents for compatibility.
1057
* Option to include only shared containers/objects in listings.
1058
* Object metadata allowed, in addition to ``X-Object-Meta-*``: ``Content-Encoding``, ``Content-Disposition``, ``X-Object-Manifest``. These are all replaced with every update operation, except if using the ``update`` parameter (in which case individual keys can also be deleted). Deleting meta by providing empty values also works when copying/moving an object.
1059
* Multi-range object ``GET`` support as outlined in RFC2616.
1060
* Object hashmap retrieval through ``GET`` and the ``format`` parameter.
1061
* Object create via hashmap through ``PUT`` and the ``format`` parameter.
1062
* The object's Merkle hash is always returned in the ``X-Object-Hash`` header.
1063
* Object create using ``POST`` to support standard HTML forms.
1064
* Partial object updates through ``POST``, using the ``Content-Length``, ``Content-Type``, ``Content-Range`` and ``Transfer-Encoding`` headers. Use another object's data to update with ``X-Source-Object`` and ``X-Source-Version``. Truncate with ``X-Object-Bytes``. New ETag corresponds to the Merkle hash of the object's hashmap.
1065
* Include new version identifier in replies for object replace/change requests.
1066
* Object ``MOVE`` support.
1067
* Conditional object create/update operations, using ``If-Match`` and ``If-None-Match`` headers.
1068
* Time-variant account/container listings via the ``until`` parameter.
1069
* Object versions - parameter ``version`` in ``HEAD``/``GET`` (list versions with ``GET``), ``X-Object-Version-*`` meta in replies, ``X-Source-Version`` in ``PUT``/``COPY``.
1070
* Sharing/publishing with ``X-Object-Sharing``, ``X-Object-Public`` at the object level. Cross-user operations are allowed - controlled by sharing directives. Available actions in cross-user requests are reported with ``X-Object-Allowed-To``. Permissions may include groups defined with ``X-Account-Group-*`` at the account level. These apply to the object - not its versions.
1071
* Support for prefix-based inheritance when enforcing permissions. Parent object carrying the authorization directives is reported in ``X-Object-Shared-By``.
1072
* Copy and move between accounts with ``X-Source-Account`` and ``Destination-Account`` headers.
1073
* Large object support with ``X-Object-Manifest``.
1074
* Trace the user that created/modified an object with ``X-Object-Modified-By``.
1075
* Purge container/object history with the ``until`` parameter in ``DELETE``.
1076

    
1077
Clarifications/suggestions:
1078

    
1079
* All non-ASCII characters in headers should be URL-encoded.
1080
* Authentication is done by another system. The token is used in the same way, but it is obtained differently. The top level ``GET`` request is kept compatible with the OOS API and allows for guest/testing operations.
1081
* Some processing is done in the variable part of all ``X-*-Meta-*`` headers. If it includes underscores, they will be converted to dashes and the first letter of all intra-dash strings will be capitalized.
1082
* A ``GET`` reply for a level will include all headers of the corresponding ``HEAD`` request.
1083
* To avoid conflicts between objects and virtual directory markers in container listings, it is recommended that object names do not end with the delimiter used.
1084
* The ``Accept`` header may be used in requests instead of the ``format`` parameter to specify the desired request/reply format. The parameter overrides the header.
1085
* Container/object lists use a ``200`` return code if the reply is of type JSON/XML. The reply will include an empty JSON/XML.
1086
* In headers, dates are formatted according to RFC 1123. In extended information listings, the ``last_modified`` field is formatted according to ISO 8601 (for OOS API compatibility). All other fields (Pithos extensions) use integer tiemstamps.
1087
* The ``Last-Modified`` header value always reflects the actual latest change timestamp, regardless of time control parameters and version requests. Time precondition checks with ``If-Modified-Since`` and ``If-Unmodified-Since`` headers are applied to this value.
1088
* A copy/move using ``PUT``/``COPY``/``MOVE`` will always update metadata, keeping all old values except the ones redefined in the request headers.
1089
* A ``HEAD`` or ``GET`` for an ``X-Object-Manifest`` object, will include modified ``Content-Length`` and ``ETag`` headers, according to the characteristics of the objects under the specified prefix. The ``Etag`` will be the MD5 hash of the corresponding ETags concatenated. In extended container listings there is no metadata processing.
1090

    
1091
The Pithos Client
1092
-----------------
1093

    
1094
User Experience
1095
^^^^^^^^^^^^^^^
1096

    
1097
Hopefully this API will allow for a multitude of client implementations, each supporting a different device or operating system. All clients will be able to manipulate containers and objects - even software only designed for OOS API compatibility. But a Pithos interface should not be only about showing containers and folders. There are some extra user interface elements and functionalities that should be common to all implementations.
1098

    
1099
Upon entrance to the service, a user is presented with the following elements - which can be represented as folders or with other related icons:
1100

    
1101
* The ``home`` element, which is used as the default entry point to the user's "files". Objects under ``home`` are represented in the usual hierarchical organization of folders and files.
1102
* The ``trash`` element, which contains files that have been marked for deletion, but can still be recovered.
1103
* The ``shared`` element, which contains all objects shared by the user to other users of the system.
1104
* The ``others`` element, which contains all objects that other users share with the user.
1105
* The ``groups`` element, which contains the names of groups the user has defined. Each group consists of a user list. Group creation, deletion, and manipulation is carried out by actions originating here.
1106
* The ``history`` element, which allows browsing past instances of ``home`` and - optionally - ``trash``.
1107

    
1108
Objects in Pithos can be:
1109

    
1110
* Moved to trash and then deleted.
1111
* Shared with specific permissions.
1112
* Made public (shared with non-Pithos users).
1113
* Restored from previous versions.
1114

    
1115
Some of these functions are performed by the client software and some by the Pithos server.
1116

    
1117
In the first version of Pithos, objects could also be assigned custom tags. This is no longer supported. Existing deployments can migrate tags into a specific metadata value, i.e. ``X-Object-Meta-Tags``.
1118

    
1119
Implementation Guidelines
1120
^^^^^^^^^^^^^^^^^^^^^^^^^
1121

    
1122
Pithos clients should use the ``pithos`` and ``trash`` containers for active and inactive objects respectively. If any of these containers is not found, the client software should create it, without interrupting the user's workflow. The ``home`` element corresponds to ``pithos`` and the ``trash`` element to ``trash``. Use ``PUT`` with the ``X-Move-From`` header, or ``MOVE`` to transfer objects from one container to the other. Use ``DELETE`` to remove from ``pithos`` without trashing, or to remove from ``trash``. When moving objects, detect naming conflicts with the ``If-Match`` or ``If-None-Match`` headers. Such conflicts should be resolved by the user.
1123

    
1124
Object names should use the ``/`` delimiter to impose a hierarchy of folders and files.
1125

    
1126
The ``shared`` element should be implemented as a read-only view of the ``pithos`` container, using the ``shared`` parameter when listing objects. The ``others`` element, should start with a top-level ``GET`` to retrieve the list of accounts accessible to the user. It is suggested that the client software hides the next step of navigation - the container - if it only includes ``pithos`` and forwards the user directly to the objects.
1127

    
1128
Public objects are not included in ``shared`` and ``others`` listings. It is suggested that they are marked in a visually distinctive way in ``pithos`` listings (for example using an icon overlay).
1129

    
1130
A special application menu, or a section in application preferences, should be devoted to managing groups (the ``groups`` element). All group-related actions are implemented at the account level.
1131

    
1132
Browsing past versions of objects should be available both at the object and the container level. At the object level, a list of past versions can be included in the screen showing details or more information on the object (metadata, permissions, etc.). At the container level, it is suggested that clients use a ``history`` element, which presents to the user a read-only, time-variable view of ``pithos`` contents. This can be accomplished via the ``until`` parameter in listings. Optionally, ``history`` may include ``trash``.
1133

    
1134
Uploading and downloading data
1135
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1136

    
1137
By using hashmaps to upload and download objects the corresponding operations can complete much faster.
1138

    
1139
In the case of an upload, only the missing blocks will be submitted to the server:
1140

    
1141
* Calculate the hash value for each block of the object to be uploaded. Use the hash algorithm and block size of the destination container.
1142
* Send a hashmap ``PUT`` request for the object.
1143

    
1144
  * Server responds with status ``201`` (Created):
1145

    
1146
    * Blocks are already on the server. The object has been created. Done.
1147

    
1148
  * Server responds with status ``409`` (Conflict):
1149

    
1150
    * Server's response body contains the hashes of the blocks that do not exist on the server.
1151
    * For each hash value in the server's response (or all hashes together):
1152

    
1153
      * Send a ``POST`` request to the destination container with the corresponding data.
1154

    
1155
* Repeat hashmap ``PUT``. Fail if the server's response is not ``201``.
1156

    
1157
Consulting hashmaps when downloading allows for resuming partially transferred objects. The client should retrieve the hashmap from the server and compare it with the hashmap computed from the respective local file. Any missing parts can be downloaded with ``GET`` requests with the additional ``Range`` header.
1158

    
1159
Syncing
1160
^^^^^^^
1161

    
1162
Consider the following algorithm for synchronizing a local folder with the server. The "state" is the complete object listing, with the corresponding attributes.
1163
 
1164
::
1165

    
1166
  L: local state (stored state from last sync with the server)
1167
  C: current state (state computed right before sync)
1168
  S: server state
1169

    
1170
  if C == L:
1171
      # No local changes
1172
      if S == L:
1173
          # No remote changes, nothing to do
1174
      else:
1175
          # Update local state to match that of the server
1176
         L = S
1177
  else:
1178
      # We have local changes
1179
      if S == L:
1180
          # No remote changes, update the server
1181
          S = C
1182
          L = S
1183
      else:
1184
          # Both we and server have changes
1185
          if C == S:
1186
              # We were lucky, we did the same change
1187
              L = S
1188
          else:
1189
              # We have conflicting changes
1190
              resolve conflict
1191

    
1192
Notes:
1193

    
1194
* States represent file hashes (either MD5 or Merkle). Deleted or non-existing files are assumed to have a magic hash (e.g. empty string).
1195
* Updating a state (either local or remote) implies downloading, uploading or deleting the appropriate file.
1196

    
1197
Recommended Practices and Examples
1198
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1199

    
1200
Assuming an authentication token is obtained, the following high-level operations are available - shown with ``curl``:
1201

    
1202
* Get account information ::
1203

    
1204
    curl -X HEAD -D - \
1205
         -H "X-Auth-Token: 0000" \
1206
         https://pithos.dev.grnet.gr/v1/user
1207

    
1208
* List available containers ::
1209

    
1210
    curl -X GET -D - \
1211
         -H "X-Auth-Token: 0000" \
1212
         https://pithos.dev.grnet.gr/v1/user
1213

    
1214
* Get container information ::
1215

    
1216
    curl -X HEAD -D - \
1217
         -H "X-Auth-Token: 0000" \
1218
         https://pithos.dev.grnet.gr/v1/user/pithos
1219

    
1220
* Add a new container ::
1221

    
1222
    curl -X PUT -D - \
1223
         -H "X-Auth-Token: 0000" \
1224
         https://pithos.dev.grnet.gr/v1/user/test
1225

    
1226
* Delete a container ::
1227

    
1228
    curl -X DELETE -D - \
1229
         -H "X-Auth-Token: 0000" \
1230
         https://pithos.dev.grnet.gr/v1/user/test
1231

    
1232
* List objects in a container ::
1233

    
1234
    curl -X GET -D - \
1235
         -H "X-Auth-Token: 0000" \
1236
         https://pithos.dev.grnet.gr/v1/user/pithos
1237

    
1238
* List objects in a container (extended reply) ::
1239

    
1240
    curl -X GET -D - \
1241
         -H "X-Auth-Token: 0000" \
1242
         https://pithos.dev.grnet.gr/v1/user/pithos?format=json
1243

    
1244
  It is recommended that extended replies are cached and subsequent requests utilize the ``If-Modified-Since`` header.
1245

    
1246
* List metadata keys used by objects in a container
1247

    
1248
  Will be in the ``X-Container-Object-Meta`` reply header, included in container information or object list (``HEAD`` or ``GET``). (**TBD**)
1249

    
1250
* List objects in a container having a specific meta defined ::
1251

    
1252
    curl -X GET -D - \
1253
         -H "X-Auth-Token: 0000" \
1254
         https://pithos.dev.grnet.gr/v1/user/pithos?meta=favorites
1255

    
1256
* Retrieve an object ::
1257

    
1258
    curl -X GET -D - \
1259
         -H "X-Auth-Token: 0000" \
1260
         https://pithos.dev.grnet.gr/v1/user/pithos/README.txt
1261

    
1262
* Retrieve an object (specific ranges of data) ::
1263

    
1264
    curl -X GET -D - \
1265
         -H "X-Auth-Token: 0000" \
1266
         -H "Range: bytes=0-9" \
1267
         https://pithos.dev.grnet.gr/v1/user/pithos/README.txt
1268

    
1269
  This will return the first 10 bytes. To get the first 10, bytes 30-39 and the last 100 use ``Range: bytes=0-9,30-39,-100``.
1270

    
1271
* Add a new object (folder type) (**TBD**) ::
1272

    
1273
    curl -X PUT -D - \
1274
         -H "X-Auth-Token: 0000" \
1275
         -H "Content-Type: application/directory" \
1276
         https://pithos.dev.grnet.gr/v1/user/pithos/folder
1277

    
1278
* Add a new object ::
1279

    
1280
    curl -X PUT -D - \
1281
         -H "X-Auth-Token: 0000" \
1282
         -H "Content-Type: text/plain" \
1283
         -T EXAMPLE.txt
1284
         https://pithos.dev.grnet.gr/v1/user/pithos/folder/EXAMPLE.txt
1285

    
1286
* Update an object ::
1287

    
1288
    curl -X POST -D - \
1289
         -H "X-Auth-Token: 0000" \
1290
         -H "Content-Length: 10" \
1291
         -H "Content-Type: application/octet-stream" \
1292
         -H "Content-Range: bytes 10-19/*" \
1293
         -d "0123456789" \
1294
         https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt
1295

    
1296
  This will update bytes 10-19 with the data specified.
1297

    
1298
* Update an object (append) ::
1299

    
1300
    curl -X POST -D - \
1301
         -H "X-Auth-Token: 0000" \
1302
         -H "Content-Length: 10" \
1303
         -H "Content-Type: application/octet-stream" \
1304
         -H "Content-Range: bytes */*" \
1305
         -d "0123456789" \
1306
         https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt
1307

    
1308
* Update an object (truncate) ::
1309

    
1310
    curl -X POST -D - \
1311
         -H "X-Auth-Token: 0000" \
1312
         -H "X-Source-Object: /folder/EXAMPLE.txt" \
1313
         -H "Content-Range: bytes 0-0/*" \
1314
         -H "X-Object-Bytes: 0" \
1315
         https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt
1316

    
1317
  This will truncate the object to 0 bytes.
1318

    
1319
* Add object metadata ::
1320

    
1321
    curl -X POST -D - \
1322
         -H "X-Auth-Token: 0000" \
1323
         -H "X-Object-Meta-First: first_meta_value" \
1324
         -H "X-Object-Meta-Second: second_meta_value" \
1325
         https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt
1326

    
1327
* Delete object metadata ::
1328

    
1329
    curl -X POST -D - \
1330
         -H "X-Auth-Token: 0000" \
1331
         -H "X-Object-Meta-First: first_meta_value" \
1332
         https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt
1333

    
1334
  Metadata can only be "set". To delete ``X-Object-Meta-Second``, reset all metadata.
1335

    
1336
* Delete an object ::
1337

    
1338
    curl -X DELETE -D - \
1339
         -H "X-Auth-Token: 0000" \
1340
         https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt