Statistics
| Branch: | Tag: | Revision:

root / docs / source / devguide.rst @ 3f839f59

History | View | Annotate | Download (26.1 kB)

1
Pithos v2 Developer Guide
2
=========================
3

    
4
Introduction
5
------------
6

    
7
Pithos is a storage service implemented by GRNET (http://www.grnet.gr). Data is stored as objects, organized in containers, belonging to an account. This hierarchy of storage layers has been inspired by the OpenStack Object Storage (OOS) API and similar CloudFiles API by Rackspace. The Pithos API follows the OOS API as closely as possible. One of the design requirements has been to be able to use Pithos with clients built for the OOS, without changes.
8

    
9
This document's goals are:
10

    
11
* Define the Pithos ReST API that allows the storage and retrieval of data and metadata via HTTP calls
12
* Specify metadata semantics and user interface guidelines for a common experience across client software implementations
13

    
14
The present document is meant to be read alongside the OOS API documentation. Thus, it is suggested that the reader is familiar with associated technologies, the OOS API as well as the first version of the Pithos API. This document refers to the second version of Pithos. Information on the first version of the storage API can be found at http://code.google.com/p/gss.
15

    
16
Whatever marked as to be determined (**TBD**), should not be considered by implementors.
17

    
18
Document Revisions
19
^^^^^^^^^^^^^^^^^^
20

    
21
=========================  ================================
22
Revision                   Description
23
=========================  ================================
24
0.2 (May 25, 2011)         Add object meta listing and filtering in containers.
25
\                          Support for multi-range object GET requests.
26
0.1 (May 17, 2011)         Initial release. Based on OpenStack Object Storage Developer Guide API v1 (Apr. 15, 2011).
27
=========================  ================================
28

    
29
The Pithos API
30
--------------
31

    
32
The URI requests supported by Pithos follow one of the following forms:
33

    
34
* Top level: ``https://hostname/v1/``
35
* Account level: ``https://hostname/v1/<account>``
36
* Container level: ``https://hostname/v1/<account>/<container>``
37
* Object level: ``https://hostname/v1/<account>/<container>/<object>``
38

    
39
All requests must include an ``X-Auth-Token``, except from those that refer to publicly available files (**TBD**). The process of obtaining the token is still to be determined (**TBD**).
40

    
41
The allowable request operations and corresponding return codes per level are presented in the remainder of this chapter. Common to all requests are the following return codes.
42

    
43
=========================  ================================
44
Return Code                Description
45
=========================  ================================
46
400 (Bad Request)          The request is invalid
47
401 (Unauthorized)         Request not allowed
48
404 (Not Found)            The requested resource was not found
49
503 (Service Unavailable)  The request cannot be completed because of an internal error
50
=========================  ================================
51

    
52
Top Level
53
^^^^^^^^^
54

    
55
List of operations:
56

    
57
=========  ==================
58
Operation  Description
59
=========  ==================
60
GET        Authentication. This is kept for compatibility with the OOS API
61
=========  ==================
62

    
63
GET
64
"""
65

    
66
If the ``X-Auth-User`` and ``X-Auth-Key`` headers are given, a dummy ``X-Auth-Token`` and ``X-Storage-Url`` will be replied, which can be used as a guest token/namespace for testing Pithos.
67

    
68
================  =====================
69
Return Code       Description
70
================  =====================
71
204 (No Content)  The request succeeded
72
================  =====================
73

    
74

    
75
Account Level
76
^^^^^^^^^^^^^
77

    
78
List of operations:
79

    
80
=========  ==================
81
Operation  Description
82
=========  ==================
83
HEAD       Retrieve account metadata
84
GET        List containers
85
POST       Update account metadata
86
=========  ==================
87

    
88
HEAD
89
""""
90

    
91
No request parameters/headers.
92

    
93
==========================  =====================
94
Reply Header Name           Value
95
==========================  =====================
96
X-Account-Container-Count   The total number of containers
97
X-Account-Object-Count      The total number of objects (**TBD**)
98
X-Account-Bytes-Used        The total number of bytes stored
99
X-Account-Bytes-Remaining   The total number of bytes remaining (**TBD**)
100
X-Account-Last-Login        The last login (**TBD**)
101
X-Account-Meta-*            Optional user defined metadata
102
Last-Modified               The last object modification date
103
==========================  =====================
104

    
105
|
106

    
107
================  =====================
108
Return Code       Description
109
================  =====================
110
204 (No Content)  The request succeeded
111
================  =====================
112

    
113

    
114
GET
115
"""
116

    
117
====================  ===========================
118
Request Header Name   Value
119
====================  ===========================
120
If-Modified-Since     Retrieve if account has changed since provided timestamp
121
If-Unmodified-Since   Retrieve if account has not changed since provided timestamp
122
====================  ===========================
123

    
124
|
125

    
126
======================  =========================
127
Request Parameter Name  Value
128
======================  =========================
129
limit                   The amount of results requested (default is 10000)
130
marker                  Return containers with name lexicographically after marker
131
format                  Optional extended reply type (can be ``json`` or ``xml``)
132
======================  =========================
133

    
134
The reply is a list of container names. Account headers (as in a ``HEAD`` request) will also be included.
135
If a ``format=xml`` or ``format=json`` argument is given, extended information on the containers will be returned, serialized in the chosen format.
136
For each container, the information will include all container metadata (names will be in lower case and with hyphens replaced with underscores):
137

    
138
===================  ============================
139
Name                 Description
140
===================  ============================
141
name                 The name of the container
142
count                The number of objects inside the container
143
bytes                The total size of the objects inside the container
144
last_modified        The last object modification date
145
x_container_meta_*   Optional user defined metadata
146
===================  ============================
147

    
148
For examples of container details returned in JSON/XML formats refer to the OOS API documentation.
149

    
150
================  =====================
151
Return Code       Description
152
================  =====================
153
200 (OK)          The request succeeded
154
204 (No Content)  The account has no containers (only for non-extended replies)
155
================  =====================
156

    
157
Will use a ``200`` return code if the reply is of type json/xml.
158

    
159

    
160
POST
161
""""
162

    
163
====================  ===========================
164
Request Header Name   Value
165
====================  ===========================
166
X-Account-Meta-*      Optional user defined metadata
167
====================  ===========================
168

    
169
No reply content/headers.
170

    
171
The update operation will overwrite all user defined metadata.
172

    
173
================  ===============================
174
Return Code       Description
175
================  ===============================
176
202 (Accepted)    The request has been accepted
177
================  ===============================
178

    
179

    
180
Container Level
181
^^^^^^^^^^^^^^^
182

    
183
List of operations:
184

    
185
=========  ============================
186
Operation  Description
187
=========  ============================
188
HEAD       Retrieve container metadata
189
GET        List objects
190
PUT        Create/update container
191
POST       Update container metadata
192
DELETE     Delete container
193
=========  ============================
194

    
195

    
196
HEAD
197
""""
198

    
199
No request parameters/headers.
200

    
201
==========================  ===============================
202
Reply Header Name           Value
203
==========================  ===============================
204
X-Container-Object-Count    The total number of objects in the container
205
X-Container-Bytes-Used      The total number of bytes of all objects stored
206
X-Container-Meta-*          Optional user defined metadata
207
X-Container-Object-Meta     A list with all meta keys used by objects
208
Last-Modified               The last object modification date
209
==========================  ===============================
210

    
211
The keys returned in ``X-Container-Object-Meta`` are all the unique strings after the ``X-Object-Meta-`` prefix.
212

    
213
================  ===============================
214
Return Code       Description
215
================  ===============================
216
204 (No Content)  The request succeeded
217
================  ===============================
218

    
219

    
220
GET
221
"""
222

    
223
====================  ===========================
224
Request Header Name   Value
225
====================  ===========================
226
If-Modified-Since     Retrieve if container has changed since provided timestamp
227
If-Unmodified-Since   Retrieve if container has not changed since provided timestamp
228
====================  ===========================
229

    
230
|
231

    
232
======================  ===================================
233
Request Parameter Name  Value
234
======================  ===================================
235
limit                   The amount of results requested (default is 10000)
236
marker                  Return containers with name lexicographically after marker
237
prefix                  Return objects starting with prefix
238
delimiter               Return objects up to the delimiter (discussion follows)
239
path                    Assume ``prefix=path`` and ``delimiter=/``
240
format                  Optional extended reply type (can be ``json`` or ``xml``)
241
meta                    Return objects having the specified meta keys (can be a comma separated list)
242
======================  ===================================
243

    
244
The ``path`` parameter overrides ``prefix`` and ``delimiter``. When using ``path``, results will include objects ending in ``delimiter``.
245

    
246
The keys given with ``meta`` will be matched with the strings after the ``X-Object-Meta-`` prefix.
247

    
248
The reply is a list of object names. Container headers (as in a ``HEAD`` request) will also be included.
249
If a ``format=xml`` or ``format=json`` argument is given, extended information on the objects will be returned, serialized in the chosen format.
250
For each object, the information will include all object metadata (names will be in lower case and with hyphens replaced with underscores):
251

    
252
===================  ======================================
253
Name                 Description
254
===================  ======================================
255
name                 The name of the object
256
hash                 The MD5 hash of the object
257
bytes                The size of the object
258
content_type         The MIME content type of the object
259
content_encoding     The encoding of the object (optional)
260
last_modified        The last object modification date
261
x_object_manifest    Large object support
262
x_object_meta_*      Optional user defined metadata
263
===================  ======================================
264

    
265
Extended replies may also include virtual directory markers in separate sections of the ``json`` or ``xml`` results.
266
Virtual directory markers are only included when ``delimiter`` is explicitly set. They correspond to the substrings up to and including the first occurrence of the delimiter.
267
In JSON results they appear as dictionaries with only a ``"subdir"`` key. In XML results they appear interleaved with ``<object>`` tags as ``<subdir name="..." />``.
268
In case there is an object with the same name as a virtual directory marker, the object will be returned.
269
 
270
For examples of object details returned in JSON/XML formats refer to the OOS API documentation.
271

    
272
================  ===============================
273
Return Code       Description
274
================  ===============================
275
200 (OK)          The request succeeded
276
204 (No Content)  The account has no containers (only for non-extended replies)
277
================  ===============================
278

    
279
Will use a ``200`` return code if the reply is of type json/xml.
280

    
281

    
282
PUT
283
"""
284

    
285
====================  ================================
286
Request Header Name   Value
287
====================  ================================
288
X-Container-Meta-*    Optional user defined metadata
289
====================  ================================
290
 
291
No reply content/headers.
292
 
293
================  ===============================
294
Return Code       Description
295
================  ===============================
296
201 (Created)     The container has been created
297
202 (Accepted)    The request has been accepted
298
================  ===============================
299

    
300

    
301
POST
302
""""
303

    
304
====================  ================================
305
Request Header Name   Value
306
====================  ================================
307
X-Container-Meta-*    Optional user defined metadata
308
====================  ================================
309

    
310
No reply content/headers.
311

    
312
The update operation will overwrite all user defined metadata.
313

    
314
================  ===============================
315
Return Code       Description
316
================  ===============================
317
202 (Accepted)    The request has been accepted
318
================  ===============================
319

    
320

    
321
DELETE
322
""""""
323

    
324
No request parameters/headers.
325

    
326
No reply content/headers.
327

    
328
================  ===============================
329
Return Code       Description
330
================  ===============================
331
204 (No Content)  The request succeeded
332
409 (Conflict)    The container is not empty
333
================  ===============================
334

    
335

    
336
Object Level
337
^^^^^^^^^^^^
338

    
339
List of operations:
340

    
341
=========  =================================
342
Operation  Description
343
=========  =================================
344
HEAD       Retrieve object metadata
345
GET        Read object data
346
PUT        Write object data or copy/move object
347
COPY       Copy object
348
MOVE       Move object
349
POST       Update object metadata
350
DELETE     Delete object
351
=========  =================================
352

    
353

    
354
HEAD
355
""""
356

    
357
No request parameters/headers.
358

    
359
==========================  ===============================
360
Reply Header Name           Value
361
==========================  ===============================
362
ETag                        The MD5 hash of the object
363
Content-Length              The size of the object
364
Content-Type                The MIME content type of the object
365
Last-Modified               The last object modification date
366
Content-Encoding            The encoding of the object (optional)
367
Content-Disposition         The presentation style of the object (optional)
368
X-Object-Manifest           Large object support (optional)
369
X-Object-Meta-*             Optional user defined metadata
370
==========================  ===============================
371

    
372
|
373

    
374
================  ===============================
375
Return Code       Description
376
================  ===============================
377
204 (No Content)  The request succeeded
378
================  ===============================
379

    
380

    
381
GET
382
"""
383

    
384
====================  ================================
385
Request Header Name   Value
386
====================  ================================
387
Range                 Optional range of data to retrieve
388
If-Match              Retrieve if ETags match
389
If-None-Match         Retrieve if ETags don't match
390
If-Modified-Since     Retrieve if object has changed since provided timestamp
391
If-Unmodified-Since   Retrieve if object has not changed since provided timestamp
392
====================  ================================
393

    
394
The reply is the object's data (or part of it). Object headers (as in a ``HEAD`` request) will also be included.
395

    
396
The ``Range`` header may include multiple ranges, as outlined in RFC2616. Then the ``Content-Type`` of the reply will be ``multipart/byteranges`` and each part will include a ``Content-Range`` header.
397

    
398
==========================  ===============================
399
Reply Header Name           Value
400
==========================  ===============================
401
ETag                        The MD5 hash of the object
402
Content-Length              The size of the data returned
403
Content-Type                The MIME content type of the object
404
Content-Range               The range of data included (only on a single range request)
405
Last-Modified               The last object modification date
406
Content-Encoding            The encoding of the object (optional)
407
Content-Disposition         The presentation style of the object (optional)
408
X-Object-Manifest           Large object support (optional)
409
X-Object-Meta-*             Optional user defined metadata
410
==========================  ===============================
411

    
412
|
413

    
414
===========================  ==============================
415
Return Code                  Description
416
===========================  ==============================
417
200 (OK)                     The request succeeded
418
206 (Partial Content)        The range request succeeded
419
304 (Not Modified)           The object has not been modified
420
412 (Precondition Failed)    The condition set can not be satisfied
421
416 (Range Not Satisfiable)  The requested range is out of limits
422
===========================  ==============================
423

    
424

    
425
PUT
426
"""
427

    
428
====================  ================================
429
Request Header Name   Value
430
====================  ================================
431
ETag                  The MD5 hash of the object (optional to check written data)
432
Content-Length        The size of the data written
433
Content-Type          The MIME content type of the object
434
Transfer-Encoding     Set to ``chunked`` to specify incremental uploading (if used, ``Content-Length`` is ignored)
435
X-Copy-From           The source path in the form ``/<container>/<object>``
436
X-Move-From           The source path in the form ``/<container>/<object>``
437
Content-Encoding      The encoding of the object (optional)
438
Content-Disposition   The presentation style of the object (optional)
439
X-Object-Manifest     Large object support (optional)
440
X-Object-Meta-*       Optional user defined metadata
441
====================  ================================
442

    
443
|
444

    
445
==========================  ===============================
446
Reply Header Name           Value
447
==========================  ===============================
448
ETag                        The MD5 hash of the object (on create)
449
==========================  ===============================
450

    
451
|
452

    
453
===========================  ==============================
454
Return Code                  Description
455
===========================  ==============================
456
201 (Created)                The object has been created
457
411 (Length Required)        Missing ``Content-Length`` or ``Content-Type`` in the request
458
422 (Unprocessable Entity)   The MD5 checksum of the data written to the storage system does not match the (optionally) supplied ETag value
459
===========================  ==============================
460

    
461

    
462
COPY
463
""""
464

    
465
====================  ================================
466
Request Header Name   Value
467
====================  ================================
468
Destination           The destination path in the form ``/<container>/<object>``
469
Content-Type          The MIME content type of the object (optional)
470
Content-Encoding      The encoding of the object (optional)
471
Content-Disposition   The presentation style of the object (optional)
472
X-Object-Manifest     Large object support (optional)
473
X-Object-Meta-*       Optional user defined metadata
474
====================  ================================
475

    
476
No reply content/headers.
477

    
478
===========================  ==============================
479
Return Code                  Description
480
===========================  ==============================
481
201 (Created)                The object has been created
482
===========================  ==============================
483

    
484

    
485
MOVE
486
""""
487

    
488
Same as ``COPY``.
489

    
490

    
491
POST
492
""""
493

    
494
====================  ================================
495
Request Header Name   Value
496
====================  ================================
497
Content-Encoding      The encoding of the object (optional)
498
Content-Disposition   The presentation style of the object (optional)
499
X-Object-Manifest     Large object support (optional)
500
X-Object-Meta-*       Optional user defined metadata
501
====================  ================================
502

    
503
No reply content/headers.
504

    
505
The allowed headers are considered to be user defined metadata. The update operation will overwrite all previous values and remove any keys not supplied.
506

    
507
===========================  ==============================
508
Return Code                  Description
509
===========================  ==============================
510
202 (Accepted)               The request has been accepted
511
===========================  ==============================
512

    
513

    
514
DELETE
515
""""""
516

    
517
No request parameters/headers.
518

    
519
No reply content/headers.
520

    
521
===========================  ==============================
522
Return Code                  Description
523
===========================  ==============================
524
204 (No Content)             The request succeeded
525
===========================  ==============================
526

    
527

    
528
Summary
529
^^^^^^^
530

    
531
List of differences from the OOS API:
532

    
533
* Support for ``X-Account-Meta-*`` style headers at the account level. Use ``POST`` to update.
534
* Support for ``X-Container-Meta-*`` style headers at the account level. Can be set when creating via ``PUT``. Use ``POST`` to update.
535
* Header ``X-Container-Object-Meta`` at the container level and parameter ``meta`` in container listings.
536
* All metadata replies, at all levels, include latest modification information.
537
* At all levels, a ``GET`` request may use ``If-Modified-Since`` and ``If-Unmodified-Since`` headers.
538
* Container/object lists include all associated metadata if the reply is of type json/xml. Some names are kept to their OOS API equivalents for compatibility. 
539
* Object metadata allowed, in addition to ``X-Object-Meta-*``: ``Content-Encoding``, ``Content-Disposition``, ``X-Object-Manifest``. These are all replaced with every update operation.
540
* Multi-range object GET support as outlined in RFC2616.
541
* Object ``MOVE`` support.
542

    
543
Clarifications/suggestions:
544

    
545
* Authentication is done by another system. The token is used in the same way, but it is obtained differently. The top level ``GET`` request is kept compatible with the OOS API and allows for guest/testing operations.
546
* Some processing is done in the variable part of all ``X-*-Meta-*`` headers. If it includes underscores, they will be converted to dashes and the first letter of all intra-dash strings will be capitalized.
547
* A ``GET`` reply for a level will include all headers of the corresponding ``HEAD`` request.
548
* To avoid conflicts between objects and virtual directory markers in container listings, it is recommended that object names do not end with the delimiter used.
549
* The ``Accept`` header may be used in requests instead of the ``format`` parameter to specify the desired reply format. The parameter overrides the header.
550
* Container/object lists use a ``200`` return code if the reply is of type json/xml. The reply will include an empty json/xml.
551
* In headers, dates are formatted according to RFC 1123. In extended information listings, dates are formatted according to ISO 8601.
552
* While ``X-Object-Manifest`` can be set and unset, large object support is not yet implemented (**TBD**).
553

    
554
The Pithos Client
555
-----------------
556

    
557
User Experience
558
^^^^^^^^^^^^^^^
559

    
560
Hopefully this API will allow for a multitude of client implementations, each supporting a different device or operating system. All clients will be able to manipulate containers and objects - even software only designed for OOS API compatibility. But a Pithos interface should not be only about showing containers and folders. There are some extra user interface elements and functionalities that should be common to all implementations.
561

    
562
Upon entrance to the service, a user is presented with the following elements - which can be represented as folders or with other related icons:
563

    
564
* The ``home`` element, which is used as the default entry point to the user's "files". Objects under ``home`` are represented in the usual hierarchical organization of folders and files.
565
* The ``trash`` element, which contains files that have been marked for deletion, but can still be recovered.
566
* The ``shared`` element, which contains all objects shared by the user to other users of the system.
567
* The ``others`` element, which contains all objects that other users share with the user.
568
* The ``tags`` element, which lists the names of tags the user has defined. This can be an entry point to list all files that have been assigned a specific tag or manage tags in general (remove a tag completely, rename a tag etc.).
569
* The ``groups`` element, which contains the names of groups the user has defined. Each group consists of a user list. Group creation, deletion, and manipulation is carried out by actions originating here.
570

    
571
Objects in Pithos can be:
572

    
573
* Assigned custom tags.
574
* Moved to trash and then deleted.
575
* Shared with specific permissions.
576
* Made public (shared with non-Pithos users).
577
* Set to monitor changes via version tracking.
578

    
579
Some of these functions are performed by the client software and some by the Pithos server. Client-driven functionality is based on specific metadata that should be handled equally across implementations. These metadata names are discussed in the next chapter. 
580

    
581
Conventions and Metadata Specification
582
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
583

    
584
Pithos clients should use the ``pithos`` container for all Pithos objects. Object names use the ``/`` delimiter to impose a hierarchy of folders and files.
585

    
586
At the object level, tags are implemented by managing metadata keys. The client software should allow the user to use any string as a tag (except ``trash``) and then set the corresponding ``X-Object-Meta-<tag>`` key at the server. The API extensions provided, allow for listing all tags in a container and filtering object listings based on one or more tags. The tag list is sufficient for implementing the ``tags`` element, either as a special, virtual folder (as done in the first version of Pithos), or as an application menu.
587

    
588
To manage the deletion of files use the same API and the ``X-Object-Meta-Trash`` key. The string ``trash`` can not be used as a tag. The ``trash`` element should be presented as a folder, although with no hierarchy.
589

    
590
The metadata specification is summarized in the following table.
591

    
592
===========================  ==============================
593
Metadata Name                Value
594
===========================  ==============================
595
X-Object-Meta-Trash          Set to ``true`` if the object has been moved to the trash
596
X-Object-Meta-*              Use for other tags that apply to the object
597
===========================  ==============================