Revision dad708b4 docs/pithos-api-guide.rst

b/docs/pithos-api-guide.rst
1 1
Pithos+ API
2 2
===========
3 3

  
4
This is the Pithos+ API guide.
4
Introduction
5
------------
5 6

  
7
Pithos is a storage service implemented by GRNET (http://www.grnet.gr). Data is stored as objects, organized in containers, belonging to an account. This hierarchy of storage layers has been inspired by the OpenStack Object Storage (OOS) API and similar CloudFiles API by Rackspace. The Pithos API follows the OOS API as closely as possible. One of the design requirements has been to be able to use Pithos with clients built for the OOS, without changes.
6 8

  
7
Overview
8
--------
9
However, to be able to take full advantage of the Pithos infrastructure, client software should be aware of the extensions that differentiate Pithos from OOS. Pithos objects can be updated, or appended to. Pithos will store sharing permissions per object and enforce corresponding authorization policies. Automatic version management, allows taking account and container listings back in time, as well as reading previous instances of objects.
9 10

  
10
Pithos+ data is stored as objects, organized in containers, belonging to an
11
account. This hierarchy of storage layers has been inspired by the OpenStack
12
Object Storage (OOS) API and similar CloudFiles API by Rackspace. The Pithos
13
API follows the OOS API as closely as possible. One of the design requirements
14
has been to be able to use Pithos with clients built for the OOS, without
15
changes.
16

  
17
However, to be able to take full advantage of the Pithos infrastructure, client
18
software should be aware of the extensions that differentiate Pithos from OOS.
11
The storage backend of Pithos is block oriented, permitting efficient, deduplicated data placement. The block structure of objects is exposed at the API layer, in order to encourage external software to implement advanced data management operations.
19 12

  
20 13
This document's goals are:
21 14

  
22
 * Define the Pithos ReST API that allows the storage and retrieval of data and
23
   metadata via HTTP calls
24
 * Specify metadata semantics and user interface guidelines for a common
25
   experience across client software implementations
15
* Define the Pithos ReST API that allows the storage and retrieval of data and metadata via HTTP calls
16
* Specify metadata semantics and user interface guidelines for a common experience across client software implementations
17

  
18
The present document is meant to be read alongside the OOS API documentation. Thus, it is suggested that the reader is familiar with associated technologies, the OOS API as well as the first version of the Pithos API. This document refers to the second version of Pithos. Information on the first version of the storage API can be found at http://code.google.com/p/gss.
26 19

  
27
The present document is meant to be read alongside the OOS API documentation.
28
Thus, it is suggested that the reader is familiar with associated technologies,
29
the OOS API as well as the first version of the Pithos API. This document
30
refers to the version of Pithos+. Information on the first version of the
31
storage API can be found at http://code.google.com/p/gss.
20
Whatever marked as to be determined (**TBD**), should not be considered by implementors.
32 21

  
33
Whatever marked as to be determined (**TBD**), should not be considered by
34
implementors.
22
More info about Pithos can be found here: https://code.grnet.gr/projects/pithos
35 23

  
36 24
Document Revisions
37 25
^^^^^^^^^^^^^^^^^^
......
93 81
0.1 (May 17, 2011)         Initial release. Based on OpenStack Object Storage Developer Guide API v1 (Apr. 15, 2011).
94 82
=========================  ================================
95 83

  
96
Users and Authentication
97
------------------------
84
Pithos Users and Authentication
85
-------------------------------
98 86

  
99
In Pithos+, each user is uniquely identified by a token. All API requests
100
require a token and each token is internally resolved to an account string. The
101
API uses the account string to identify the user's own files, thus whether a
102
request is local or cross-account.
87
In Pithos, each user is uniquely identified by a token. All API requests require a token and each token is internally resolved to an account string. The API uses the account string to identify the user's own files, thus whether a request is local or cross-account.
103 88

  
104
Pithos+ does not keep a user database. For development and testing purposes,
105
user identifiers and their corresponding tokens can be defined in the settings
106
file. However, Pithos is designed with an external authentication service in
107
mind. This service must handle the details of validating user credentials and
108
communicate with Pithos via a middleware software component that, given a
109
token, fills in the internal request account variable.
89
Pithos does not keep a user database. For development and testing purposes, user identifiers and their corresponding tokens can be defined in the settings file. However, Pithos is designed with an external authentication service in mind. This service must handle the details of validating user credentials and communicate with Pithos via a middleware software component that, given a token, fills in the internal request account variable.
110 90

  
111
Client software using Pithos+, if not already knowing a user's identifier and
112
token, should forward to the ``/login`` URI. The Pithos server, depending on
113
its configuration will redirect to the appropriate login page.
91
Client software using Pithos, if not already knowing a user's identifier and token, should forward to the ``/login`` URI. The Pithos server, depending on its configuration will redirect to the appropriate login page.
114 92

  
115 93
The login URI accepts the following parameters:
116 94

  
......
126 104

  
127 105
A user management service that implements a login URI according to these conventions is Astakos (https://code.grnet.gr/projects/astakos), by GRNET.
128 106

  
129
API Operations
107
The Pithos API
130 108
--------------
131 109

  
132
The URI requests supported by the Pithos+ API follow one of the following forms:
110
The URI requests supported by the Pithos API follow one of the following forms:
133 111

  
134 112
* Top level: ``https://hostname/v1/``
135 113
* Account level: ``https://hostname/v1/<account>``
......
1120 1098
* The ``Last-Modified`` header value always reflects the actual latest change timestamp, regardless of time control parameters and version requests. Time precondition checks with ``If-Modified-Since`` and ``If-Unmodified-Since`` headers are applied to this value.
1121 1099
* A copy/move using ``PUT``/``COPY``/``MOVE`` will always update metadata, keeping all old values except the ones redefined in the request headers.
1122 1100
* A ``HEAD`` or ``GET`` for an ``X-Object-Manifest`` object, will include modified ``Content-Length`` and ``ETag`` headers, according to the characteristics of the objects under the specified prefix. The ``Etag`` will be the MD5 hash of the corresponding ETags concatenated. In extended container listings there is no metadata processing.
1101

  
1102
Recommended Practices and Examples
1103
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1104

  
1105
Assuming an authentication token is obtained, the following high-level operations are available - shown with ``curl``:
1106

  
1107
* Get account information ::
1108

  
1109
    curl -X HEAD -D - \
1110
         -H "X-Auth-Token: 0000" \
1111
         https://pithos.dev.grnet.gr/v1/user
1112

  
1113
* List available containers ::
1114

  
1115
    curl -X GET -D - \
1116
         -H "X-Auth-Token: 0000" \
1117
         https://pithos.dev.grnet.gr/v1/user
1118

  
1119
* Get container information ::
1120

  
1121
    curl -X HEAD -D - \
1122
         -H "X-Auth-Token: 0000" \
1123
         https://pithos.dev.grnet.gr/v1/user/pithos
1124

  
1125
* Add a new container ::
1126

  
1127
    curl -X PUT -D - \
1128
         -H "X-Auth-Token: 0000" \
1129
         https://pithos.dev.grnet.gr/v1/user/test
1130

  
1131
* Delete a container ::
1132

  
1133
    curl -X DELETE -D - \
1134
         -H "X-Auth-Token: 0000" \
1135
         https://pithos.dev.grnet.gr/v1/user/test
1136

  
1137
* List objects in a container ::
1138

  
1139
    curl -X GET -D - \
1140
         -H "X-Auth-Token: 0000" \
1141
         https://pithos.dev.grnet.gr/v1/user/pithos
1142

  
1143
* List objects in a container (extended reply) ::
1144

  
1145
    curl -X GET -D - \
1146
         -H "X-Auth-Token: 0000" \
1147
         https://pithos.dev.grnet.gr/v1/user/pithos?format=json
1148

  
1149
  It is recommended that extended replies are cached and subsequent requests utilize the ``If-Modified-Since`` header.
1150

  
1151
* List metadata keys used by objects in a container
1152

  
1153
  Will be in the ``X-Container-Object-Meta`` reply header, included in container information or object list (``HEAD`` or ``GET``). (**TBD**)
1154

  
1155
* List objects in a container having a specific meta defined ::
1156

  
1157
    curl -X GET -D - \
1158
         -H "X-Auth-Token: 0000" \
1159
         https://pithos.dev.grnet.gr/v1/user/pithos?meta=favorites
1160

  
1161
* Retrieve an object ::
1162

  
1163
    curl -X GET -D - \
1164
         -H "X-Auth-Token: 0000" \
1165
         https://pithos.dev.grnet.gr/v1/user/pithos/README.txt
1166

  
1167
* Retrieve an object (specific ranges of data) ::
1168

  
1169
    curl -X GET -D - \
1170
         -H "X-Auth-Token: 0000" \
1171
         -H "Range: bytes=0-9" \
1172
         https://pithos.dev.grnet.gr/v1/user/pithos/README.txt
1173

  
1174
  This will return the first 10 bytes. To get the first 10, bytes 30-39 and the last 100 use ``Range: bytes=0-9,30-39,-100``.
1175

  
1176
* Add a new object (folder type) (**TBD**) ::
1177

  
1178
    curl -X PUT -D - \
1179
         -H "X-Auth-Token: 0000" \
1180
         -H "Content-Type: application/directory" \
1181
         https://pithos.dev.grnet.gr/v1/user/pithos/folder
1182

  
1183
* Add a new object ::
1184

  
1185
    curl -X PUT -D - \
1186
         -H "X-Auth-Token: 0000" \
1187
         -H "Content-Type: text/plain" \
1188
         -T EXAMPLE.txt
1189
         https://pithos.dev.grnet.gr/v1/user/pithos/folder/EXAMPLE.txt
1190

  
1191
* Update an object ::
1192

  
1193
    curl -X POST -D - \
1194
         -H "X-Auth-Token: 0000" \
1195
         -H "Content-Length: 10" \
1196
         -H "Content-Type: application/octet-stream" \
1197
         -H "Content-Range: bytes 10-19/*" \
1198
         -d "0123456789" \
1199
         https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt
1200

  
1201
  This will update bytes 10-19 with the data specified.
1202

  
1203
* Update an object (append) ::
1204

  
1205
    curl -X POST -D - \
1206
         -H "X-Auth-Token: 0000" \
1207
         -H "Content-Length: 10" \
1208
         -H "Content-Type: application/octet-stream" \
1209
         -H "Content-Range: bytes */*" \
1210
         -d "0123456789" \
1211
         https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt
1212

  
1213
* Update an object (truncate) ::
1214

  
1215
    curl -X POST -D - \
1216
         -H "X-Auth-Token: 0000" \
1217
         -H "X-Source-Object: /folder/EXAMPLE.txt" \
1218
         -H "Content-Range: bytes 0-0/*" \
1219
         -H "X-Object-Bytes: 0" \
1220
         https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt
1221

  
1222
  This will truncate the object to 0 bytes.
1223

  
1224
* Add object metadata ::
1225

  
1226
    curl -X POST -D - \
1227
         -H "X-Auth-Token: 0000" \
1228
         -H "X-Object-Meta-First: first_meta_value" \
1229
         -H "X-Object-Meta-Second: second_meta_value" \
1230
         https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt
1231

  
1232
* Delete object metadata ::
1233

  
1234
    curl -X POST -D - \
1235
         -H "X-Auth-Token: 0000" \
1236
         -H "X-Object-Meta-First: first_meta_value" \
1237
         https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt
1238

  
1239
  Metadata can only be "set". To delete ``X-Object-Meta-Second``, reset all metadata.
1240

  
1241
* Delete an object ::
1242

  
1243
    curl -X DELETE -D - \
1244
         -H "X-Auth-Token: 0000" \
1245
         https://pithos.dev.grnet.gr/v1/user/folder/EXAMPLE.txt

Also available in: Unified diff