Revision f6c97079

b/docs/source/client-lib.rst
1 1
Client Library
2
=============
2
==============
3 3

  
4
.. automodule:: pithos.lib.client
4
.. automodule:: pithos.lib.client
b/docs/source/devguide.rst
25 25
=========================  ================================
26 26
Revision                   Description
27 27
=========================  ================================
28
0.5 (July 16, 2011)        Object update from another object's data.
28
0.5 (July 19, 2011)        Object update from another object's data.
29 29
\                          Support object truncate.
30 30
\                          Create object using a standard HTML form.
31 31
\                          Purge container/object history.
32
\                          List other accounts that share objects with a user.
32 33
0.4 (July 01, 2011)        Object permissions and account groups.
33 34
\                          Control versioning behavior and container quotas with container policy directives.
34 35
\                          Support updating/deleting individual metadata with ``POST``.
......
77 78
=========  ==================
78 79
Operation  Description
79 80
=========  ==================
80
GET        Authentication. This is kept for compatibility with the OOS API
81
GET        Authentication (for compatibility with the OOS API) or list allowed accounts
81 82
=========  ==================
82 83

  
83 84
GET
......
91 92
204 (No Content)  The request succeeded
92 93
================  =====================
93 94

  
95
If an ``X-Auth-Token`` is already present, the operation will be interpreted as a request to list other accounts that share objects to the user.
96

  
97
======================  =========================
98
Request Parameter Name  Value
99
======================  =========================
100
limit                   The amount of results requested (default is 10000)
101
marker                  Return containers with name lexicographically after marker
102
format                  Optional extended reply type (can be ``json`` or ``xml``)
103
======================  =========================
104

  
105
The reply is a list of account names.
106
If a ``format=xml`` or ``format=json`` argument is given, extended information on the containers will be returned, serialized in the chosen format.
107
For each account, the information will include the following (names will be in lower case and with hyphens replaced with underscores):
108

  
109
===========================  ============================
110
Name                         Description
111
===========================  ============================
112
name                         The name of the account
113
last_modified                The last container modification date (regardless of ``until``)
114
===========================  ============================
115

  
116
Example ``format=json`` reply:
117

  
118
::
119

  
120
  [{"name": "user", "last_modified": "2011-07-19T10:48:16"}, ...]
121

  
122
Example ``format=xml`` reply:
123

  
124
::
125

  
126
  <?xml version="1.0" encoding="UTF-8"?>
127
  <accounts>
128
    <account>
129
      <name>user</name>
130
      <last_modified>2011-07-19T10:48:16</last_modified>
131
    </account>
132
    <account>...</account>
133
  </accounts>
134

  
135
===========================  =====================
136
Return Code                  Description
137
===========================  =====================
138
200 (OK)                     The request succeeded
139
204 (No Content)             The account has no containers (only for non-extended replies)
140
===========================  =====================
141

  
142
Will use a ``200`` return code if the reply is of type json/xml.
94 143

  
95 144
Account Level
96 145
^^^^^^^^^^^^^
......
114 163
until                   Optional timestamp
115 164
======================  ===================================
116 165

  
117
|
166
Cross-user requests are not allowed to use ``until`` and only include the account modification date in the reply.
118 167

  
119 168
==========================  =====================
120 169
Reply Header Name           Value
......
161 210
======================  =========================
162 211

  
163 212
The reply is a list of container names. Account headers (as in a ``HEAD`` request) will also be included.
213
Cross-user requests are not allowed to use ``until`` and only include the account/container modification dates in the reply.
214

  
164 215
If a ``format=xml`` or ``format=json`` argument is given, extended information on the containers will be returned, serialized in the chosen format.
165 216
For each container, the information will include all container metadata (names will be in lower case and with hyphens replaced with underscores):
166 217

  
......
245 296
until                   Optional timestamp
246 297
======================  ===================================
247 298

  
248
|
299
Cross-user requests are not allowed to use ``until`` and only include the container modification date in the reply.
249 300

  
250 301
===========================  ===============================
251 302
Reply Header Name            Value
......
300 351
The keys given with ``meta`` will be matched with the strings after the ``X-Object-Meta-`` prefix.
301 352

  
302 353
The reply is a list of object names. Container headers (as in a ``HEAD`` request) will also be included.
354
Cross-user requests are not allowed to use ``until`` and include the following limited set of headers in the reply:
355

  
356
===========================  ===============================
357
Reply Header Name            Value
358
===========================  ===============================
359
X-Container-Block-Size       The block size used by the storage backend
360
X-Container-Block-Hash       The hash algorithm used for block identifiers in object hashmaps
361
X-Container-Object-Meta      A list with all meta keys used by allowed objects (**TBD**)
362
Last-Modified                The last container modification date
363
===========================  ===============================
364

  
303 365
If a ``format=xml`` or ``format=json`` argument is given, extended information on the objects will be returned, serialized in the chosen format.
304 366
For each object, the information will include all object metadata (names will be in lower case and with hyphens replaced with underscores):
305 367

  
......
791 853

  
792 854
Read and write control in Pithos is managed by setting appropriate permissions with the ``X-Object-Sharing`` header. The permissions are applied using prefix-based inheritance. Thus, each set of authorization directives is applied to all objects sharing the same prefix with the object where the corresponding ``X-Object-Sharing`` header is defined. For simplicity, nested/overlapping permissions are not allowed. Setting ``X-Object-Sharing`` will fail, if the object is already "covered", or another object with a longer common-prefix name already has permissions. When retrieving an object, the ``X-Object-Shared-By`` header reports where it gets its permissions from. If not present, the object is the actual source of authorization directives.
793 855

  
794
Objects that are marked as public, via the ``X-Object-Public`` meta, are also available at the corresponding URI returned for ``HEAD`` or ``GET``. Requests for public objects do not need to include an ``X-Auth-Token``. Pithos will ignore request parameters and only include the following headers in the reply (all ``X-Object-*`` meta is hidden).
856
A user may ``GET`` another account or container. The result will include a limited reply, containing only the allowed containers or objects respectively. A top-level request with an authentication token, will return a list of allowed accounts, so the user can easily find out which other users share objects.
857

  
858
Objects that are marked as public, via the ``X-Object-Public`` meta, are also available at the corresponding URI returned for ``HEAD`` or ``GET``. Requests for public objects do not need to include an ``X-Auth-Token``. Pithos will ignore request parameters and only include the following headers in the reply (all ``X-Object-*`` meta is hidden):
795 859

  
796 860
==========================  ===============================
797 861
Reply Header Name           Value
......
805 869
Content-Disposition         The presentation style of the object (optional)
806 870
==========================  ===============================
807 871

  
872
Public objects are not included and do not influence cross-user listings. They are, however, readable by all users.
873

  
808 874
Summary
809 875
^^^^^^^
810 876

  
......
827 893
* Object ``MOVE`` support.
828 894
* Time-variant account/container listings via the ``until`` parameter.
829 895
* Object versions - parameter ``version`` in ``HEAD``/``GET`` (list versions with ``GET``), ``X-Object-Version-*`` meta in replies, ``X-Source-Version`` in ``PUT``/``COPY``.
830
* Sharing/publishing with ``X-Object-Sharing``, ``X-Object-Public`` at the object level. Permissions may include groups defined with ``X-Account-Group-*`` at the account level. These apply to the object - not its versions.
896
* Sharing/publishing with ``X-Object-Sharing``, ``X-Object-Public`` at the object level. Cross-user operations are allowed - controlled by sharing directives. Permissions may include groups defined with ``X-Account-Group-*`` at the account level. These apply to the object - not its versions.
831 897
* Support for prefix-based inheritance when enforcing permissions. Parent object carrying the authorization directives is reported in ``X-Object-Shared-By``.
832 898
* Large object support with ``X-Object-Manifest``.
833 899
* Trace the user that created/modified an object with ``X-Object-Modified-By``.
b/pithos/api/functions.py
59 59

  
60 60
def top_demux(request):
61 61
    if request.method == 'GET':
62
        if request.user:
63
            return account_list(request)
62 64
        return authenticate(request)
63 65
    else:
64 66
        return method_not_allowed(request)
......
125 127
                                            x_auth_user)
126 128
    return response
127 129

  
130
@api_method('GET', format_allowed=True)
131
def account_list(request):
132
    # Normal Response Codes: 200, 204
133
    # Error Response Codes: serviceUnavailable (503),
134
    #                       badRequest (400)
135
    
136
    response = HttpResponse()
137
    
138
    marker = request.GET.get('marker')
139
    limit = get_int_parameter(request.GET.get('limit'))
140
    if not limit:
141
        limit = 10000
142
    
143
    accounts = backend.list_accounts(request.user, marker, limit)
144
    
145
    if request.serialization == 'text':
146
        if len(accounts) == 0:
147
            # The cloudfiles python bindings expect 200 if json/xml.
148
            response.status_code = 204
149
            return response
150
        response.status_code = 200
151
        response.content = '\n'.join(accounts) + '\n'
152
        return response
153
    
154
    account_meta = []
155
    for x in accounts:
156
        try:
157
            meta = backend.get_account_meta(request.user, x)
158
            groups = backend.get_account_groups(request.user, x)
159
        except NotAllowedError:
160
            raise Unauthorized('Access denied')
161
        else:
162
            for k, v in groups.iteritems():
163
                meta['X-Container-Group-' + k] = ','.join(v)
164
            account_meta.append(printable_header_dict(meta))
165
    if request.serialization == 'xml':
166
        data = render_to_string('accounts.xml', {'accounts': account_meta})
167
    elif request.serialization  == 'json':
168
        data = json.dumps(account_meta)
169
    response.status_code = 200
170
    response.content = data
171
    return response
172

  
128 173
@api_method('HEAD')
129 174
def account_meta(request, v_account):
130 175
    # Normal Response Codes: 204
......
188 233
    put_account_headers(response, meta, groups)
189 234
    
190 235
    marker = request.GET.get('marker')
191
    limit = request.GET.get('limit')
192
    if limit:
193
        try:
194
            limit = int(limit)
195
            if limit <= 0:
196
                raise ValueError
197
        except ValueError:
198
            limit = 10000
236
    limit = get_int_parameter(request.GET.get('limit'))
237
    if not limit:
238
        limit = 10000
199 239
    
200 240
    try:
201 241
        containers = backend.list_containers(request.user, v_account, marker, limit, until)
......
210 250
            response.status_code = 204
211 251
            return response
212 252
        response.status_code = 200
213
        response.content = '\n'.join([x[0] for x in containers]) + '\n'
253
        response.content = '\n'.join(containers) + '\n'
214 254
        return response
215 255
    
216 256
    container_meta = []
217 257
    for x in containers:
218
        if x[1] is not None:
219
            try:
220
                meta = backend.get_container_meta(request.user, v_account, x[0], until)
221
                policy = backend.get_container_policy(request.user, v_account, x[0])
222
            except NotAllowedError:
223
                raise Unauthorized('Access denied')
224
            except NameError:
225
                pass
226
            else:
227
                for k, v in policy.iteritems():
228
                    meta['X-Container-Policy-' + k] = v
229
                container_meta.append(printable_header_dict(meta))
258
        try:
259
            meta = backend.get_container_meta(request.user, v_account, x, until)
260
            policy = backend.get_container_policy(request.user, v_account, x)
261
        except NotAllowedError:
262
            raise Unauthorized('Access denied')
263
        except NameError:
264
            pass
265
        else:
266
            for k, v in policy.iteritems():
267
                meta['X-Container-Policy-' + k] = v
268
            container_meta.append(printable_header_dict(meta))
230 269
    if request.serialization == 'xml':
231 270
        data = render_to_string('containers.xml', {'account': v_account, 'containers': container_meta})
232 271
    elif request.serialization  == 'json':
......
376 415
    prefix = prefix.lstrip('/')
377 416
    
378 417
    marker = request.GET.get('marker')
379
    limit = request.GET.get('limit')
380
    if limit:
381
        try:
382
            limit = int(limit)
383
            if limit <= 0:
384
                raise ValueError
385
        except ValueError:
386
            limit = 10000
418
    limit = get_int_parameter(request.GET.get('limit'))
419
    if not limit:
420
        limit = 10000
387 421
    
388 422
    keys = request.GET.get('meta')
389 423
    if keys:
b/pithos/api/util.py
96 96
    return meta, groups
97 97

  
98 98
def put_account_headers(response, meta, groups):
99
    response['X-Account-Container-Count'] = meta['count']
100
    response['X-Account-Bytes-Used'] = meta['bytes']
99
    if 'count' in meta:
100
        response['X-Account-Container-Count'] = meta['count']
101
    if 'bytes' in meta:
102
        response['X-Account-Bytes-Used'] = meta['bytes']
101 103
    if 'modified' in meta:
102 104
        response['Last-Modified'] = http_date(int(meta['modified']))
103 105
    for k in [x for x in meta.keys() if x.startswith('X-Account-Meta-')]:
......
113 115
    return meta, policy
114 116

  
115 117
def put_container_headers(response, meta, policy):
116
    response['X-Container-Object-Count'] = meta['count']
117
    response['X-Container-Bytes-Used'] = meta['bytes']
118
    if 'count' in meta:
119
        response['X-Container-Object-Count'] = meta['count']
120
    if 'bytes' in meta:
121
        response['X-Container-Bytes-Used'] = meta['bytes']
118 122
    response['Last-Modified'] = http_date(int(meta['modified']))
119 123
    for k in [x for x in meta.keys() if x.startswith('X-Container-Meta-')]:
120 124
        response[k.encode('utf-8')] = meta[k].encode('utf-8')
b/pithos/backends/base.py
49 49
        'block_size': Suggested is 4MB
50 50
    """
51 51
    
52
    def list_accounts(self, user, marker=None, limit=10000):
53
        """Return a list of accounts the user can access.
54
        
55
        Parameters:
56
            'marker': Start list from the next item after 'marker'
57
            'limit': Number of containers to return
58
        """
59
        return []
60
    
52 61
    def get_account_meta(self, user, account, until=None):
53 62
        """Return a dictionary with the account metadata.
54 63
        
......
103 112
        return
104 113
    
105 114
    def list_containers(self, user, account, marker=None, limit=10000, until=None):
106
        """Return a list of container (name, version_id) tuples existing under an account.
115
        """Return a list of container names existing under an account.
107 116
        
108 117
        Parameters:
109 118
            'marker': Start list from the next item after 'marker'
b/pithos/backends/simple.py
130 130
        self.mapper = Mapper(**params)
131 131
    
132 132
    @backend_method
133
    def list_accounts(self, user, marker=None, limit=10000):
134
        """Return a list of accounts the user can access."""
135
        
136
        allowed = self._allowed_accounts(user)
137
        start, limit = self._list_limits(allowed, marker, limit)
138
        return allowed[start:start + limit]
139
    
140
    @backend_method
133 141
    def get_account_meta(self, user, account, until=None):
134 142
        """Return a dictionary with the account metadata."""
135 143
        
136 144
        logger.debug("get_account_meta: %s %s", account, until)
137 145
        if user != account:
138
            raise NotAllowedError
146
            if until or account not in self._allowed_accounts(user):
147
                raise NotAllowedError
139 148
        try:
140 149
            version_id, mtime = self._get_accountinfo(account, until)
141 150
        except NameError:
......
158 167
        row = c.fetchone()
159 168
        count = row[0]
160 169
        
161
        meta = self._get_metadata(account, version_id)
162
        meta.update({'name': account, 'count': count, 'bytes': bytes})
170
        if user != account:
171
            meta = {'name': account}
172
        else:
173
            meta = self._get_metadata(account, version_id)
174
            meta.update({'name': account, 'count': count, 'bytes': bytes})
175
            if until is not None:
176
                meta.update({'until_timestamp': tstamp})
163 177
        if modified:
164 178
            meta.update({'modified': modified})
165
        if until is not None:
166
            meta.update({'until_timestamp': tstamp})
167 179
        return meta
168 180
    
169 181
    @backend_method
......
181 193
        
182 194
        logger.debug("get_account_groups: %s", account)
183 195
        if user != account:
184
            raise NotAllowedError
196
            if account not in self._allowed_accounts(user):
197
                raise NotAllowedError
198
            return {}
185 199
        return self._get_groups(account)
186 200
    
187 201
    @backend_method
......
229 243
        
230 244
        logger.debug("list_containers: %s %s %s %s", account, marker, limit, until)
231 245
        if user != account:
232
            if until:
246
            if until or account not in self._allowed_accounts(user):
233 247
                raise NotAllowedError
234
            containers = self._allowed_containers(user, account)
235
            start = 0
236
            if marker:
237
                try:
238
                    start = containers.index(marker) + 1
239
                except ValueError:
240
                    pass
241
            if not limit or limit > 10000:
242
                limit = 10000
243
            return containers[start:start + limit]
244
        return self._list_objects(account, '', '/', marker, limit, False, [], until)
248
            allowed = self._allowed_containers(user, account)
249
            start, limit = self._list_limits(allowed, marker, limit)
250
            return allowed[start:start + limit]
251
        return [x[0] for x in self._list_objects(account, '', '/', marker, limit, False, [], until)]
245 252
    
246 253
    @backend_method
247 254
    def get_container_meta(self, user, account, container, until=None):
......
249 256
        
250 257
        logger.debug("get_container_meta: %s %s %s", account, container, until)
251 258
        if user != account:
252
            raise NotAllowedError
259
            if until or container not in self._allowed_containers(user, account):
260
                raise NotAllowedError
253 261
        path, version_id, mtime = self._get_containerinfo(account, container, until)
254 262
        count, bytes, tstamp = self._get_pathstats(path, until)
255 263
        if mtime > tstamp:
......
261 269
            if mtime > modified:
262 270
                modified = mtime
263 271
        
264
        meta = self._get_metadata(path, version_id)
265
        meta.update({'name': container, 'count': count, 'bytes': bytes, 'modified': modified})
266
        if until is not None:
267
            meta.update({'until_timestamp': tstamp})
272
        if user != account:
273
            meta = {'name': container, 'modified': modified}
274
        else:
275
            meta = self._get_metadata(path, version_id)
276
            meta.update({'name': container, 'count': count, 'bytes': bytes, 'modified': modified})
277
            if until is not None:
278
                meta.update({'until_timestamp': tstamp})
268 279
        return meta
269 280
    
270 281
    @backend_method
......
283 294
        
284 295
        logger.debug("get_container_policy: %s %s", account, container)
285 296
        if user != account:
286
            raise NotAllowedError
297
            if container not in self._allowed_containers(user, account):
298
                raise NotAllowedError
299
            return {}
287 300
        path = self._get_containerinfo(account, container)[0]
288 301
        return self._get_policy(path)
289 302
    
......
360 373
        """Return a list of objects existing under a container."""
361 374
        
362 375
        logger.debug("list_objects: %s %s %s %s %s %s %s %s %s", account, container, prefix, delimiter, marker, limit, virtual, keys, until)
376
        allowed = []
363 377
        if user != account:
364
            raise NotAllowedError
378
            if until:
379
                raise NotAllowedError
380
            allowed = self._allowed_paths(user, os.path.join(account, container))
381
            if not allowed:
382
                raise NotAllowedError
365 383
        path, version_id, mtime = self._get_containerinfo(account, container, until)
366
        return self._list_objects(path, prefix, delimiter, marker, limit, virtual, keys, until)
384
        return self._list_objects(path, prefix, delimiter, marker, limit, virtual, keys, until, allowed)
367 385
    
368 386
    @backend_method
369 387
    def list_object_meta(self, user, account, container, until=None):
370 388
        """Return a list with all the container's object meta keys."""
371 389
        
372 390
        logger.debug("list_object_meta: %s %s %s", account, container, until)
391
        allowed = []
373 392
        if user != account:
374
            raise NotAllowedError
393
            if until:
394
                raise NotAllowedError
395
            allowed = self._allowed_paths(user, os.path.join(account, container))
396
            if not allowed:
397
                raise NotAllowedError
375 398
        path, version_id, mtime = self._get_containerinfo(account, container, until)
376 399
        sql = '''select distinct m.key from (%s) o, metadata m
377 400
                    where m.version_id = o.version_id and o.name like ?'''
378 401
        sql = sql % self._sql_until(until)
379
        c = self.con.execute(sql, (path + '/%',))
402
        param = (path + '/%',)
403
        if allowed:
404
            for x in allowed:
405
                sql += ' and o.name like ?'
406
                param += (x,)
407
        c = self.con.execute(sql, param)
380 408
        return [x[0] for x in c.fetchall()]
381 409
    
382 410
    @backend_method
......
723 751
        c = self.con.execute(sql, (path,))
724 752
        return dict(c.fetchall())
725 753
    
726
    def _list_objects(self, path, prefix='', delimiter=None, marker=None, limit=10000, virtual=True, keys=[], until=None):
754
    
755
    def _list_limits(self, listing, marker, limit):
756
        start = 0
757
        if marker:
758
            try:
759
                start = listing.index(marker) + 1
760
            except ValueError:
761
                pass
762
        if not limit or limit > 10000:
763
            limit = 10000
764
        return start, limit
765
    
766
    def _list_objects(self, path, prefix='', delimiter=None, marker=None, limit=10000, virtual=True, keys=[], until=None, allowed=[]):
727 767
        cont_prefix = path + '/'
728 768
        if keys and len(keys) > 0:
729 769
            sql = '''select distinct o.name, o.version_id from (%s) o, metadata m where o.name like ? and
730
                        m.version_id = o.version_id and m.key in (%s) order by o.name'''
770
                        m.version_id = o.version_id and m.key in (%s)'''
731 771
            sql = sql % (self._sql_until(until), ', '.join('?' * len(keys)))
732 772
            param = (cont_prefix + prefix + '%',) + tuple(keys)
773
            if allowed:
774
                for x in allowed:
775
                    sql += ' and o.name like ?'
776
                    param += (x,)
777
            sql += ' order by o.name'
733 778
        else:
734
            sql = 'select name, version_id from (%s) where name like ? order by name'
779
            sql = 'select name, version_id from (%s) where name like ?'
735 780
            sql = sql % self._sql_until(until)
736 781
            param = (cont_prefix + prefix + '%',)
782
            if allowed:
783
                for x in allowed:
784
                    sql += ' and name like ?'
785
                    param += (x,)
786
            sql += ' order by name'
737 787
        c = self.con.execute(sql, param)
738 788
        objects = [(x[0][len(cont_prefix):], x[1]) for x in c.fetchall()]
739 789
        if delimiter:
......
757 807
                            pseudo_objects.append((pseudo_name, None))
758 808
            objects = pseudo_objects
759 809
        
760
        start = 0
761
        if marker:
762
            try:
763
                start = [x[0] for x in objects].index(marker) + 1
764
            except ValueError:
765
                pass
766
        if not limit or limit > 10000:
767
            limit = 10000
810
        start, limit = self._list_limits([x[0] for x in objects], marker, limit)
768 811
        return objects[start:start + limit]
769 812
    
770 813
    def _del_version(self, version):

Also available in: Unified diff