Client Library
-=============
+==============
-.. automodule:: pithos.lib.client
\ No newline at end of file
+.. automodule:: pithos.lib.client
========================= ================================
Revision Description
========================= ================================
-0.5 (July 16, 2011) Object update from another object's data.
+0.5 (July 19, 2011) Object update from another object's data.
\ Support object truncate.
\ Create object using a standard HTML form.
\ Purge container/object history.
+\ List other accounts that share objects with a user.
0.4 (July 01, 2011) Object permissions and account groups.
\ Control versioning behavior and container quotas with container policy directives.
\ Support updating/deleting individual metadata with ``POST``.
========= ==================
Operation Description
========= ==================
-GET Authentication. This is kept for compatibility with the OOS API
+GET Authentication (for compatibility with the OOS API) or list allowed accounts
========= ==================
GET
204 (No Content) The request succeeded
================ =====================
+If an ``X-Auth-Token`` is already present, the operation will be interpreted as a request to list other accounts that share objects to the user.
+
+====================== =========================
+Request Parameter Name Value
+====================== =========================
+limit The amount of results requested (default is 10000)
+marker Return containers with name lexicographically after marker
+format Optional extended reply type (can be ``json`` or ``xml``)
+====================== =========================
+
+The reply is a list of account names.
+If a ``format=xml`` or ``format=json`` argument is given, extended information on the containers will be returned, serialized in the chosen format.
+For each account, the information will include the following (names will be in lower case and with hyphens replaced with underscores):
+
+=========================== ============================
+Name Description
+=========================== ============================
+name The name of the account
+last_modified The last container modification date (regardless of ``until``)
+=========================== ============================
+
+Example ``format=json`` reply:
+
+::
+
+ [{"name": "user", "last_modified": "2011-07-19T10:48:16"}, ...]
+
+Example ``format=xml`` reply:
+
+::
+
+ <?xml version="1.0" encoding="UTF-8"?>
+ <accounts>
+ <account>
+ <name>user</name>
+ <last_modified>2011-07-19T10:48:16</last_modified>
+ </account>
+ <account>...</account>
+ </accounts>
+
+=========================== =====================
+Return Code Description
+=========================== =====================
+200 (OK) The request succeeded
+204 (No Content) The account has no containers (only for non-extended replies)
+=========================== =====================
+
+Will use a ``200`` return code if the reply is of type json/xml.
Account Level
^^^^^^^^^^^^^
until Optional timestamp
====================== ===================================
-|
+Cross-user requests are not allowed to use ``until`` and only include the account modification date in the reply.
========================== =====================
Reply Header Name Value
====================== =========================
The reply is a list of container names. Account headers (as in a ``HEAD`` request) will also be included.
+Cross-user requests are not allowed to use ``until`` and only include the account/container modification dates in the reply.
+
If a ``format=xml`` or ``format=json`` argument is given, extended information on the containers will be returned, serialized in the chosen format.
For each container, the information will include all container metadata (names will be in lower case and with hyphens replaced with underscores):
until Optional timestamp
====================== ===================================
-|
+Cross-user requests are not allowed to use ``until`` and only include the container modification date in the reply.
=========================== ===============================
Reply Header Name Value
The keys given with ``meta`` will be matched with the strings after the ``X-Object-Meta-`` prefix.
The reply is a list of object names. Container headers (as in a ``HEAD`` request) will also be included.
+Cross-user requests are not allowed to use ``until`` and include the following limited set of headers in the reply:
+
+=========================== ===============================
+Reply Header Name Value
+=========================== ===============================
+X-Container-Block-Size The block size used by the storage backend
+X-Container-Block-Hash The hash algorithm used for block identifiers in object hashmaps
+X-Container-Object-Meta A list with all meta keys used by allowed objects (**TBD**)
+Last-Modified The last container modification date
+=========================== ===============================
+
If a ``format=xml`` or ``format=json`` argument is given, extended information on the objects will be returned, serialized in the chosen format.
For each object, the information will include all object metadata (names will be in lower case and with hyphens replaced with underscores):
Read and write control in Pithos is managed by setting appropriate permissions with the ``X-Object-Sharing`` header. The permissions are applied using prefix-based inheritance. Thus, each set of authorization directives is applied to all objects sharing the same prefix with the object where the corresponding ``X-Object-Sharing`` header is defined. For simplicity, nested/overlapping permissions are not allowed. Setting ``X-Object-Sharing`` will fail, if the object is already "covered", or another object with a longer common-prefix name already has permissions. When retrieving an object, the ``X-Object-Shared-By`` header reports where it gets its permissions from. If not present, the object is the actual source of authorization directives.
-Objects that are marked as public, via the ``X-Object-Public`` meta, are also available at the corresponding URI returned for ``HEAD`` or ``GET``. Requests for public objects do not need to include an ``X-Auth-Token``. Pithos will ignore request parameters and only include the following headers in the reply (all ``X-Object-*`` meta is hidden).
+A user may ``GET`` another account or container. The result will include a limited reply, containing only the allowed containers or objects respectively. A top-level request with an authentication token, will return a list of allowed accounts, so the user can easily find out which other users share objects.
+
+Objects that are marked as public, via the ``X-Object-Public`` meta, are also available at the corresponding URI returned for ``HEAD`` or ``GET``. Requests for public objects do not need to include an ``X-Auth-Token``. Pithos will ignore request parameters and only include the following headers in the reply (all ``X-Object-*`` meta is hidden):
========================== ===============================
Reply Header Name Value
Content-Disposition The presentation style of the object (optional)
========================== ===============================
+Public objects are not included and do not influence cross-user listings. They are, however, readable by all users.
+
Summary
^^^^^^^
* Object ``MOVE`` support.
* Time-variant account/container listings via the ``until`` parameter.
* Object versions - parameter ``version`` in ``HEAD``/``GET`` (list versions with ``GET``), ``X-Object-Version-*`` meta in replies, ``X-Source-Version`` in ``PUT``/``COPY``.
-* Sharing/publishing with ``X-Object-Sharing``, ``X-Object-Public`` at the object level. Permissions may include groups defined with ``X-Account-Group-*`` at the account level. These apply to the object - not its versions.
+* Sharing/publishing with ``X-Object-Sharing``, ``X-Object-Public`` at the object level. Cross-user operations are allowed - controlled by sharing directives. Permissions may include groups defined with ``X-Account-Group-*`` at the account level. These apply to the object - not its versions.
* Support for prefix-based inheritance when enforcing permissions. Parent object carrying the authorization directives is reported in ``X-Object-Shared-By``.
* Large object support with ``X-Object-Manifest``.
* Trace the user that created/modified an object with ``X-Object-Modified-By``.
def top_demux(request):
if request.method == 'GET':
+ if request.user:
+ return account_list(request)
return authenticate(request)
else:
return method_not_allowed(request)
x_auth_user)
return response
+@api_method('GET', format_allowed=True)
+def account_list(request):
+ # Normal Response Codes: 200, 204
+ # Error Response Codes: serviceUnavailable (503),
+ # badRequest (400)
+
+ response = HttpResponse()
+
+ marker = request.GET.get('marker')
+ limit = get_int_parameter(request.GET.get('limit'))
+ if not limit:
+ limit = 10000
+
+ accounts = backend.list_accounts(request.user, marker, limit)
+
+ if request.serialization == 'text':
+ if len(accounts) == 0:
+ # The cloudfiles python bindings expect 200 if json/xml.
+ response.status_code = 204
+ return response
+ response.status_code = 200
+ response.content = '\n'.join(accounts) + '\n'
+ return response
+
+ account_meta = []
+ for x in accounts:
+ try:
+ meta = backend.get_account_meta(request.user, x)
+ groups = backend.get_account_groups(request.user, x)
+ except NotAllowedError:
+ raise Unauthorized('Access denied')
+ else:
+ for k, v in groups.iteritems():
+ meta['X-Container-Group-' + k] = ','.join(v)
+ account_meta.append(printable_header_dict(meta))
+ if request.serialization == 'xml':
+ data = render_to_string('accounts.xml', {'accounts': account_meta})
+ elif request.serialization == 'json':
+ data = json.dumps(account_meta)
+ response.status_code = 200
+ response.content = data
+ return response
+
@api_method('HEAD')
def account_meta(request, v_account):
# Normal Response Codes: 204
put_account_headers(response, meta, groups)
marker = request.GET.get('marker')
- limit = request.GET.get('limit')
- if limit:
- try:
- limit = int(limit)
- if limit <= 0:
- raise ValueError
- except ValueError:
- limit = 10000
+ limit = get_int_parameter(request.GET.get('limit'))
+ if not limit:
+ limit = 10000
try:
containers = backend.list_containers(request.user, v_account, marker, limit, until)
response.status_code = 204
return response
response.status_code = 200
- response.content = '\n'.join([x[0] for x in containers]) + '\n'
+ response.content = '\n'.join(containers) + '\n'
return response
container_meta = []
for x in containers:
- if x[1] is not None:
- try:
- meta = backend.get_container_meta(request.user, v_account, x[0], until)
- policy = backend.get_container_policy(request.user, v_account, x[0])
- except NotAllowedError:
- raise Unauthorized('Access denied')
- except NameError:
- pass
- else:
- for k, v in policy.iteritems():
- meta['X-Container-Policy-' + k] = v
- container_meta.append(printable_header_dict(meta))
+ try:
+ meta = backend.get_container_meta(request.user, v_account, x, until)
+ policy = backend.get_container_policy(request.user, v_account, x)
+ except NotAllowedError:
+ raise Unauthorized('Access denied')
+ except NameError:
+ pass
+ else:
+ for k, v in policy.iteritems():
+ meta['X-Container-Policy-' + k] = v
+ container_meta.append(printable_header_dict(meta))
if request.serialization == 'xml':
data = render_to_string('containers.xml', {'account': v_account, 'containers': container_meta})
elif request.serialization == 'json':
prefix = prefix.lstrip('/')
marker = request.GET.get('marker')
- limit = request.GET.get('limit')
- if limit:
- try:
- limit = int(limit)
- if limit <= 0:
- raise ValueError
- except ValueError:
- limit = 10000
+ limit = get_int_parameter(request.GET.get('limit'))
+ if not limit:
+ limit = 10000
keys = request.GET.get('meta')
if keys:
return meta, groups
def put_account_headers(response, meta, groups):
- response['X-Account-Container-Count'] = meta['count']
- response['X-Account-Bytes-Used'] = meta['bytes']
+ if 'count' in meta:
+ response['X-Account-Container-Count'] = meta['count']
+ if 'bytes' in meta:
+ response['X-Account-Bytes-Used'] = meta['bytes']
if 'modified' in meta:
response['Last-Modified'] = http_date(int(meta['modified']))
for k in [x for x in meta.keys() if x.startswith('X-Account-Meta-')]:
return meta, policy
def put_container_headers(response, meta, policy):
- response['X-Container-Object-Count'] = meta['count']
- response['X-Container-Bytes-Used'] = meta['bytes']
+ if 'count' in meta:
+ response['X-Container-Object-Count'] = meta['count']
+ if 'bytes' in meta:
+ response['X-Container-Bytes-Used'] = meta['bytes']
response['Last-Modified'] = http_date(int(meta['modified']))
for k in [x for x in meta.keys() if x.startswith('X-Container-Meta-')]:
response[k.encode('utf-8')] = meta[k].encode('utf-8')
'block_size': Suggested is 4MB
"""
+ def list_accounts(self, user, marker=None, limit=10000):
+ """Return a list of accounts the user can access.
+
+ Parameters:
+ 'marker': Start list from the next item after 'marker'
+ 'limit': Number of containers to return
+ """
+ return []
+
def get_account_meta(self, user, account, until=None):
"""Return a dictionary with the account metadata.
return
def list_containers(self, user, account, marker=None, limit=10000, until=None):
- """Return a list of container (name, version_id) tuples existing under an account.
+ """Return a list of container names existing under an account.
Parameters:
'marker': Start list from the next item after 'marker'
self.mapper = Mapper(**params)
@backend_method
+ def list_accounts(self, user, marker=None, limit=10000):
+ """Return a list of accounts the user can access."""
+
+ allowed = self._allowed_accounts(user)
+ start, limit = self._list_limits(allowed, marker, limit)
+ return allowed[start:start + limit]
+
+ @backend_method
def get_account_meta(self, user, account, until=None):
"""Return a dictionary with the account metadata."""
logger.debug("get_account_meta: %s %s", account, until)
if user != account:
- raise NotAllowedError
+ if until or account not in self._allowed_accounts(user):
+ raise NotAllowedError
try:
version_id, mtime = self._get_accountinfo(account, until)
except NameError:
row = c.fetchone()
count = row[0]
- meta = self._get_metadata(account, version_id)
- meta.update({'name': account, 'count': count, 'bytes': bytes})
+ if user != account:
+ meta = {'name': account}
+ else:
+ meta = self._get_metadata(account, version_id)
+ meta.update({'name': account, 'count': count, 'bytes': bytes})
+ if until is not None:
+ meta.update({'until_timestamp': tstamp})
if modified:
meta.update({'modified': modified})
- if until is not None:
- meta.update({'until_timestamp': tstamp})
return meta
@backend_method
logger.debug("get_account_groups: %s", account)
if user != account:
- raise NotAllowedError
+ if account not in self._allowed_accounts(user):
+ raise NotAllowedError
+ return {}
return self._get_groups(account)
@backend_method
logger.debug("list_containers: %s %s %s %s", account, marker, limit, until)
if user != account:
- if until:
+ if until or account not in self._allowed_accounts(user):
raise NotAllowedError
- containers = self._allowed_containers(user, account)
- start = 0
- if marker:
- try:
- start = containers.index(marker) + 1
- except ValueError:
- pass
- if not limit or limit > 10000:
- limit = 10000
- return containers[start:start + limit]
- return self._list_objects(account, '', '/', marker, limit, False, [], until)
+ allowed = self._allowed_containers(user, account)
+ start, limit = self._list_limits(allowed, marker, limit)
+ return allowed[start:start + limit]
+ return [x[0] for x in self._list_objects(account, '', '/', marker, limit, False, [], until)]
@backend_method
def get_container_meta(self, user, account, container, until=None):
logger.debug("get_container_meta: %s %s %s", account, container, until)
if user != account:
- raise NotAllowedError
+ if until or container not in self._allowed_containers(user, account):
+ raise NotAllowedError
path, version_id, mtime = self._get_containerinfo(account, container, until)
count, bytes, tstamp = self._get_pathstats(path, until)
if mtime > tstamp:
if mtime > modified:
modified = mtime
- meta = self._get_metadata(path, version_id)
- meta.update({'name': container, 'count': count, 'bytes': bytes, 'modified': modified})
- if until is not None:
- meta.update({'until_timestamp': tstamp})
+ if user != account:
+ meta = {'name': container, 'modified': modified}
+ else:
+ meta = self._get_metadata(path, version_id)
+ meta.update({'name': container, 'count': count, 'bytes': bytes, 'modified': modified})
+ if until is not None:
+ meta.update({'until_timestamp': tstamp})
return meta
@backend_method
logger.debug("get_container_policy: %s %s", account, container)
if user != account:
- raise NotAllowedError
+ if container not in self._allowed_containers(user, account):
+ raise NotAllowedError
+ return {}
path = self._get_containerinfo(account, container)[0]
return self._get_policy(path)
"""Return a list of objects existing under a container."""
logger.debug("list_objects: %s %s %s %s %s %s %s %s %s", account, container, prefix, delimiter, marker, limit, virtual, keys, until)
+ allowed = []
if user != account:
- raise NotAllowedError
+ if until:
+ raise NotAllowedError
+ allowed = self._allowed_paths(user, os.path.join(account, container))
+ if not allowed:
+ raise NotAllowedError
path, version_id, mtime = self._get_containerinfo(account, container, until)
- return self._list_objects(path, prefix, delimiter, marker, limit, virtual, keys, until)
+ return self._list_objects(path, prefix, delimiter, marker, limit, virtual, keys, until, allowed)
@backend_method
def list_object_meta(self, user, account, container, until=None):
"""Return a list with all the container's object meta keys."""
logger.debug("list_object_meta: %s %s %s", account, container, until)
+ allowed = []
if user != account:
- raise NotAllowedError
+ if until:
+ raise NotAllowedError
+ allowed = self._allowed_paths(user, os.path.join(account, container))
+ if not allowed:
+ raise NotAllowedError
path, version_id, mtime = self._get_containerinfo(account, container, until)
sql = '''select distinct m.key from (%s) o, metadata m
where m.version_id = o.version_id and o.name like ?'''
sql = sql % self._sql_until(until)
- c = self.con.execute(sql, (path + '/%',))
+ param = (path + '/%',)
+ if allowed:
+ for x in allowed:
+ sql += ' and o.name like ?'
+ param += (x,)
+ c = self.con.execute(sql, param)
return [x[0] for x in c.fetchall()]
@backend_method
c = self.con.execute(sql, (path,))
return dict(c.fetchall())
- def _list_objects(self, path, prefix='', delimiter=None, marker=None, limit=10000, virtual=True, keys=[], until=None):
+
+ def _list_limits(self, listing, marker, limit):
+ start = 0
+ if marker:
+ try:
+ start = listing.index(marker) + 1
+ except ValueError:
+ pass
+ if not limit or limit > 10000:
+ limit = 10000
+ return start, limit
+
+ def _list_objects(self, path, prefix='', delimiter=None, marker=None, limit=10000, virtual=True, keys=[], until=None, allowed=[]):
cont_prefix = path + '/'
if keys and len(keys) > 0:
sql = '''select distinct o.name, o.version_id from (%s) o, metadata m where o.name like ? and
- m.version_id = o.version_id and m.key in (%s) order by o.name'''
+ m.version_id = o.version_id and m.key in (%s)'''
sql = sql % (self._sql_until(until), ', '.join('?' * len(keys)))
param = (cont_prefix + prefix + '%',) + tuple(keys)
+ if allowed:
+ for x in allowed:
+ sql += ' and o.name like ?'
+ param += (x,)
+ sql += ' order by o.name'
else:
- sql = 'select name, version_id from (%s) where name like ? order by name'
+ sql = 'select name, version_id from (%s) where name like ?'
sql = sql % self._sql_until(until)
param = (cont_prefix + prefix + '%',)
+ if allowed:
+ for x in allowed:
+ sql += ' and name like ?'
+ param += (x,)
+ sql += ' order by name'
c = self.con.execute(sql, param)
objects = [(x[0][len(cont_prefix):], x[1]) for x in c.fetchall()]
if delimiter:
pseudo_objects.append((pseudo_name, None))
objects = pseudo_objects
- start = 0
- if marker:
- try:
- start = [x[0] for x in objects].index(marker) + 1
- except ValueError:
- pass
- if not limit or limit > 10000:
- limit = 10000
+ start, limit = self._list_limits([x[0] for x in objects], marker, limit)
return objects[start:start + limit]
def _del_version(self, version):
projects / pithos / commitdiff