Synnefo

This is where the docs will be built.

Administrator Guide

===================

Simple Setup

------------

Assuming a clean debian squeeze (stable) installation, use the following steps to run the software.

Install packages::

  apt-get install git python-django python-setuptools python-sphinx

  apt-get install python-sqlalchemy python-mysqldb python-psycopg2

  apt-get install apache2 libapache2-mod-wsgi

Get the source::

  cd /

  git clone https://code.grnet.gr/git/pithos

Setup the files::

  cd /pithos/pithos

  python manage.py syncdb

  cd /pithos

  python setup.py build_sphinx

It is advised that you create a ``settings.local`` file to place any configuration overrides (at least change ``SECRET_KEY``).

Edit ``/etc/apache2/sites-available/pithos`` (change the ``ServerName`` directive)::

  <VirtualHost *:80>

    ServerAdmin webmaster@pithos.dev.grnet.gr

    ServerName pithos.dev.grnet.gr

    DocumentRoot /pithos/htdocs

    Alias /ui "/var/www/pithos_web_client"

    Alias /docs "/pithos/docs/build/html"

    <Directory />

        Options Indexes FollowSymLinks

        AllowOverride None

        Order allow,deny

        Allow from all

    </Directory>

    SetEnv no-gzip

    SetEnv dont-vary

    RewriteEngine On

    RewriteRule ^/v(.*) /api/v$1 [PT,NE]

    RewriteRule ^/public(.*) /api/public$1 [PT,NE]

    RewriteRule ^/tools(.*) /api/ui$1 [PT,NE]

    RewriteRule ^/im(.*) https://%{HTTP_HOST}%{REQUEST_URI} [NE]

    RewriteRule ^/login(.*) https://%{HTTP_HOST}%{REQUEST_URI} [NE]

    RequestHeader set X-Forwarded-Protocol "http"

    WSGIScriptAlias /api /pithos/pithos/wsgi/pithos.wsgi

    # WSGIDaemonProcess pithos

    # WSGIProcessGroup pithos

    LogLevel warn

    ErrorLog ${APACHE_LOG_DIR}/pithos.error.log

    CustomLog ${APACHE_LOG_DIR}/pithos.access.log combined

  </VirtualHost>

To disable non-SSL connections, ``/etc/apache2/sites-available/pithos`` should be::

  <VirtualHost *:80>

    ServerAdmin webmaster@pithos.dev.grnet.gr

    ServerName pithos.dev.grnet.gr

    RewriteEngine On

    RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}

  </VirtualHost>

Edit ``/etc/apache2/sites-available/pithos-ssl`` (assuming files in ``/etc/ssl/private/pithos.dev.grnet.gr.key`` and ``/etc/ssl/certs/pithos.dev.grnet.gr.crt`` - change the ``ServerName`` directive)::

  <IfModule mod_ssl.c>

  <VirtualHost _default_:443>

    ServerAdmin webmaster@pithos.dev.grnet.gr

    ServerName pithos.dev.grnet.gr

    DocumentRoot /pithos/htdocs

    Alias /ui "/var/www/pithos_web_client"

    Alias /docs "/pithos/docs/build/html"

    <Directory />

        Options Indexes FollowSymLinks

        AllowOverride None

        Order allow,deny

        Allow from all

    </Directory>

    SetEnv no-gzip

    SetEnv dont-vary

    RewriteEngine On

    RewriteRule ^/v(.*) /api/v$1 [PT,NE]

    RewriteRule ^/public(.*) /api/public$1 [PT,NE]

    RewriteRule ^/tools(.*) /api/ui$1 [PT,NE]

    RewriteRule ^/im(.*) /api/im$1 [PT,NE]

    RewriteRule ^/login(.*) /api/im/login/dummy$1 [PT,NE]

    RequestHeader set X-Forwarded-Protocol "https"

    WSGIScriptAlias /api /pithos/pithos/wsgi/pithos.wsgi

    # WSGIDaemonProcess pithos

    # WSGIProcessGroup pithos

    LogLevel warn

    ErrorLog ${APACHE_LOG_DIR}/pithos.error.log

    CustomLog ${APACHE_LOG_DIR}/pithos.access.log combined

    SSLEngine on

    SSLCertificateFile    /etc/ssl/certs/pithos.dev.grnet.gr.crt

    SSLCertificateKeyFile /etc/ssl/private/pithos.dev.grnet.gr.key

  </VirtualHost>

  </IfModule>

Add in ``/etc/apache2/mods-available/wsgi.conf``::

  WSGIChunkedRequest On

Make sure the data folder is writable by the web server user::

  chown -R www-data:www-data /pithos/pithos/data

If using an SQLite database, the same goes for the database file and the containing folder::

  chown www-data:www-data /pithos/pithos/

  chown www-data:www-data /pithos/pithos/backend.db

Configure and run apache::

  a2enmod ssl

  a2enmod rewrite

  a2dissite default

  a2ensite pithos

  a2ensite pithos-ssl

  mkdir /var/www/pithos

  mkdir /var/www/pithos_web_client

  /etc/init.d/apache2 restart

Useful alias to add in ``~/.bashrc``::

  alias sync-pithos='cd /pithos && git pull && python setup.py build_sphinx && /etc/init.d/apache2 restart'

Gunicorn Setup

--------------

Add in ``/etc/apt/sources.list``::

  deb http://backports.debian.org/debian-backports squeeze-backports main

Then::

  apt-get update

  apt-get -t squeeze-backports install gunicorn

  apt-get -t squeeze-backports install python-gevent

Create ``/etc/gunicorn.d/pithos``::

  CONFIG = {

   'mode': 'django',

   'working_dir': '/pithos/pithos',

   'user': 'www-data',

   'group': 'www-data',

   'args': (

        '--bind=[::]:8080',

        '--worker-class=egg:gunicorn#gevent',

        '--workers=4',

        '--log-level=debug',

        '/pithos/pithos/settings.py',

   ),

  }

Replace the ``WSGI*`` directives in ``/etc/apache2/sites-available/pithos`` and ``/etc/apache2/sites-available/pithos-ssl`` with::

  <Proxy *>

    Order allow,deny

    Allow from all

  </Proxy>

  SetEnv                proxy-sendchunked

  SSLProxyEngine        off

  ProxyErrorOverride    off

  ProxyPass        /api http://localhost:8080 retry=0

  ProxyPassReverse /api http://localhost:8080

Make sure that in ``settings.local``::

  USE_X_FORWARDED_HOST = True

Configure and run::

  /etc/init.d/gunicorn restart

  a2enmod proxy

  a2enmod proxy_http

  /etc/init.d/apache2 restart

If experiencing timeout problems, try adding to ``/etc/gunicorn.d/pithos``::

        ...

        '--timeout=43200',

        ...

Shibboleth Setup

----------------

Install package::

  apt-get install libapache2-mod-shib2

Setup the files in ``/etc/shibboleth``.

Add in ``/etc/apache2/sites-available/pithos-ssl``::

  ShibConfig /etc/shibboleth/shibboleth2.xml

  Alias      /shibboleth-sp /usr/share/shibboleth 

  <Location /api/im/login/shibboleth>

    AuthType shibboleth

    ShibRequireSession On

    ShibUseHeaders On

    require valid-user

  </Location>

Configure and run apache::

  a2enmod shib2

  /etc/init.d/apache2 restart

  /etc/init.d/shibd restart

The following tokens should be available at the destination, after passing through the apache module::

  eppn # eduPersonPrincipalName

  Shib-InetOrgPerson-givenName

  Shib-Person-surname

  Shib-Person-commonName

  Shib-InetOrgPerson-displayName

  Shib-EP-Affiliation

  Shib-Session-ID

MySQL Setup

-----------

If using MySQL instead of SQLite for the database engine, consider the following.

Server side::

  apt-get install mysql-server

Add in ``/etc/mysql/conf.d/pithos.cnf``::

  [mysqld]

  sql-mode="NO_AUTO_VALUE_ON_ZERO"

Edit ``/etc/mysql/my.cnf`` to allow network connections and restart the server.

Create database and user::

  CREATE DATABASE pithos CHARACTER SET utf8 COLLATE utf8_bin;

  GRANT ALL ON pithos.* TO pithos@localhost IDENTIFIED BY 'password';

  GRANT ALL ON pithos.* TO pithos@'%' IDENTIFIED BY 'password';

Client side::

  apt-get install mysql-client

It helps to create a ``~/.my.cnf`` file, for automatically connecting to the server::

  [client]

  user = pithos

  password = 'password'

  host = pithos-storage.dev.grnet.gr

  [mysql]

  database = pithos

PostgreSQL Setup

----------------

If using PostgreSQL instead of SQLite for the database engine, consider the following.

Server side::

  apt-get install postgresql

Edit ``/etc/postgresql/8.4/main/postgresql.conf`` and ``/etc/postgresql/8.4/main/pg_hba.conf`` to allow network connections and restart the server.

Create database and user::

  CREATE DATABASE pithos WITH ENCODING 'UTF8' LC_COLLATE='C' LC_CTYPE='C' TEMPLATE=template0;

  CREATE USER pithos WITH PASSWORD 'password';

  GRANT ALL PRIVILEGES ON DATABASE pithos TO pithos;

Client side::

  apt-get install postgresql-client

It helps to create a ``~/.pgpass`` file, for automatically passing the password to the server::

  pithos-storage.dev.grnet.gr:5432:pithos:pithos:password

Connect with::

  psql -h pithos-storage.dev.grnet.gr -U pithos

Backends

========

.. toctree::

   :maxdepth: 3

BaseBackend

-----------

.. autoclass:: pithos.backends.base.BaseBackend

ModularBackend

--------------

.. autoclass:: pithos.backends.modular.ModularBackend

   :show-inheritance:

   :members:

   :inherited-members:

Node

~~~~

.. automodule:: pithos.backends.lib.sqlalchemy.node

   :show-inheritance:

   :members:

   :undoc-members:

Permissions

~~~~~~~~~~~

.. automodule:: pithos.backends.lib.sqlalchemy.permissions

   :show-inheritance:

   :members:

   :undoc-members:

Store

~~~~~

.. automodule:: pithos.backends.lib.hashfiler.store

   :show-inheritance:

   :members:

   :undoc-members:

Client Library

==============

.. automodule:: pithos.tools.lib.client

   :show-inheritance:

   :members:

   :undoc-members:

# -*- coding: utf-8 -*-

#

# Pithos documentation build configuration file, created by

# sphinx-quickstart on Wed May 18 12:42:48 2011.

#

# This file is execfile()d with the current directory set to its containing dir.

#

# Note that not all possible configuration values are present in this

# autogenerated file.

#

# All configuration values have a default; values that are commented out

# serve to show the default.

import sys, os

# If extensions (or modules to document with autodoc) are in another directory,

# add these directories to sys.path here. If the directory is relative to the

# documentation root, use os.path.abspath to make it absolute, like shown here.

sys.path.insert(0, os.path.abspath('../..'))

from synnefo import settings

from django.core.management import setup_environ

setup_environ(settings)

# -- General configuration -----------------------------------------------------

# If your documentation needs a minimal Sphinx version, state it here.

#needs_sphinx = '1.0'

# Add any Sphinx extension module names here, as strings. They can be extensions

# coming with Sphinx (named 'sphinx.ext.*') or your custom ones.

extensions = ['sphinx.ext.autodoc', 'sphinx.ext.todo']

# Add any paths that contain templates here, relative to this directory.

templates_path = ['_templates']

# The suffix of source filenames.

source_suffix = '.rst'

# The encoding of source files.

#source_encoding = 'utf-8-sig'

# The master toctree document.

master_doc = 'index'

# General information about the project.

project = u'Pithos'

copyright = u'2011, Pithos Storage Team'

# The version info for the project you're documenting, acts as replacement for

# |version| and |release|, also used in various other places throughout the

# built documents.

#

# The short X.Y version.

version = '2'

# The full version, including alpha/beta/rc tags.

release = '2'

# The language for content autogenerated by Sphinx. Refer to documentation

# for a list of supported languages.

#language = None

# There are two options for replacing |today|: either, you set today to some

# non-false value, then it is used:

#today = ''

# Else, today_fmt is used as the format for a strftime call.

#today_fmt = '%B %d, %Y'

# List of patterns, relative to source directory, that match files and

# directories to ignore when looking for source files.

exclude_patterns = ['_build']

# The reST default role (used for this markup: `text`) to use for all documents.

#default_role = None

# If true, '()' will be appended to :func: etc. cross-reference text.

#add_function_parentheses = True

# If true, the current module name will be prepended to all description

# unit titles (such as .. function::).

#add_module_names = True

# If true, sectionauthor and moduleauthor directives will be shown in the

# output. They are ignored by default.

#show_authors = False

# The name of the Pygments (syntax highlighting) style to use.

pygments_style = 'sphinx'

# A list of ignored prefixes for module index sorting.

#modindex_common_prefix = []

# -- Options for HTML output ---------------------------------------------------

# The theme to use for HTML and HTML Help pages.  See the documentation for

# a list of builtin themes.

html_theme = 'default'

# Theme options are theme-specific and customize the look and feel of a theme

# further.  For a list of options available for each theme, see the

# documentation.

#html_theme_options = {}

# Add any paths that contain custom themes here, relative to this directory.

#html_theme_path = []

# The name for this set of Sphinx documents.  If None, it defaults to

# "<project> v<release> documentation".

#html_title = None

# A shorter title for the navigation bar.  Default is the same as html_title.

#html_short_title = None

# The name of an image file (relative to this directory) to place at the top

# of the sidebar.

#html_logo = None

# The name of an image file (within the static path) to use as favicon of the

# docs.  This file should be a Windows icon file (.ico) being 16x16 or 32x32

# pixels large.

#html_favicon = None

# Add any paths that contain custom static files (such as style sheets) here,

# relative to this directory. They are copied after the builtin static files,

# so a file named "default.css" will overwrite the builtin "default.css".

#html_static_path = ['_static']

# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,

# using the given strftime format.

#html_last_updated_fmt = '%b %d, %Y'

# If true, SmartyPants will be used to convert quotes and dashes to

# typographically correct entities.

#html_use_smartypants = True

# Custom sidebar templates, maps document names to template names.

#html_sidebars = {}

# Additional templates that should be rendered to pages, maps page names to

# template names.

#html_additional_pages = {}

# If false, no module index is generated.

#html_domain_indices = True

# If false, no index is generated.

#html_use_index = True

# If true, the index is split into individual pages for each letter.

#html_split_index = False

# If true, links to the reST sources are added to the pages.

#html_show_sourcelink = True

# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.

#html_show_sphinx = True

# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.

#html_show_copyright = True

# If true, an OpenSearch description file will be output, and all pages will

# contain a <link> tag referring to it.  The value of this option must be the

# base URL from which the finished HTML is served.

#html_use_opensearch = ''

# This is the file name suffix for HTML files (e.g. ".xhtml").

#html_file_suffix = None

# Output file base name for HTML help builder.

htmlhelp_basename = 'Pithosdoc'

# -- Options for LaTeX output --------------------------------------------------

# The paper size ('letter' or 'a4').

#latex_paper_size = 'letter'

# The font size ('10pt', '11pt' or '12pt').

#latex_font_size = '10pt'

# Grouping the document tree into LaTeX files. List of tuples

# (source start file, target name, title, author, documentclass [howto/manual]).

latex_documents = [

  ('index', 'Pithos.tex', u'Pithos Documentation',

   u'Pithos Storage Team', 'manual'),

]

# The name of an image file (relative to this directory) to place at the top of

# the title page.

#latex_logo = None

# For "manual" documents, if this is true, then toplevel headings are parts,

# not chapters.

#latex_use_parts = False

# If true, show page references after internal links.

#latex_show_pagerefs = False

# If true, show URL addresses after external links.

#latex_show_urls = False

# Additional stuff for the LaTeX preamble.

#latex_preamble = ''

# Documents to append as an appendix to all manuals.

#latex_appendices = []

# If false, no module index is generated.

#latex_domain_indices = True

# -- Options for manual page output --------------------------------------------

# One entry per manual page. List of tuples

# (source start file, name, description, authors, manual section).

man_pages = [

    ('index', 'pithos', u'Pithos Documentation',

     [u'Pithos Storage Team'], 1)

]

autodoc_default_flags = ['members']

Pithos v2 Developer Guide

=========================

Introduction

------------

Pithos is a storage service implemented by GRNET (http://www.grnet.gr). Data is stored as objects, organized in containers, belonging to an account. This hierarchy of storage layers has been inspired by the OpenStack Object Storage (OOS) API and similar CloudFiles API by Rackspace. The Pithos API follows the OOS API as closely as possible. One of the design requirements has been to be able to use Pithos with clients built for the OOS, without changes.

However, to be able to take full advantage of the Pithos infrastructure, client software should be aware of the extensions that differentiate Pithos from OOS. Pithos objects can be updated, or appended to. Pithos will store sharing permissions per object and enforce corresponding authorization policies. Automatic version management, allows taking account and container listings back in time, as well as reading previous instances of objects.

The storage backend of Pithos is block oriented, permitting efficient, deduplicated data placement. The block structure of objects is exposed at the API layer, in order to encourage external software to implement advanced data management operations.

This document's goals are:

* Define the Pithos ReST API that allows the storage and retrieval of data and metadata via HTTP calls

* Specify metadata semantics and user interface guidelines for a common experience across client software implementations

The present document is meant to be read alongside the OOS API documentation. Thus, it is suggested that the reader is familiar with associated technologies, the OOS API as well as the first version of the Pithos API. This document refers to the second version of Pithos. Information on the first version of the storage API can be found at http://code.google.com/p/gss.

Whatever marked as to be determined (**TBD**), should not be considered by implementors.

More info about Pithos can be found here: https://code.grnet.gr/projects/pithos

Document Revisions

^^^^^^^^^^^^^^^^^^

=========================  ================================

Revision                   Description

=========================  ================================

0.9 (Feb 17, 2012)         Change permissions model.

\                          Do not include user-defined metadata in account/container/object listings.

0.8 (Jan 24, 2012)         Update allowed versioning values.

\                          Change policy/meta formatting in JSON/XML replies.

\                          Document that all non-ASCII characters in headers should be URL-encoded.

\                          Support metadata-based queries when listing objects at the container level.

\                          Note Content-Type issue when using the internal django web server.

\                          Add object UUID field.

\                          Always reply with the MD5 in the ETag.

\                          Note that ``/login`` will only work if an external authentication system is defined.

\                          Include option to ignore Content-Type on ``COPY``/``MOVE``.

\                          Use format parameter for conflict (409) and uploaded hash list (container level) replies.

0.7 (Nov 21, 2011)         Suggest upload/download methods using hashmaps.

\                          Propose syncing algorithm.

\                          Support cross-account object copy and move.

\                          Pass token as a request parameter when using ``POST`` via an HTML form.

\                          Optionally use source account to update object from another object.

\                          Use container ``POST`` to upload missing blocks of data.

\                          Report policy in account headers.

\                          Add insufficient quota reply.

\                          Use special meta to always report Merkle hash.

0.6 (Sept 13, 2011)        Reply with Merkle hash as the ETag when updating objects.

\                          Include version id in object replace/change replies.

\                          Change conflict (409) replies format to text.

\                          Tags should be migrated to a meta value.

\                          Container ``PUT`` updates metadata/policy.

\                          Report allowed actions in shared object replies.

\                          Provide ``https://hostname/login`` for Shibboleth authentication.

\                          Use ``hashmap`` parameter in object ``GET``/``PUT`` to use hashmaps.

0.5 (July 22, 2011)        Object update from another object's data.

\                          Support object truncate.

\                          Create object using a standard HTML form.

\                          Purge container/object history.

\                          List other accounts that share objects with a user.

\                          List shared containers/objects.

\                          Update implementation guidelines.

\                          Check preconditions when creating/updating objects.

0.4 (July 01, 2011)        Object permissions and account groups.

\                          Control versioning behavior and container quotas with container policy directives.

\                          Support updating/deleting individual metadata with ``POST``.

\                          Create object using hashmap.

0.3 (June 14, 2011)        Large object support with ``X-Object-Manifest``.

\                          Allow for publicly available objects via ``https://hostname/public``.

\                          Support time-variant account/container listings. 

\                          Add source version when duplicating with ``PUT``/``COPY``.

\                          Request version in object ``HEAD``/``GET`` requests (list versions with ``GET``).

0.2 (May 31, 2011)         Add object meta listing and filtering in containers.

\                          Include underlying storage characteristics in container meta.

\                          Support for partial object updates through ``POST``.

\                          Expose object hashmaps through ``GET``.

\                          Support for multi-range object ``GET`` requests.

0.1 (May 17, 2011)         Initial release. Based on OpenStack Object Storage Developer Guide API v1 (Apr. 15, 2011).

=========================  ================================

Pithos Users and Authentication

-------------------------------

In Pithos, each user is uniquely identified by a token. All API requests require a token and each token is internally resolved to an account string. The API uses the account string to identify the user's own files, thus whether a request is local or cross-account.

Pithos does not keep a user database. For development and testing purposes, user identifiers and their corresponding tokens can be defined in the settings file. However, Pithos is designed with an external authentication service in mind. This service must handle the details of validating user credentials and communicate with Pithos via a middleware software component that, given a token, fills in the internal request account variable.

Client software using Pithos, if not already knowing a user's identifier and token, should forward to the ``/login`` URI. The Pithos server, depending on its configuration will redirect to the appropriate login page.

The login URI accepts the following parameters:

======================  =========================

Request Parameter Name  Value

======================  =========================

next                    The URI to redirect to when the process is finished

renew                   Force token renewal (no value parameter)

force                   Force logout current user (no value parameter)

======================  =========================

When done with logging in, the service's login URI should redirect to the URI provided with ``next``, adding ``user`` and ``token`` parameters, which contain the account and token fields respectively.

A user management service that implements a login URI according to these conventions is Astakos (https://code.grnet.gr/projects/astakos), by GRNET.

The Pithos API

--------------

The URI requests supported by the Pithos API follow one of the following forms:

* Top level: ``https://hostname/v1/``

* Account level: ``https://hostname/v1/<account>``

* Container level: ``https://hostname/v1/<account>/<container>``

* Object level: ``https://hostname/v1/<account>/<container>/<object>``

All requests must include an ``X-Auth-Token`` - as a header, or a parameter.

The allowable request operations and respective return codes per level are presented in the remainder of this chapter. Common to all requests are the following return codes.

=========================  ================================

Return Code                Description

=========================  ================================

400 (Bad Request)          The request is invalid

401 (Unauthorized)         Missing or invalid token

403 (Forbidden)            Request not allowed

404 (Not Found)            The requested resource was not found

503 (Service Unavailable)  The request cannot be completed because of an internal error

=========================  ================================

Top Level

^^^^^^^^^

List of operations:

=========  ==================

Operation  Description

=========  ==================

GET        Authentication (for compatibility with the OOS API) or list allowed accounts

=========  ==================

GET

"""

If the ``X-Auth-User`` and ``X-Auth-Key`` headers are given, a dummy ``X-Auth-Token`` and ``X-Storage-Url`` will be replied, which can be used as a guest token/namespace for testing Pithos.

================  =====================

Return Code       Description

================  =====================

204 (No Content)  The request succeeded

================  =====================

If an ``X-Auth-Token`` is already present, the operation will be interpreted as a request to list other accounts that share objects to the user.

======================  =========================

Request Parameter Name  Value

======================  =========================

limit                   The amount of results requested (default is 10000)

marker                  Return containers with name lexicographically after marker

format                  Optional extended reply type (can be ``json`` or ``xml``)

======================  =========================

The reply is a list of account names.

If a ``format=xml`` or ``format=json`` argument is given, extended information on the accounts will be returned, serialized in the chosen format.

For each account, the information will include the following (names will be in lower case and with hyphens replaced with underscores):

===========================  ============================

Name                         Description

===========================  ============================

name                         The name of the account

last_modified                The last account modification date (regardless of ``until``)

===========================  ============================

Example ``format=json`` reply:

::

  [{"name": "user", "last_modified": "2011-12-02T08:10:41.565891+00:00"}, ...]

Example ``format=xml`` reply:

::

  <?xml version="1.0" encoding="UTF-8"?>

  <accounts>

    <account>

      <name>user</name>

      <last_modified>2011-12-02T08:10:41.565891+00:00</last_modified>

    </account>

    <account>...</account>

  </accounts>

===========================  =====================

Return Code                  Description

===========================  =====================

200 (OK)                     The request succeeded

204 (No Content)             The user has no access to other accounts (only for non-extended replies)

===========================  =====================

Will use a ``200`` return code if the reply is of type JSON/XML.

Account Level

^^^^^^^^^^^^^

List of operations:

=========  ==================

Operation  Description

=========  ==================

HEAD       Retrieve account metadata

GET        List containers

POST       Update account metadata

=========  ==================

HEAD

""""

====================  ===========================

Request Header Name   Value

====================  ===========================

If-Modified-Since     Retrieve if account has changed since provided timestamp

If-Unmodified-Since   Retrieve if account has not changed since provided timestamp

====================  ===========================

|

======================  ===================================

Request Parameter Name  Value

======================  ===================================

until                   Optional timestamp

======================  ===================================

Cross-user requests are not allowed to use ``until`` and only include the account modification date in the reply.

==========================  =====================

Reply Header Name           Value

==========================  =====================

X-Account-Container-Count   The total number of containers

X-Account-Bytes-Used        The total number of bytes stored

X-Account-Until-Timestamp   The last account modification date until the timestamp provided

X-Account-Group-*           Optional user defined groups

X-Account-Policy-*          Account behavior and limits

X-Account-Meta-*            Optional user defined metadata

Last-Modified               The last account modification date (regardless of ``until``)

==========================  =====================

|

================  =====================

Return Code       Description

================  =====================

204 (No Content)  The request succeeded

================  =====================

GET

"""

====================  ===========================

Request Header Name   Value

====================  ===========================

If-Modified-Since     Retrieve if account has changed since provided timestamp

If-Unmodified-Since   Retrieve if account has not changed since provided timestamp

====================  ===========================

|

======================  =========================

Request Parameter Name  Value

======================  =========================

limit                   The amount of results requested (default is 10000)

marker                  Return containers with name lexicographically after marker

format                  Optional extended reply type (can be ``json`` or ``xml``)

shared                  Show only shared containers (no value parameter)

until                   Optional timestamp

======================  =========================

The reply is a list of container names. Account headers (as in a ``HEAD`` request) will also be included.

Cross-user requests are not allowed to use ``until`` and only include the account/container modification dates in the reply.

If a ``format=xml`` or ``format=json`` argument is given, extended information on the containers will be returned, serialized in the chosen format.

For each container, the information will include all container metadata, except user-defined (names will be in lower case and with hyphens replaced with underscores):

===========================  ============================

Name                         Description

===========================  ============================

name                         The name of the container

count                        The number of objects inside the container

bytes                        The total size of the objects inside the container

last_modified                The last container modification date (regardless of ``until``)

x_container_until_timestamp  The last container modification date until the timestamp provided

x_container_policy           Container behavior and limits

===========================  ============================

Example ``format=json`` reply:

::

  [{"name": "pithos",

    "bytes": 62452,

    "count": 8374,

    "last_modified": "2011-12-02T08:10:41.565891+00:00",

    "x_container_policy": {"quota": "53687091200", "versioning": "auto"}}, ...]

Example ``format=xml`` reply:

::

  <?xml version="1.0" encoding="UTF-8"?>

  <account name="user">

    <container>

      <name>pithos</name>

      <bytes>62452</bytes>

      <count>8374</count>

      <last_modified>2011-12-02T08:10:41.565891+00:00</last_modified>

      <x_container_policy>

        <key>quota</key><value>53687091200</value>

        <key>versioning</key><value>auto</value>

      </x_container_policy>

    </container>

    <container>...</container>

  </account>

For more examples of container details returned in JSON/XML formats refer to the OOS API documentation. In addition to the OOS API, Pithos returns policy fields, grouped as key-value pairs.

===========================  =====================

Return Code                  Description

===========================  =====================

200 (OK)                     The request succeeded

204 (No Content)             The account has no containers (only for non-extended replies)

304 (Not Modified)           The account has not been modified

412 (Precondition Failed)    The condition set can not be satisfied

===========================  =====================

Will use a ``200`` return code if the reply is of type JSON/XML.

POST

""""

====================  ===========================

Request Header Name   Value

====================  ===========================

X-Account-Group-*     Optional user defined groups

X-Account-Meta-*      Optional user defined metadata

====================  ===========================

|

======================  ============================================

Request Parameter Name  Value

======================  ============================================

update                  Do not replace metadata/groups (no value parameter)

======================  ============================================

No reply content/headers.

The operation will overwrite all user defined metadata, except if ``update`` is defined.

To create a group, include an ``X-Account-Group-*`` header with the name in the key and a comma separated list of user names in the value. If no ``X-Account-Group-*`` header is present, no changes will be applied to groups. The ``update`` parameter also applies to groups. To delete a specific group, use ``update`` and an empty header value.

================  ===============================

Return Code       Description

================  ===============================

202 (Accepted)    The request has been accepted

================  ===============================

Container Level

^^^^^^^^^^^^^^^

List of operations:

=========  ============================

Operation  Description

=========  ============================

HEAD       Retrieve container metadata

GET        List objects

PUT        Create/update container

POST       Update container metadata

DELETE     Delete container

=========  ============================

HEAD

""""

====================  ===========================

Request Header Name   Value

====================  ===========================

If-Modified-Since     Retrieve if container has changed since provided timestamp

If-Unmodified-Since   Retrieve if container has not changed since provided timestamp

====================  ===========================

|

======================  ===================================

Request Parameter Name  Value

======================  ===================================

until                   Optional timestamp

======================  ===================================

Cross-user requests are not allowed to use ``until`` and only include the container modification date in the reply.

===========================  ===============================

Reply Header Name            Value

===========================  ===============================

X-Container-Object-Count     The total number of objects in the container

X-Container-Bytes-Used       The total number of bytes of all objects stored

X-Container-Block-Size       The block size used by the storage backend

X-Container-Block-Hash       The hash algorithm used for block identifiers in object hashmaps

X-Container-Until-Timestamp  The last container modification date until the timestamp provided

X-Container-Object-Meta      A list with all meta keys used by objects (**TBD**)

X-Container-Policy-*         Container behavior and limits

X-Container-Meta-*           Optional user defined metadata

Last-Modified                The last container modification date (regardless of ``until``)

===========================  ===============================

The keys returned in ``X-Container-Object-Meta`` are all the unique strings after the ``X-Object-Meta-`` prefix, formatted as a comma-separated list. See container ``PUT`` for a reference of policy directives. (**TBD**)

================  ===============================

Return Code       Description

================  ===============================

204 (No Content)  The request succeeded

================  ===============================

GET

"""

====================  ===========================

Request Header Name   Value

====================  ===========================

If-Modified-Since     Retrieve if container has changed since provided timestamp

If-Unmodified-Since   Retrieve if container has not changed since provided timestamp

====================  ===========================

|

======================  ===================================

Request Parameter Name  Value

======================  ===================================

limit                   The amount of results requested (default is 10000)

marker                  Return containers with name lexicographically after marker

prefix                  Return objects starting with prefix

delimiter               Return objects up to the delimiter (discussion follows)

path                    Assume ``prefix=path`` and ``delimiter=/``

format                  Optional extended reply type (can be ``json`` or ``xml``)

meta                    Return objects that satisfy the key queries in the specified comma separated list (use ``<key>``, ``!<key>`` for existence queries, ``<key><op><value>`` for value queries, where ``<op>`` can be one of ``=``, ``!=``, ``<=``, ``>=``, ``<``, ``>``)

shared                  Show only shared objects (no value parameter)

until                   Optional timestamp

======================  ===================================

The ``path`` parameter overrides ``prefix`` and ``delimiter``. When using ``path``, results will include objects ending in ``delimiter``.

The keys given with ``meta`` will be matched with the strings after the ``X-Object-Meta-`` prefix.

The reply is a list of object names. Container headers (as in a ``HEAD`` request) will also be included.

Cross-user requests are not allowed to use ``until`` and include the following limited set of headers in the reply:

===========================  ===============================

Reply Header Name            Value

===========================  ===============================

X-Container-Block-Size       The block size used by the storage backend

X-Container-Block-Hash       The hash algorithm used for block identifiers in object hashmaps

X-Container-Object-Meta      A list with all meta keys used by allowed objects (**TBD**)

Last-Modified                The last container modification date

===========================  ===============================

If a ``format=xml`` or ``format=json`` argument is given, extended information on the objects will be returned, serialized in the chosen format.

For each object, the information will include all object metadata, except user-defined (names will be in lower case and with hyphens replaced with underscores). User-defined metadata includes ``X-Object-Meta-*``, ``X-Object-Manifest``, ``Content-Disposition`` and ``Content-Encoding`` keys. Also, sharing directives will only be included with the actual shared objects (inherited permissions are not calculated):

==========================  ======================================

Name                        Description

==========================  ======================================

name                        The name of the object

hash                        The ETag of the object

bytes                       The size of the object

content_type                The MIME content type of the object

last_modified               The last object modification date (regardless of version)

x_object_hash               The Merkle hash

x_object_uuid               The object's UUID

x_object_version            The object's version identifier

x_object_version_timestamp  The object's version timestamp

x_object_modified_by        The user that committed the object's version

x_object_sharing            Object permissions (optional)

x_object_allowed_to         Allowed actions on object (optional)

x_object_public             Object's publicly accessible URI (optional)

==========================  ======================================

Sharing metadata and last modification timestamp will only be returned if there is no ``until`` parameter defined.

Extended replies may also include virtual directory markers in separate sections of the ``json`` or ``xml`` results.

Virtual directory markers are only included when ``delimiter`` is explicitly set. They correspond to the substrings up to and including the first occurrence of the delimiter.

In JSON results they appear as dictionaries with only a ``subdir`` key. In XML results they appear interleaved with ``<object>`` tags as ``<subdir name="..." />``.

In case there is an object with the same name as a virtual directory marker, the object will be returned.

Example ``format=json`` reply:

::

  [{"name": "object",

    "bytes": 0,

    "hash": "d41d8cd98f00b204e9800998ecf8427e",

    "content_type": "application/octet-stream",

    "last_modified": "2011-12-02T08:10:41.565891+00:00",

    "x_object_hash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",

    "x_object_uuid": "8ed9af1b-c948-4bb6-82b0-48344f5c822c",

    "x_object_version": 98,

    "x_object_version_timestamp": "1322813441.565891",

    "x_object_modified_by": "user"}, ...]

Example ``format=xml`` reply:

::

  <?xml version="1.0" encoding="UTF-8"?>

  <container name="pithos">

    <object>

      <name>object</name>

      <bytes>0</bytes>

      <hash>d41d8cd98f00b204e9800998ecf8427e</hash>

      <content_type>application/octet-stream</content_type>

      <last_modified>2011-12-02T08:10:41.565891+00:00</last_modified>

      <x_object_hash>e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855</x_object_hash>

      <x_object_uuid>8ed9af1b-c948-4bb6-82b0-48344f5c822c</x_object_uuid>

      <x_object_version>98</x_object_version>

      <x_object_version_timestamp>1322813441.565891</x_object_version_timestamp>

      <x_object_modified_by>chazapis</x_object_modified_by>

    </object>

    <object>...</object>

  </container>

For more examples of container details returned in JSON/XML formats refer to the OOS API documentation. In addition to the OOS API, Pithos returns more fields that should help with synchronization.

===========================  ===============================

Return Code                  Description

===========================  ===============================

200 (OK)                     The request succeeded

204 (No Content)             The account has no containers (only for non-extended replies)

304 (Not Modified)           The container has not been modified

412 (Precondition Failed)    The condition set can not be satisfied

===========================  ===============================

Will use a ``200`` return code if the reply is of type JSON/XML.

PUT

"""

====================  ================================

Request Header Name   Value

====================  ================================

X-Container-Policy-*  Container behavior and limits

X-Container-Meta-*    Optional user defined metadata

====================  ================================

No reply content/headers.

If no policy is defined, the container will be created with the default values.

Available policy directives:

* ``versioning``: Set to ``auto`` or ``none`` (default is ``auto``)

* ``quota``: Size limit in KB (default is ``0`` - unlimited)

If the container already exists, the operation is equal to a ``POST`` with ``update`` defined.

================  ===============================

Return Code       Description

================  ===============================

201 (Created)     The container has been created

202 (Accepted)    The request has been accepted

================  ===============================

POST

""""

====================  ================================

Request Header Name   Value

====================  ================================

Content-Length        The size of the supplied data (optional, to upload)

Content-Type          The MIME content type of the supplied data (optional, to upload)

Transfer-Encoding     Set to ``chunked`` to specify incremental uploading (if used, ``Content-Length`` is ignored)

X-Container-Policy-*  Container behavior and limits

X-Container-Meta-*    Optional user defined metadata

====================  ================================

|

======================  ============================================

Request Parameter Name  Value

======================  ============================================

format                  Optional hash list reply type (can be ``json`` or ``xml``)

update                  Do not replace metadata/policy (no value parameter)

======================  ============================================

No reply content/headers, except when uploading data, where the reply consists of a list of hashes for the blocks received (in the format specified).

The operation will overwrite all user defined metadata, except if ``update`` is defined.

To change policy, include an ``X-Container-Policy-*`` header with the name in the key. If no ``X-Container-Policy-*`` header is present, no changes will be applied to policy. The ``update`` parameter also applies to policy - deleted values will revert to defaults. To delete/revert a specific policy directive, use ``update`` and an empty header value. See container ``PUT`` for a reference of policy directives.

To upload blocks of data to the container, set ``Content-Type`` to ``application/octet-stream`` and ``Content-Length`` to a valid value (except if using ``chunked`` as the ``Transfer-Encoding``).

================  ===============================

Return Code       Description

================  ===============================

202 (Accepted)    The request has been accepted

================  ===============================

DELETE

""""""

======================  ===================================

Request Parameter Name  Value

======================  ===================================

until                   Optional timestamp

======================  ===================================

If ``until`` is defined, the container is "purged" up to that time (the history of all objects up to then is deleted).

No reply content/headers.

================  ===============================

Return Code       Description

================  ===============================

204 (No Content)  The request succeeded

409 (Conflict)    The container is not empty

================  ===============================

Object Level

^^^^^^^^^^^^

List of operations:

=========  =================================

Operation  Description

=========  =================================

HEAD       Retrieve object metadata

GET        Read object data

PUT        Write object data or copy/move object

COPY       Copy object

MOVE       Move object

POST       Update object metadata/data

DELETE     Delete object

=========  =================================

HEAD

""""

====================  ================================

Request Header Name   Value

====================  ================================

If-Match              Retrieve if ETags match

If-None-Match         Retrieve if ETags don't match

If-Modified-Since     Retrieve if object has changed since provided timestamp

If-Unmodified-Since   Retrieve if object has not changed since provided timestamp

====================  ================================

|

======================  ===================================

Request Parameter Name  Value

======================  ===================================

version                 Optional version identifier

======================  ===================================

|

==========================  ===============================

Reply Header Name           Value

==========================  ===============================

ETag                        The ETag of the object

Content-Length              The size of the object

Content-Type                The MIME content type of the object

Last-Modified               The last object modification date (regardless of version)

Content-Encoding            The encoding of the object (optional)

Content-Disposition         The presentation style of the object (optional)

X-Object-Hash               The Merkle hash

X-Object-UUID               The object's UUID

X-Object-Version            The object's version identifier

X-Object-Version-Timestamp  The object's version timestamp

X-Object-Modified-By        The user that comitted the object's version

X-Object-Manifest           Object parts prefix in ``<container>/<object>`` form (optional)

X-Object-Sharing            Object permissions (optional)

X-Object-Shared-By          Object inheriting permissions (optional)

X-Object-Allowed-To         Allowed actions on object (optional)

X-Object-Public             Object's publicly accessible URI (optional)

X-Object-Meta-*             Optional user defined metadata

==========================  ===============================

|

================  ===============================

Return Code       Description

================  ===============================

200 (No Content)  The request succeeded

================  ===============================

GET

"""

====================  ================================

Request Header Name   Value

====================  ================================

Range                 Optional range of data to retrieve

If-Range              Retrieve the missing part if entity is unchanged; otherwise, retrieve the entire new entity (used together with Range header)

If-Match              Retrieve if ETags match

If-None-Match         Retrieve if ETags don't match

If-Modified-Since     Retrieve if object has changed since provided timestamp

If-Unmodified-Since   Retrieve if object has not changed since provided timestamp

====================  ================================

|

======================  ===================================

Request Parameter Name  Value

======================  ===================================

format                  Optional extended reply type (can be ``json`` or ``xml``)

hashmap                 Optional request for hashmap (no value parameter)

version                 Optional version identifier or ``list`` (specify a format if requesting a list)

======================  ===================================

The reply is the object's data (or part of it), except if a hashmap is requested with ``hashmap``, or a version list with ``version=list`` (in both cases an extended reply format must be specified). Object headers (as in a ``HEAD`` request) are always included.

Hashmaps expose the underlying storage format of the object. Note that each hash is computed after trimming trailing null bytes of the corresponding block. The ``X-Object-Hash`` header reports the single Merkle hash of the object's hashmap (refer to http://bittorrent.org/beps/bep_0030.html for more information).

Example ``format=json`` reply:

::

  {"block_hash": "sha1", "hashes": ["7295c41da03d7f916440b98e32c4a2a39351546c", ...], "block_size": 131072, "bytes": 242}

Example ``format=xml`` reply:

::

  <?xml version="1.0" encoding="UTF-8"?>

  <object name="file" bytes="24223726" block_size="131072" block_hash="sha1">

    <hash>7295c41da03d7f916440b98e32c4a2a39351546c</hash>

    <hash>...</hash>

  </object>

Version lists include the version identifier and timestamp for each available object version. Version identifiers can be arbitrary strings, so use the timestamp to find newer versions.

Example ``format=json`` reply:

::

  {"versions": [[85, "1322734861.248469"], [86, "1322734905.009272"], ...]}

Example ``format=xml`` reply:

::

  <?xml version="1.0" encoding="UTF-8"?>

  <object name="file">

    <version timestamp="1322734861.248469">85</version>

    <version timestamp="1322734905.009272">86</version>

    <version timestamp="...">...</version>

  </object>

The ``Range`` header may include multiple ranges, as outlined in RFC2616. Then the ``Content-Type`` of the reply will be ``multipart/byteranges`` and each part will include a ``Content-Range`` header.

==========================  ===============================

Reply Header Name           Value

==========================  ===============================

ETag                        The ETag of the object

Content-Length              The size of the data returned

Content-Type                The MIME content type of the object

Content-Range               The range of data included (only on a single range request)

Last-Modified               The last object modification date (regardless of version)

Content-Encoding            The encoding of the object (optional)

Content-Disposition         The presentation style of the object (optional)

X-Object-Hash               The Merkle hash

X-Object-UUID               The object's UUID

X-Object-Version            The object's version identifier

X-Object-Version-Timestamp  The object's version timestamp

X-Object-Modified-By        The user that comitted the object's version

X-Object-Manifest           Object parts prefix in ``<container>/<object>`` form (optional)

X-Object-Sharing            Object permissions (optional)

X-Object-Shared-By          Object inheriting permissions (optional)

X-Object-Allowed-To         Allowed actions on object (optional)

X-Object-Public             Object's publicly accessible URI (optional)

X-Object-Meta-*             Optional user defined metadata

==========================  ===============================

Sharing headers (``X-Object-Sharing``, ``X-Object-Shared-By`` and ``X-Object-Allowed-To``) are only included if the request is for the object's latest version (no specific ``version`` parameter is set).

===========================  ==============================

Return Code                  Description

===========================  ==============================

200 (OK)                     The request succeeded

206 (Partial Content)        The range request succeeded

304 (Not Modified)           The object has not been modified

412 (Precondition Failed)    The condition set can not be satisfied

416 (Range Not Satisfiable)  The requested range is out of limits

===========================  ==============================

PUT

"""

====================  ================================

Request Header Name   Value

====================  ================================

If-Match              Put if ETags match with current object

If-None-Match         Put if ETags don't match with current object

ETag                  The MD5 hash of the object (optional to check written data)

Content-Length        The size of the data written

Content-Type          The MIME content type of the object

Transfer-Encoding     Set to ``chunked`` to specify incremental uploading (if used, ``Content-Length`` is ignored)

X-Copy-From           The source path in the form ``/<container>/<object>``

X-Move-From           The source path in the form ``/<container>/<object>``

X-Source-Account      The source account to copy/move from

X-Source-Version      The source version to copy from

Content-Encoding      The encoding of the object (optional)

Content-Disposition   The presentation style of the object (optional)

X-Object-Manifest     Object parts prefix in ``<container>/<object>`` form (optional)

X-Object-Sharing      Object permissions (optional)

X-Object-Public       Object is publicly accessible (optional)

X-Object-Meta-*       Optional user defined metadata

====================  ================================

|

======================  ===================================

Request Parameter Name  Value

======================  ===================================

format                  Optional extended request/conflict response type (can be ``json`` or ``xml``)

hashmap                 Optional hashmap provided instead of data (no value parameter)

======================  ===================================

The request is the object's data (or part of it), except if a hashmap is provided (using ``hashmap`` and ``format`` parameters). If using a hashmap and all different parts are stored in the server, the object is created. Otherwise the server returns Conflict (409) with the list of the missing parts (in simple text format, with one hash per line, or in JSON/XML - depending on the ``format`` parameter).

Hashmaps should be formatted as outlined in ``GET``.

==========================  ===============================

Reply Header Name           Value

==========================  ===============================

ETag                        The MD5 hash of the object

X-Object-Version            The object's new version

==========================  ===============================

The ``X-Object-Sharing`` header may include either a ``read=...`` comma-separated user/group list, or a ``write=...`` comma-separated user/group list, or both separated by a semicolon (``;``). Groups are specified as ``<account>:<group>``. To publish the object, set ``X-Object-Public`` to ``true``. To unpublish, set to ``false``, or use an empty header value.

==============================  ==============================

Return Code                     Description

==============================  ==============================

201 (Created)                   The object has been created

409 (Conflict)                  The object can not be created from the provided hashmap (a list of missing hashes will be included in the reply)

411 (Length Required)           Missing ``Content-Length`` or ``Content-Type`` in the request

413 (Request Entity Too Large)  Insufficient quota to complete the request

422 (Unprocessable Entity)      The MD5 checksum of the data written to the storage system does not match the (optionally) supplied ETag value

==============================  ==============================

COPY

""""

====================  ================================

Request Header Name   Value

====================  ================================

If-Match              Proceed if ETags match with object

If-None-Match         Proceed if ETags don't match with object

Destination           The destination path in the form ``/<container>/<object>``

Destination-Account   The destination account to copy to

Content-Type          The MIME content type of the object (optional :sup:`*`)

Content-Encoding      The encoding of the object (optional)

Content-Disposition   The presentation style of the object (optional)

X-Source-Version      The source version to copy from

X-Object-Manifest     Object parts prefix in ``<container>/<object>`` form (optional)

X-Object-Sharing      Object permissions (optional)