Synnefo

Storage API with several important extensions. It uses a

block-based mechanism to allow users to upload, download, and share

files, keep different versions of a file, and attach policies to them.

be used as a storage service by the total set of the Greek research

and academic community (counting tens of thousands of users) but is

free and open to use by anybody, under a BSD-2 clause license.

Introduction

------------

12,000 users.

In 2011 GRNET decided to offer a new, evolved online storage

main requirements expressed by the Pithos users in the first two years of

operation:

  open up Pithos to non-Shibboleth authenticated users as well.

* Use open standards as far as possible.   

of its design and the capabilities it affords. We touch on related

work and we provide some discussion on our experiences and thoughts on

the future.

used a home-grown API). We decided to adopt an open standard

API in order to leverage existing clients that implement the

use features going beyond those offered by the OpenStack API.

able to access and test the service with existing clients, while also

developing new clients based on open source OpenStack clients.

  OpenStack stores objects, which may be files, but this is not

  necessary - large files (longer than 5GBytes), for instance, must be

  stored as a series of distinct objects accompanied by a manifest.

* Permissions on individual files and folders. Note that folders

  do not exist in the OpenStack API, but are simulated by

  avoid incompatibility.

* Fully-versioned objects.

* Metadata-based queries. Users are free to set metadata on their

* Partial upload and download based on HTTP request

  headers and parameters.

* Object updates, where data may even come from other objects

  other objects without uploading data.

* All objects are assigned UUIDs on creation, which can be

  used to reference them regardless of their path location.

operations implement an extended OpenStack Object Storage API.

The back end is a library meant to be used by internal code and

other front ends. For instance, the back end library, apart from being

is also used in our implementation of the OpenStack Image

Service API. Moreover, the back end library allows specification

of different namespaces for metadata, so that the same object can be

viewed by different front end APIs with different sets of

with one set of metadata, or as an image with a different set of

metadata, in our implementation of the OpenStack Image Service.

Block-based Storage for the Client

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

operations are no longer required to refer to the whole object. We can

handle parts of objects as needed when uploading, downloading, or

copying and moving data.

In particular, a client, provided it has access permissions, can

object. If the request includes the ``hashmap`` parameter, then the

request refers to a hashmap, that is, a set containing the

object's block hashes. The reply is of the form::

server and containing the root Merkle hash of the object's

hashmap.

submitted to the server, with the following algorithm:

* Calculate the hash value for each block of the object to be

In effect, we are deduplicating data based on their block hashes,

transparently to the users. This results to perceived instantaneous

Block-based Storage Processing

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Hashmaps themselves are saved in blocks. All blocks are persisted to

storage using content-based addressing. It follows that to read a

* The client issues a request to get a file, via HTTP ``GET``.

* The API front end asks from the back end the metadata

data will come from another object, instead of being uploaded by the

client. 

path entry (see Figure). Nodes can be accounts,

containers, and objects. A user may have multiple

accounts, each account may have multiple containers, and each

simulate pseudo-hierarchical folders. In particular, object names that

contain the forward slash character and have an accompanying marker

object with a ``Content-Type: application/directory`` as part of

node corresponds to a unique path, and we keep its parent in the

account/container/object hierarchy (that is, all objects have a

container as their parent).

For each object version we keep the root Merkle hash of the object it

refers to, the size of the object, the last modification time and the

.. image:: images/pithos-backend-versions.png

contents listing of their accounts. In effect, this also allows them

to take their containers back in time. This is implemented

conceptually by taking a vertical line in the Figure and

presenting to the user the state on the left side of the line.

individual users or groups of users. Groups as collections of users

created at the account level by users themselves, and are flat - a

group cannot contain or reference another group. Ownership of a file

cannot be delegated.

object is readable by all. When an object is made public, it is

assigned a URL that can be used to access the object from

Permissions can be assigned to objects, which may be actual files, or

directories. When listing objects, the back end uses the permissions as

Discussion

----------

as a database abstraction layer. It is currently about

17,000 lines of code, and it has taken about 50 person months of

development effort. This development was done from scratch, with no

The desktop clients allow synchronization with local directories, a

feature that existing users of Pithos have been asking for, probably

influenced by services like DropBox. These clients are offered in

build on top of the API front end - we treat our own web

application as just another client that has to go through the API

front end, without granting it access to the back end directly.

a step further, with new projects we have in our pipeline, in which a

use again the API front end, so that repository users will have

functionality such as full text search, Dublin Core metadata storage

and querying, streaming, and so on.

available to users by invitation. We will extend our user base as we

move to beta in the coming months, and to our full set of users in the

second half of 2012. We are eager to see how our ideas fare as we will

Acknowledgments

---------------

Services for the Research and Academic Community", of the Greek

National Strategic Reference Framework.

Availability

------------

https://code.grnet.gr/projects/pithos/repository

The code can also be accessed from its source repository: