Feature #1887: Provide an interface to query blocks/hashmaps - Synnefo - Greek Research and Technology Network's projects

Feature #1887

Provide an interface to query blocks/hashmaps

Added by Antony Chazapis over 11 years ago. Updated over 10 years ago.

Status:

Closed

Start date:

01/19/2012

Priority:

Medium

Due date:

Assignee:

% Done:

Category:

Pithos

Spent time:

Target version:

Description

When uploading a file with its hashmap, you may need to read the file twice - one to create the hashmap and one to send the data. To avoid this, investigate an interface for querying blocks, so the client software can upload blocks and create the hashmap in one step.

History

#1 Updated by Antony Chazapis over 11 years ago

Target version deleted (~~1.0.0~~)

This has been opened mainly for discussion.

The interface provided now, assures the least communication with the server:

Request an object with the provided hashes
Upload any missing hashes
Request the object again

If this is converted to:

For each block:

Check if the block is on the server

Upload if not there

Request an object with the provided hashes

Then, for a very large object, the process may impose a great load on the server. Sure, a bad client software can flood the server with requests, but this can not be the "suggested" method.

Moreover:

We expect that most of the blocks will be online after some time.
We expect that client hardware has an excess of available computing power, while we should minimize the server load as much as possible. For objects up to a few hundred MB, it should be trivial for modern hardware to retrieve and process data.

So for small files, the current method seems best. For large files - like a 10GB vm system image, uploading with one request per 4MB block will require thousands of requests.

This is one opinion. Will leave it as is, except if new findings suggest a change is necessary.