Archipelago

sosd: Saving WIP

filed: Use the new XSEG protocol

This commit updates the filed request handling functions, in order to use
the newer XSEG protocol. Note that it does not alter the current logic of
filed.

Add helper functions to the new archipelago protocol

xseg: First attempt to allow flexible protocols

First attempt to decouple the communication mechanism data, with the data of the
protocol using the xseg mechanism.
In this commit, we introduce op data for each operation and a set of flexible
data buffers that accompany them. The actual op data, are defined by the...

archipelago: Fix wait request in common.py

filed: Respect req->size for copy requests

archipelago: Catch exceptions in argument parsing

archipelago: Enforce nr_ops as a power of 2 for all peers

peer: Respect the nr of ops enforced by xq size.

filed: Fix custom usage

Install archipelago.conf.rados_example

archipelago: Fix wrong identation and typos

Change location of settings file to /etc/archipelago

archipelago: Remove duplicate get_hash_request method

archipelago: Add wrapper function to get segment object

vlmc: Add X_HASH operation

vlmc: Fix get_vlmc_port for all operations

archipelago: Fix tabs in python modules

bench: Turn ping off by default

bench: Create separate option for progress type

Progress type option is now decoupled from the progress option. This
paves the way for silent progress reports.

archipelago: Fix concurrent vlmc ops

archipelago: Add support for concurrent vlmc ops

bench: Improve benchmark results

Changes include:

• Show the average request latency when run with insanity "eccentric"
• Don't show bandwidth results for info/delete requests

bench: Ping target peer before starting timers

In some cases, the target peer may take some time before it finishes
with its initializations. Since we don't want to measure this idle time,
we send a ping request first and can start start the benchmark after we...

bench: Add --prefix option

The --prefix option allows the user to choose whatever prefix he/she
wants for the object names that will be generated.

This commit also cleans code that has to do with seeding and creating
object names. It also paves the way for the --objname option.

bench: Add --objname option

The --objname option allows the user to define the name of the target.
This implies that for the duration of the benchmark, a sole object will
be used. This also comes along with a set of restrictions, that are
throughly explained in the help message

bench: Track elapsed time for each timer

Also, create two functions timespecsub() and timespecadd() that can
subtract and add two timespec structs.

bench: Fix data verification

Current situation: In full verification mode, each number written across
a data chunk has 64 bits length, yet, due to a wrong LFSR
initialization, only 31 bits are utilized.

Implement a 63/64 bits utilization.

bench: Improve benchmark results

Changes include:

• Show the average request latency when run with insanity "eccentric"
• Don't show bandwidth results for info/delete requests

bench: Fix an LFSR init issue and minor comments

If bench is requested to issue only one request, the LFSR initialization
fails. Since LFSR is not needed in this case, we simply switch to
sequential pattern for this request.

python Makefile: replace zero-length array with pointer

Replace zero-length array with pointer for the xseg_reply_map flexible array
member.

tests: Add SosdTest

mapper: Use correct size for epoch when loading v2 metadata

peers: Fix bug in hexlify.

Ditch strncpy which alwayes NULL terminates when hexlifing and use a custom
hexlify function.

python: Extend request interface with X_HASH

xseg: Add X_HASH to xseg protocol.

Add X_HASH operator to xseg, and the appopriate X_HASH reply structure. Remove
obsolete snapshot reply structure.

mapper: Add hash support for snapshots.

Add the ability to hash snapshots and create pithos compatible images.

Also, as a minor improvement add object_to_map representation to map functions.

tests: Add testing for cloning archipelago snapshots

mapper: Fix get_mapnode boundary.

Return NULL, when requesting object index == nr_obs

mapper: Always use datalen buffers when loading v2

testing: Add tests for concurrent operations

filed: Add debug logging while reading/writing

Improve python archipelago common infrastructure.

The changes include: * Convert pfiled to filed and add the extra arguments * Check if arguments are not None instead of simply checking for arguments * Move request to seperate method and remove del method...

mapper: Fail loading of deleted volumes

Set zero sized clone to inherit the size of the parent

Uniformly set that when the requested size is zero, then either inherit the size
of the parent map, or fail if there is no parent.

mapper: Add resize request on X_INF

mapper: add debug messages while handling objects in v2

mt-sosd: Use common hash.c for hashing

filed: Change req->datalen with req->size.

Do not use req->datalen to fullfill requests, but req->size, which was the
requested size.

mapperd: Add counter on map to count used objects

Add map counter to count used objects, so we can quickly calculate whether all
objects are in READY state, and there are no pending actions on them.