Filippos Giannakos [Mon, 5 Aug 2013 13:42:49 +0000 (16:42 +0300)]
docs: Add initial docs.
Add initial sphinx docs of archipelago. Also add ci script for automatic doc
compiling.
Filippos Giannakos [Wed, 7 Aug 2013 14:30:21 +0000 (17:30 +0300)]
Merge branch 'feature-powerof2' into develop
Filippos Giannakos [Wed, 7 Aug 2013 14:29:23 +0000 (17:29 +0300)]
archipelago: Update conf documentation about nr_ops.
Upate config documentation that a peer nr_ops must be a power of 2.
Filippos Giannakos [Wed, 7 Aug 2013 14:29:04 +0000 (17:29 +0300)]
archipelago: Catch exceptions in argument parsing
Filippos Giannakos [Tue, 6 Aug 2013 10:06:59 +0000 (13:06 +0300)]
archipelago: Set nr_ops = nr_threads for filed
Filippos Giannakos [Tue, 6 Aug 2013 10:06:42 +0000 (13:06 +0300)]
archipelago: Enforce nr_ops as a power of 2 for all peers
Filippos Giannakos [Tue, 6 Aug 2013 10:05:31 +0000 (13:05 +0300)]
filed: Check fd limit against the real cache size
Filippos Giannakos [Tue, 6 Aug 2013 10:04:39 +0000 (13:04 +0300)]
peer: Respect the nr of ops enforced by xq size.
Filippos Giannakos [Tue, 6 Aug 2013 10:02:11 +0000 (13:02 +0300)]
xq: Respect size when allocating and initializing
When allocating and initializing a new xq, respect the xq size limit that the
initialization sets, and do not initialize past that.
Filippos Giannakos [Mon, 5 Aug 2013 10:44:51 +0000 (13:44 +0300)]
filed: Fix custom usage
Filippos Giannakos [Mon, 5 Aug 2013 10:44:28 +0000 (13:44 +0300)]
archipelago conf: Add comments on settings
Filippos Giannakos [Fri, 2 Aug 2013 09:44:05 +0000 (12:44 +0300)]
Install archipelago.conf.rados_example
Filippos Giannakos [Thu, 1 Aug 2013 12:03:59 +0000 (15:03 +0300)]
ci: Insert licence headers
Filippos Giannakos [Thu, 1 Aug 2013 10:51:16 +0000 (13:51 +0300)]
Merge branch 'feature-testingrados' into develop
Filippos Giannakos [Thu, 1 Aug 2013 09:27:26 +0000 (12:27 +0300)]
ci: Add connection to a RADOS cluster
Filippos Giannakos [Wed, 31 Jul 2013 07:12:27 +0000 (10:12 +0300)]
Merge branch 'feature-ci' into develop
Filippos Giannakos [Mon, 29 Jul 2013 11:47:18 +0000 (14:47 +0300)]
archipelago: Fix wrong identation and typos
Filippos Giannakos [Fri, 26 Jul 2013 11:47:55 +0000 (14:47 +0300)]
ci: Do not ping servers on okeanos io
Filippos Giannakos [Fri, 26 Jul 2013 09:47:49 +0000 (12:47 +0300)]
ci: Transfer VM creation to okeanos.io
Filippos Giannakos [Fri, 26 Jul 2013 07:15:37 +0000 (10:15 +0300)]
ci: Add basic functionality test on commit.
Also split unittests to execute each component sepereatly
Filippos Giannakos [Thu, 25 Jul 2013 16:37:02 +0000 (19:37 +0300)]
ci: Reduce logging level to info to cluster.py
Filippos Giannakos [Thu, 25 Jul 2013 15:14:00 +0000 (18:14 +0300)]
ci: Add simple tests.sh
Filippos Giannakos [Thu, 25 Jul 2013 15:11:28 +0000 (18:11 +0300)]
ci: Add cluster management files
Filippos Giannakos [Fri, 26 Jul 2013 07:35:51 +0000 (10:35 +0300)]
Change archipelago default settings
Change archipelago default settings to file blocker. Also provide a working
example for rados blocker
Filippos Giannakos [Fri, 26 Jul 2013 07:26:55 +0000 (10:26 +0300)]
Change location of settings file to /etc/archipelago
Filippos Giannakos [Thu, 25 Jul 2013 11:38:18 +0000 (14:38 +0300)]
Adjust tests to the new Segment class
Filippos Giannakos [Thu, 25 Jul 2013 11:37:42 +0000 (14:37 +0300)]
archipelago: Remove duplicate get_hash_request method
Filippos Giannakos [Tue, 23 Jul 2013 09:26:58 +0000 (12:26 +0300)]
archipelago: Add wrapper function to get segment object
Filippos Giannakos [Mon, 22 Jul 2013 10:36:05 +0000 (13:36 +0300)]
ext_storage: Check for 'pithos:' on create.
Filippos Giannakos [Mon, 22 Jul 2013 10:15:47 +0000 (13:15 +0300)]
vlmc: Add X_HASH operation
Filippos Giannakos [Mon, 22 Jul 2013 10:14:25 +0000 (13:14 +0300)]
archipelago: Declare segment as Class and modify settings
Declare xseg segment as class, and make use of its methods. Also modify config
file check to expect segment parameters instead of one segment spec.
Filippos Giannakos [Fri, 19 Jul 2013 15:24:46 +0000 (18:24 +0300)]
vlmc: Fix get_vlmc_port for all operations
Filippos Giannakos [Fri, 19 Jul 2013 14:48:11 +0000 (17:48 +0300)]
archipelago: Add spec config when creating segment
Filippos Giannakos [Fri, 19 Jul 2013 14:42:33 +0000 (17:42 +0300)]
archipelago: Fix tabs in python modules
Filippos Giannakos [Fri, 19 Jul 2013 14:14:33 +0000 (17:14 +0300)]
Merge branch 'feature-bench-improvements-rebased2' into develop
Alex Pyrgiotis [Fri, 19 Jul 2013 08:57:42 +0000 (11:57 +0300)]
bench: Create separate option for progress type
Progress type option is now decoupled from the progress option. This
paves the way for silent progress reports.
Alex Pyrgiotis [Fri, 19 Jul 2013 07:49:10 +0000 (10:49 +0300)]
bench: Break usage message in sections
Alex Pyrgiotis [Fri, 19 Jul 2013 07:28:03 +0000 (10:28 +0300)]
bench: Turn ping off by default
Alex Pyrgiotis [Fri, 19 Jul 2013 07:12:47 +0000 (10:12 +0300)]
bench: Break large source file into smaller files
The bench-utils.c file started to have utilities that could stand
semantically under a separate category. Namely, verification is now at
bench-verify.c and progress report under bench-report.c.
Filippos Giannakos [Fri, 19 Jul 2013 14:03:38 +0000 (17:03 +0300)]
Merge branch 'feature-vlmcmap' into develop
Filippos Giannakos [Thu, 18 Jul 2013 16:16:22 +0000 (19:16 +0300)]
vlmc: Do not map an already mounted volume
Filippos Giannakos [Fri, 19 Jul 2013 14:02:12 +0000 (17:02 +0300)]
Merge branch 'feature-vlmcmultipleports' into develop
Filippos Giannakos [Fri, 19 Jul 2013 09:30:22 +0000 (12:30 +0300)]
archipelago: Fix concurrent vlmc ops
Filippos Giannakos [Fri, 19 Jul 2013 06:28:46 +0000 (09:28 +0300)]
archipelago: Add support for nested locking
Filippos Giannakos [Fri, 19 Jul 2013 06:12:53 +0000 (09:12 +0300)]
archipelago: Add support for concurrent vlmc ops
Filippos Giannakos [Thu, 18 Jul 2013 12:20:02 +0000 (15:20 +0300)]
Merge branch 'feature-bench-improvements-rebased2' into develop
Alex Pyrgiotis [Wed, 17 Jul 2013 19:19:54 +0000 (22:19 +0300)]
bench: Introduce new progress report and intervals
This commit renews the progress report that bench has by allowing to
report bandwidth results during benchmark.
Also, an additional functionality is that of report intervals. Report
intervals allow the user to choose exactly when progress report will be
printed.
Both of these features are documented extensively in the usage section.
Alex Pyrgiotis [Wed, 17 Jul 2013 19:12:23 +0000 (22:12 +0300)]
bench: Track elapsed time for each timer
Also, create two functions timespecsub() and timespecadd() that can
subtract and add two timespec structs.
Alex Pyrgiotis [Tue, 16 Jul 2013 10:39:13 +0000 (13:39 +0300)]
bench: Handle progress report better
Pave the way to include bandwidth/IOPS results during progress report
Alex Pyrgiotis [Mon, 8 Jul 2013 11:45:10 +0000 (14:45 +0300)]
bench: Add --objname option
The --objname option allows the user to define the name of the target.
This implies that for the duration of the benchmark, a sole object will
be used. This also comes along with a set of restrictions, that are
throughly explained in the help message
Alex Pyrgiotis [Mon, 8 Jul 2013 10:59:33 +0000 (13:59 +0300)]
bench: Fix segfault while freeing resources
Alex Pyrgiotis [Mon, 8 Jul 2013 10:46:03 +0000 (13:46 +0300)]
bench: Add --prefix option
The --prefix option allows the user to choose whatever prefix he/she
wants for the object names that will be generated.
This commit also cleans code that has to do with seeding and creating
object names. It also paves the way for the --objname option.
Alex Pyrgiotis [Mon, 10 Jun 2013 15:44:25 +0000 (18:44 +0300)]
bench: Fix minor validation issue
Alex Pyrgiotis [Sat, 22 Jun 2013 20:00:28 +0000 (23:00 +0300)]
bench: Ping target peer before starting timers
In some cases, the target peer may take some time before it finishes
with its initializations. Since we don't want to measure this idle time,
we send a ping request first and can start start the benchmark after we
receive an ACK.
NOTE: This behaviour can be switched off with the "--ping no" option.
Alex Pyrgiotis [Mon, 3 Jun 2013 11:05:07 +0000 (14:05 +0300)]
bench: Remove dead code
Alex Pyrgiotis [Fri, 31 May 2013 10:35:35 +0000 (13:35 +0300)]
bench: Improve benchmark results
Changes include:
* Show the average request latency when run with insanity "eccentric"
* Don't show bandwidth results for info/delete requests
Alex Pyrgiotis [Mon, 3 Jun 2013 11:05:07 +0000 (14:05 +0300)]
bench: Remove dead code
Alex Pyrgiotis [Mon, 3 Jun 2013 10:35:48 +0000 (13:35 +0300)]
bench: Fix an LFSR init issue and minor comments
If bench is requested to issue only one request, the LFSR initialization
fails. Since LFSR is not needed in this case, we simply switch to
sequential pattern for this request.
Alex Pyrgiotis [Fri, 31 May 2013 15:25:17 +0000 (18:25 +0300)]
bench: Remove unnecessary malloc
Alex Pyrgiotis [Fri, 31 May 2013 10:35:35 +0000 (13:35 +0300)]
bench: Improve benchmark results
Changes include:
* Show the average request latency when run with insanity "eccentric"
* Don't show bandwidth results for info/delete requests
Alex Pyrgiotis [Mon, 20 May 2013 13:02:24 +0000 (16:02 +0300)]
bench: Minor correction on seed validation
Alex Pyrgiotis [Thu, 16 May 2013 08:14:17 +0000 (11:14 +0300)]
bench: Fix data verification
Current situation: In full verification mode, each number written across
a data chunk has 64 bits length, yet, due to a wrong LFSR
initialization, only 31 bits are utilized.
Implement a 63/64 bits utilization.
Filippos Giannakos [Thu, 18 Jul 2013 09:58:59 +0000 (12:58 +0300)]
Merge branch 'feature-snapshot' into develop
Filippos Giannakos [Thu, 18 Jul 2013 09:29:53 +0000 (12:29 +0300)]
vlmc: add cont_addr option in snapshot command
Filippos Giannakos [Thu, 18 Jul 2013 09:26:13 +0000 (12:26 +0300)]
python Makefile: replace zero-length array with pointer
Replace zero-length array with pointer for the xseg_reply_map flexible array
member.
Filippos Giannakos [Thu, 18 Jul 2013 07:15:25 +0000 (10:15 +0300)]
Peers Makefile: Reenable sosd
Filippos Giannakos [Thu, 18 Jul 2013 07:15:04 +0000 (10:15 +0300)]
tests: Add SosdTest
Filippos Giannakos [Thu, 18 Jul 2013 07:14:01 +0000 (10:14 +0300)]
sosd: Fix compilation error and hashing bug.
* Fix compilation error
* Correctly return the hased value in hash reply
* Resize X_INFO request when necessary
Filippos Giannakos [Thu, 18 Jul 2013 07:13:23 +0000 (10:13 +0300)]
mapper: Use correct size for epoch when loading v2 metadata
Filippos Giannakos [Thu, 18 Jul 2013 07:12:03 +0000 (10:12 +0300)]
mapper: Fix calculation of copy up target.
When hexlifing epoch and index, use the correct size of the source.
Filippos Giannakos [Thu, 18 Jul 2013 07:10:42 +0000 (10:10 +0300)]
peers: Fix bug in hexlify.
Ditch strncpy which alwayes NULL terminates when hexlifing and use a custom
hexlify function.
Filippos Giannakos [Wed, 17 Jul 2013 12:40:13 +0000 (15:40 +0300)]
tests: Add extra tests for hashing.
Filippos Giannakos [Wed, 17 Jul 2013 12:39:49 +0000 (15:39 +0300)]
python: Extend request interface with X_HASH
Filippos Giannakos [Wed, 17 Jul 2013 12:39:04 +0000 (15:39 +0300)]
sosd: Add X_HASH support with precomputed values
Filippos Giannakos [Wed, 17 Jul 2013 12:37:43 +0000 (15:37 +0300)]
xseg: Add X_HASH to xseg protocol.
Add X_HASH operator to xseg, and the appopriate X_HASH reply structure. Remove
obsolete snapshot reply structure.
Filippos Giannakos [Wed, 17 Jul 2013 12:37:09 +0000 (15:37 +0300)]
vlmc: Remove obsolete snapshot handling functionality
Filippos Giannakos [Wed, 17 Jul 2013 12:35:47 +0000 (15:35 +0300)]
mapper: Add hash support for snapshots.
Add the ability to hash snapshots and create pithos compatible images.
Also, as a minor improvement add object_to_map representation to map functions.
Filippos Giannakos [Wed, 17 Jul 2013 12:31:07 +0000 (15:31 +0300)]
filed: Support X_HASH with precomputed hashes.
Support X_HASH with precomputed hash values, if the same block was hashed
before.
Filippos Giannakos [Tue, 16 Jul 2013 08:14:39 +0000 (11:14 +0300)]
tests: Add testing for cloning archipelago snapshots
Filippos Giannakos [Tue, 16 Jul 2013 08:13:49 +0000 (11:13 +0300)]
mapper: Add XF_CONTADDR flag and improve cloning
Support cloning from both pithos images and archipelago snapshots.
Filippos Giannakos [Tue, 16 Jul 2013 08:12:49 +0000 (11:12 +0300)]
mapper: Fix get_mapnode boundary.
Return NULL, when requesting object index == nr_obs
Filippos Giannakos [Tue, 16 Jul 2013 08:11:35 +0000 (11:11 +0300)]
mapper: Fail when issuing a map object load fails
Filippos Giannakos [Tue, 16 Jul 2013 08:10:44 +0000 (11:10 +0300)]
mapper: Always use datalen buffers when loading v2
Filippos Giannakos [Tue, 16 Jul 2013 08:09:57 +0000 (11:09 +0300)]
mapper: Set correct map object when copying up
Filippos Giannakos [Mon, 15 Jul 2013 13:39:13 +0000 (16:39 +0300)]
Merge branch 'feature-functionaltest' into feature-snapshot
Filippos Giannakos [Mon, 15 Jul 2013 13:28:43 +0000 (16:28 +0300)]
testing: Add tests for concurrent operations
Filippos Giannakos [Mon, 15 Jul 2013 13:26:22 +0000 (16:26 +0300)]
python xseg: Enrich request/xseg python api.
Add more constructors for the Request class.
Add xseg_wait_requests to wait on multiple requests.
Modify vlmc script to use the new constructors.
Filippos Giannakos [Mon, 15 Jul 2013 13:25:29 +0000 (16:25 +0300)]
filed: Add debug logging while reading/writing
Filippos Giannakos [Fri, 12 Jul 2013 07:47:24 +0000 (10:47 +0300)]
tests: Add basic functionality testing for peers
Filippos Giannakos [Thu, 11 Jul 2013 15:12:20 +0000 (18:12 +0300)]
Improve python archipelago common infrastructure.
The changes include:
* Convert pfiled to filed and add the extra arguments
* Check if arguments are not None instead of simply checking for arguments
* Move request to seperate method and remove __del__ method
* Add get_serviced and get_datalen to request
* Add support to set string add data request
* Make all peer executables relative to a BIN_DIR
* Create/destroy segment takes spec as arguments
* Parsing spec on Xsegctx __init__ does not modify spec parameter
Filippos Giannakos [Thu, 11 Jul 2013 15:11:23 +0000 (18:11 +0300)]
peer: Create pidfile with read/write permissions
Filippos Giannakos [Thu, 11 Jul 2013 14:38:18 +0000 (17:38 +0300)]
mapper: Fail loading of deleted volumes
Filippos Giannakos [Thu, 11 Jul 2013 14:37:43 +0000 (17:37 +0300)]
mapper: Fail request when range is not valid
Filippos Giannakos [Thu, 11 Jul 2013 14:35:15 +0000 (17:35 +0300)]
Set zero sized clone to inherit the size of the parent
Uniformly set that when the requested size is zero, then either inherit the size
of the parent map, or fail if there is no parent.
Filippos Giannakos [Thu, 11 Jul 2013 14:33:08 +0000 (17:33 +0300)]
vlmcd: resize request on X_INFO
Filippos Giannakos [Thu, 11 Jul 2013 14:32:46 +0000 (17:32 +0300)]
mapper: Add resize request on X_INF
Filippos Giannakos [Thu, 11 Jul 2013 14:30:49 +0000 (17:30 +0300)]
filed: Add resize request on X_INFO
Filippos Giannakos [Thu, 11 Jul 2013 14:30:04 +0000 (17:30 +0300)]
mapper: add debug messages while handling objects in v2
Filippos Giannakos [Thu, 11 Jul 2013 14:23:34 +0000 (17:23 +0300)]
xseg: remove some unused logging
Filippos Giannakos [Mon, 8 Jul 2013 08:59:51 +0000 (11:59 +0300)]
mapperd: Modify version2 map IO functions.
Modify version2 map IO functions to support load/write map data for a specified
object range.