archipelago
11 years agoDisable update stamp during request transfer.
Filippos Giannakos [Thu, 14 Mar 2013 09:12:26 +0000 (11:12 +0200)]
Disable update stamp during request transfer.

11 years agoFix clean option in Makefile
Alex Pyrgiotis [Thu, 14 Mar 2013 15:51:49 +0000 (17:51 +0200)]
Fix clean option in Makefile

11 years agoMerge branch 'feature-bench-rebased' into develop
Filippos Giannakos [Thu, 14 Mar 2013 10:11:36 +0000 (12:11 +0200)]
Merge branch 'feature-bench-rebased' into develop

11 years agoBuild only the necessary peers.
Filippos Giannakos [Thu, 14 Mar 2013 10:10:32 +0000 (12:10 +0200)]
Build only the necessary peers.

11 years agoFix LFSR spin bug
Alex Pyrgiotis [Wed, 13 Mar 2013 18:14:57 +0000 (20:14 +0200)]
Fix LFSR spin bug

11 years agoStyle changes
Alex Pyrgiotis [Wed, 13 Mar 2013 16:04:35 +0000 (18:04 +0200)]
Style changes

11 years agoAdd option for read/write ops to hit unique objs
Alex Pyrgiotis [Wed, 13 Mar 2013 16:00:29 +0000 (18:00 +0200)]
Add option for read/write ops to hit unique objs

Read/write ops can now use the -to option, instead of the -ts option, to
make each read/write request touch a different object.

11 years agoAdd request cap
Alex Pyrgiotis [Wed, 13 Mar 2013 13:12:10 +0000 (15:12 +0200)]
Add request cap

Implemented by popular request, the request cap option allows the
benchmarking peer to stop the benchmark earlier, at a given number
of received requests.

11 years agoFix verification full mode
Alex Pyrgiotis [Wed, 13 Mar 2013 11:52:42 +0000 (13:52 +0200)]
Fix verification full mode

11 years agoAdd partial support for verification
Alex Pyrgiotis [Tue, 12 Mar 2013 16:29:54 +0000 (18:29 +0200)]
Add partial support for verification

Adds an option to choose between verification modes:

1. meta: On this mode, we write our signature both at the start and the
   end of the chunk size. It should be very fast and moderately safe,
   due to the fact that corruptions happen it the chunk's edges.
2. full: Write our a random sequence of numbers throughout the chunk's
   data range, which is reproducible given the object name and the
   offset inside the object.

At the moment, only meta mode is available, but full will be supported
soon

11 years agoRefactor struct bench and appropriate code paths
Alex Pyrgiotis [Tue, 12 Mar 2013 16:16:50 +0000 (18:16 +0200)]
Refactor struct bench and appropriate code paths

Changes:

1. Move flag values (such as pattern, insanity, verify) in a specific
   bit-field.
2. Create a struct req_status where the status of requests will be
   stored.
3. Tidy up code.

Rationale:

1. For parameters that can have only a few different values, it's an
   overkill to use a specific field in struct bench to store them, when
   we can just as well store them in a flag. This reduces the size of
   struct bench and makes the code easier to grasp.
2. Previously, the requests' statuses would be scattered amongst
   different timers, which is counter-intuitive at best. By placing them
   in a specific struct, we make the code more intuitive and readable.

11 years agoAdd skeleton for verification in write path
Alex Pyrgiotis [Sun, 10 Mar 2013 23:37:52 +0000 (01:37 +0200)]
Add skeleton for verification in write path

11 years agoImprove LFSR implementation
Alex Pyrgiotis [Sun, 10 Mar 2013 11:49:25 +0000 (13:49 +0200)]
Improve LFSR implementation

Also, prepare the code for the addition of verification support

11 years agoChange input option to bench
Alex Pyrgiotis [Sat, 2 Mar 2013 11:22:14 +0000 (13:22 +0200)]
Change input option to bench

Since -dp is reserved in peer as defer port number, we change the -dp
option in bench (which stood for "destination port") to -tp, which
stands for "target port".

11 years agoAdd support for seed values given by the user
Alex Pyrgiotis [Wed, 27 Feb 2013 18:52:12 +0000 (20:52 +0200)]
Add support for seed values given by the user

A new option has been added (--seed) which allows the user to pre-define
the seed that he/she wants. This has the following effects:

* The target names are prefixed with the first 9 digits of the seed.
* LFSR produces a predictable sequence of "random" numbers

11 years agoMinor fixes
Alex Pyrgiotis [Mon, 25 Feb 2013 16:52:35 +0000 (18:52 +0200)]
Minor fixes

11 years agoAllow LFSR to be used for sizes smaller than 2^3
Alex Pyrgiotis [Mon, 25 Feb 2013 16:42:29 +0000 (18:42 +0200)]
Allow LFSR to be used for sizes smaller than 2^3

11 years agoGet correct iodepth
Alex Pyrgiotis [Sat, 23 Feb 2013 12:42:15 +0000 (14:42 +0200)]
Get correct iodepth

11 years agoFix incorrect reading of "insanity" parameter
Alex Pyrgiotis [Thu, 21 Feb 2013 15:51:32 +0000 (17:51 +0200)]
Fix incorrect reading of "insanity" parameter

11 years agoFix identation
Alex Pyrgiotis [Thu, 21 Feb 2013 12:04:19 +0000 (14:04 +0200)]
Fix identation

11 years agoMake bench snappier when asked to terminate
Alex Pyrgiotis [Thu, 21 Feb 2013 10:44:46 +0000 (12:44 +0200)]
Make bench snappier when asked to terminate

11 years agoPrint more useful results
Alex Pyrgiotis [Thu, 21 Feb 2013 10:34:35 +0000 (12:34 +0200)]
Print more useful results

11 years agoFix signal handling of peers
Alex Pyrgiotis [Wed, 20 Feb 2013 22:16:55 +0000 (00:16 +0200)]
Fix signal handling of peers

11 years agoTweak performance of bench
Alex Pyrgiotis [Wed, 20 Feb 2013 15:28:23 +0000 (17:28 +0200)]
Tweak performance of bench

11 years agoFix LFSR behavior
Alex Pyrgiotis [Wed, 20 Feb 2013 07:58:43 +0000 (09:58 +0200)]
Fix LFSR behavior

11 years agoFix flag handling of struct bench
Alex Pyrgiotis [Tue, 19 Feb 2013 17:15:05 +0000 (19:15 +0200)]
Fix flag handling of struct bench

11 years agoFix target names and chunk offsets
Alex Pyrgiotis [Tue, 19 Feb 2013 16:59:37 +0000 (18:59 +0200)]
Fix target names and chunk offsets

11 years agoIncrement completed requests of insane timers
Alex Pyrgiotis [Tue, 19 Feb 2013 15:24:06 +0000 (17:24 +0200)]
Increment completed requests of insane timers

Even if a timer is not used for a benchmark, we need to increment its
completed requests since these are needed to calculate how many requests
are left etc.

11 years agoImplement most semi-finished functionalities
Alex Pyrgiotis [Tue, 19 Feb 2013 14:28:43 +0000 (16:28 +0200)]
Implement most semi-finished functionalities

Summary of new functionalities:

* Add support for 4 different xseg operations (read, write, info,
  delete).
* Add support for random I/O patterns with a fast Galois LFSR.
* Add insanity levels in timers, which allow the user to exclude obscure
  timers from the benchmark (or explicitly allow them)
* Generate target names based on seeds given by the user or produced by
  the benchmark (useful for reproducibility of benchmark)

11 years agoAdd an LFSR PRNG for bench
Alex Pyrgiotis [Mon, 18 Feb 2013 19:56:20 +0000 (21:56 +0200)]
Add an LFSR PRNG for bench

The PRNG introduced by this commit is a Galois-XNOR LFSR, which can
generate unique pseudorandom numbers. Also, seeded with the same
seed, it will produce the same sequence.

This implementation is driven by the needs of XSEG for a blazing fast
random generator. Initial tests show that it's capable of producing 2^38
unique pseudorandom numbers in less than 10 minutes (which equals to 2
nanoseconds per number).

11 years agoAdd utility functions to header file
Alex Pyrgiotis [Sat, 16 Feb 2013 19:24:24 +0000 (21:24 +0200)]
Add utility functions to header file

11 years agoDesign the program's skeleton for new features
Alex Pyrgiotis [Sat, 16 Feb 2013 18:51:10 +0000 (20:51 +0200)]
Design the program's skeleton for new features

Bench peer must be able to test all xseg operations in a synchronous or
asynchronous way. Also, it must have a validation mode to sanity-check
if xseg requests are issued properly. Most of the above are present in
this commit, either in code or TODO form.

WARNING: code is incomplete and won't compile

11 years agoFix segfault, parameters validation
Alex Pyrgiotis [Fri, 15 Feb 2013 09:40:11 +0000 (11:40 +0200)]
Fix segfault, parameters validation

11 years agoInitial support for xseg ops
Alex Pyrgiotis [Thu, 14 Feb 2013 18:00:03 +0000 (20:00 +0200)]
Initial support for xseg ops

11 years agoAlpha release of bench peer
Alex Pyrgiotis [Thu, 14 Feb 2013 13:38:51 +0000 (15:38 +0200)]
Alpha release of bench peer

On this release, the bench peer can measure the total elapsed time
of the benchmark. TODO:

* Allow bench peer to be multi-threaded.
* Adjust benchmarking precision, by eliminating or including more
  trivial timers.
* Measure mean time, standard deviation of requests.

11 years agoMake bench send requests faster
Alex Pyrgiotis [Tue, 12 Feb 2013 17:39:26 +0000 (19:39 +0200)]
Make bench send requests faster

11 years agoFix missing function declaration
Alex Pyrgiotis [Tue, 12 Feb 2013 12:02:58 +0000 (14:02 +0200)]
Fix missing function declaration

11 years agoSimplify port binding code
Alex Pyrgiotis [Tue, 12 Feb 2013 11:28:37 +0000 (13:28 +0200)]
Simplify port binding code

11 years agoFix segfault
Alex Pyrgiotis [Tue, 12 Feb 2013 07:02:09 +0000 (09:02 +0200)]
Fix segfault

11 years agoKeep naming in peerd_loop consistent
Alex Pyrgiotis [Tue, 12 Feb 2013 06:57:14 +0000 (08:57 +0200)]
Keep naming in peerd_loop consistent

11 years agoFix timers and add them to critical paths
Alex Pyrgiotis [Mon, 11 Feb 2013 16:38:06 +0000 (18:38 +0200)]
Fix timers and add them to critical paths

Fixed incorrect handling of elapsed time and used appropriate timers to
benchmark crtical paths.

11 years agoComplete unification of thread_loop/peerd_loop
Alex Pyrgiotis [Mon, 11 Feb 2013 16:34:50 +0000 (18:34 +0200)]
Complete unification of thread_loop/peerd_loop

11 years agoFix segfault
Alex Pyrgiotis [Mon, 11 Feb 2013 07:00:08 +0000 (09:00 +0200)]
Fix segfault

11 years agoUnify thread loop with peerd_loop
Alex Pyrgiotis [Mon, 11 Feb 2013 01:10:12 +0000 (03:10 +0200)]
Unify thread loop with peerd_loop

Since peerd_loop with thread_loop don't have many differences between
them, they can be unified in a single loop. TODO is to choose a unified
name for both threads and peers, as well as fault check the
implementation.

11 years agoAdd timer implementation
Alex Pyrgiotis [Sat, 9 Feb 2013 11:32:49 +0000 (13:32 +0200)]
Add timer implementation

This an initial timer implementation. It is not very polished at the
momment and some implementation choices will have to be reviewed later
on.

11 years agoAdd support for custom peer loop
Alex Pyrgiotis [Fri, 8 Feb 2013 09:28:04 +0000 (11:28 +0200)]
Add support for custom peer loop

There is a new field in struct peerd called "custom_peerd_loop". If any
peer wants to use its own loop instead of the generic loop, it can plug
it in that field.

11 years agoParse and error-check arguments
Alex Pyrgiotis [Thu, 7 Feb 2013 11:58:58 +0000 (13:58 +0200)]
Parse and error-check arguments

11 years agoAdd initial support for benchmarking xseg
Alex Pyrgiotis [Wed, 6 Feb 2013 15:54:15 +0000 (17:54 +0200)]
Add initial support for benchmarking xseg

11 years agoFix grep_xseg to work both with include/exclude
Alex Pyrgiotis [Mon, 4 Feb 2013 10:29:45 +0000 (12:29 +0200)]
Fix grep_xseg to work both with include/exclude

11 years agoFix case where user is root
Alex Pyrgiotis [Fri, 1 Feb 2013 18:47:30 +0000 (20:47 +0200)]
Fix case where user is root

In this case, /home/root/... path doesn't exist, so we use a hacky way
to get the scripts' path.

11 years agoImprove arch-scripts
Alex Pyrgiotis [Fri, 1 Feb 2013 11:12:30 +0000 (13:12 +0200)]
Improve arch-scripts

11 years agoAdd O_EXCL flag when opening segment for creation
Alex Pyrgiotis [Fri, 1 Feb 2013 10:52:53 +0000 (12:52 +0200)]
Add O_EXCL flag when opening segment for creation

Security fix. One could potentially pass a wrong argument in xseg_create
and destroy an already initialized segment. Ensuring creation
exclusiveness makes xseg less error-prone.

11 years agoAdd S_IWUSR permission bit for O_CREAT
Alex Pyrgiotis [Fri, 1 Feb 2013 10:41:14 +0000 (12:41 +0200)]
Add S_IWUSR permission bit for O_CREAT

In Ubuntu -haven't tested it elsewhere yet- gcc has a global cflag
called "D_FORTIFY_SOURCE=2" that errs when files are not created
with explicit user permissions. Since files are O_WRONLY, I added
the S_IWUSR permission bit.

11 years agoAdd simple bash scripts to make and grep xseg
Alex Pyrgiotis [Fri, 1 Feb 2013 09:28:32 +0000 (11:28 +0200)]
Add simple bash scripts to make and grep xseg

11 years agoMerge branch 'feature-impovexseg' into develop
Filippos Giannakos [Wed, 6 Mar 2013 11:27:48 +0000 (13:27 +0200)]
Merge branch 'feature-impovexseg' into develop

11 years agoAdd defer support to peers
Filippos Giannakos [Wed, 27 Feb 2013 16:12:43 +0000 (18:12 +0200)]
Add defer support to peers

Add support to peer skeleton to defer requests to another peer, by forwarding
requests to another port.

11 years agoReimplement xseg paths and add xseg_forward
Filippos Giannakos [Fri, 22 Feb 2013 16:50:13 +0000 (18:50 +0200)]
Reimplement xseg paths and add xseg_forward

Reimplement xseg paths. Paths are a tool for the administrator to dynamically
alter a running xseg setup, by forcing a request to be submitted to a different
port than the original destination.

xseg_forward, on the other hand, allows a peer to alter the normal path of a
request by changing the effective destination of the request, and thus allows
the dynamic creation of temporal paths based on various criteria.

Also, add CAN_RECEIVE and CAN_ACCEPT port flags, allowing each peer to notify
the segment whether it is able to accept or receive on the specified port.

11 years agoChange libxseg minor version to 2
Filippos Giannakos [Fri, 22 Feb 2013 15:55:59 +0000 (17:55 +0200)]
Change libxseg minor version to 2

11 years agoFix permissions in ext storage log directory.
Filippos Giannakos [Wed, 27 Feb 2013 10:27:13 +0000 (12:27 +0200)]
Fix permissions in ext storage log directory.

11 years agoRevert "Add ceph dependency to python archipelago setup.py"
Filippos Giannakos [Tue, 26 Feb 2013 10:47:14 +0000 (12:47 +0200)]
Revert "Add ceph dependency to python archipelago setup.py"

This reverts commit 9c609c9b409d2be8c6559f965770773edf6e22ee.

11 years agoAdd ceph dependency to python archipelago setup.py
Filippos Giannakos [Mon, 25 Feb 2013 16:06:29 +0000 (18:06 +0200)]
Add ceph dependency to python archipelago setup.py

11 years agoRemove dummy, vlmc-xseg, pfiled from peers build.
Filippos Giannakos [Mon, 25 Feb 2013 16:00:41 +0000 (18:00 +0200)]
Remove dummy, vlmc-xseg, pfiled from peers build.

11 years agoBump version.
Filippos Giannakos [Mon, 25 Feb 2013 15:23:48 +0000 (17:23 +0200)]
Bump version.

Bumped version to 0.2next.

11 years agoAdd python xseg dependencies to setup.py
Filippos Giannakos [Mon, 25 Feb 2013 15:23:01 +0000 (17:23 +0200)]
Add python xseg dependencies to setup.py

11 years agoAdd python archipelago dependencies to setup.py
Filippos Giannakos [Mon, 25 Feb 2013 15:22:05 +0000 (17:22 +0200)]
Add python archipelago dependencies to setup.py

11 years agoChange libxseg minor from 0.1 to 1
Filippos Giannakos [Fri, 22 Feb 2013 15:33:38 +0000 (17:33 +0200)]
Change libxseg minor from 0.1 to 1

11 years agoFix xseg, archipelago python cleanup
Filippos Giannakos [Fri, 22 Feb 2013 13:33:23 +0000 (15:33 +0200)]
Fix xseg, archipelago python cleanup

11 years agoFix python package names
Filippos Giannakos [Mon, 18 Feb 2013 17:26:21 +0000 (19:26 +0200)]
Fix python package names

11 years agoFix xseg dependency for archipelago tool
Filippos Giannakos [Mon, 18 Feb 2013 16:19:19 +0000 (18:19 +0200)]
Fix xseg dependency for archipelago tool

11 years agoMake archipelago tool handle updates
Filippos Giannakos [Tue, 12 Feb 2013 17:53:45 +0000 (19:53 +0200)]
Make archipelago tool handle updates

11 years agoMakefile now supports userspace-only build
Filippos Giannakos [Tue, 12 Feb 2013 17:35:31 +0000 (19:35 +0200)]
Makefile now supports userspace-only build

11 years agoMerge branch 'feature-segdev' into develop
Filippos Giannakos [Tue, 12 Feb 2013 15:22:42 +0000 (17:22 +0200)]
Merge branch 'feature-segdev' into develop

11 years agoRemove segdev usercount logging
Filippos Giannakos [Mon, 11 Feb 2013 14:41:09 +0000 (16:41 +0200)]
Remove segdev usercount logging

11 years agoxseg_segdev user driver: close fd on deallocate
Filippos Giannakos [Mon, 4 Feb 2013 17:22:56 +0000 (19:22 +0200)]
xseg_segdev user driver: close fd on deallocate

11 years agoxseg_segdev kernel driver: put segdev on initialization
Filippos Giannakos [Mon, 4 Feb 2013 16:57:14 +0000 (18:57 +0200)]
xseg_segdev kernel driver: put segdev on initialization

11 years agoprint usercount on segdev destroy
Filippos Giannakos [Mon, 4 Feb 2013 16:30:17 +0000 (18:30 +0200)]
print usercount on segdev destroy

11 years agoxsegbd: use xseg leave instead of unmapping the segment
Filippos Giannakos [Mon, 4 Feb 2013 16:28:19 +0000 (18:28 +0200)]
xsegbd: use xseg leave instead of unmapping the segment

11 years agoxseg_segdev user driver: do not try to destroy old segment on allocation
Filippos Giannakos [Mon, 4 Feb 2013 15:52:41 +0000 (17:52 +0200)]
xseg_segdev user driver: do not try to destroy old segment on allocation

11 years agosegdev: put segdev on write error
Filippos Giannakos [Mon, 4 Feb 2013 15:39:25 +0000 (17:39 +0200)]
segdev: put segdev on write error

11 years agoxseg_segdev driver: segdev_put if allocate fails
Filippos Giannakos [Mon, 4 Feb 2013 15:35:27 +0000 (17:35 +0200)]
xseg_segdev driver: segdev_put if allocate fails

11 years agoxseg_segdev driver: do not destroy old segment when allocating a new one
Filippos Giannakos [Mon, 4 Feb 2013 15:33:51 +0000 (17:33 +0200)]
xseg_segdev driver: do not destroy old segment when allocating a new one

11 years agoFix possible race in segdev_get and segdev_destroy segment
Filippos Giannakos [Mon, 4 Feb 2013 15:32:26 +0000 (17:32 +0200)]
Fix possible race in segdev_get and segdev_destroy segment

11 years agoFix autopkg_debian to work with buildbot
Filippos Giannakos [Tue, 12 Feb 2013 11:32:29 +0000 (13:32 +0200)]
Fix autopkg_debian to work with buildbot

11 years agoMerge branch 'feature-merklehash' into develop
Filippos Giannakos [Tue, 12 Feb 2013 11:01:14 +0000 (13:01 +0200)]
Merge branch 'feature-merklehash' into develop

11 years agoMapper: use merkle hash to calculate name of snapshot
Filippos Giannakos [Mon, 11 Feb 2013 10:57:04 +0000 (12:57 +0200)]
Mapper: use merkle hash to calculate name of snapshot

11 years agoMakefiles do not remove version files
Filippos Giannakos [Mon, 11 Feb 2013 18:05:59 +0000 (20:05 +0200)]
Makefiles do not remove version files

11 years agoAdd xseg version file to gitignore
Filippos Giannakos [Mon, 11 Feb 2013 18:04:40 +0000 (20:04 +0200)]
Add xseg version file to gitignore

11 years agoFix wrong entry points in python-xseg
Filippos Giannakos [Mon, 11 Feb 2013 17:26:22 +0000 (19:26 +0200)]
Fix wrong entry points in python-xseg

11 years agovlmc: Fix config STORAGE option in mapinfo
Filippos Giannakos [Mon, 11 Feb 2013 13:38:15 +0000 (15:38 +0200)]
vlmc: Fix config STORAGE option in mapinfo

11 years agoAdd ctypes import to vlmc.py
Filippos Giannakos [Fri, 8 Feb 2013 14:07:35 +0000 (16:07 +0200)]
Add ctypes import to vlmc.py

11 years agoMerge branch 'feature-fixwarnings' into develop
Filippos Giannakos [Fri, 8 Feb 2013 13:44:33 +0000 (15:44 +0200)]
Merge branch 'feature-fixwarnings' into develop

11 years agoFix compilation warnings
Filippos Giannakos [Mon, 4 Feb 2013 14:29:32 +0000 (16:29 +0200)]
Fix compilation warnings

11 years agoPEP8 fixes for python xseg
Filippos Giannakos [Fri, 8 Feb 2013 12:32:56 +0000 (14:32 +0200)]
PEP8 fixes for python xseg

11 years agoPEP8 fixes in vlmc ext_scripts
Filippos Giannakos [Fri, 8 Feb 2013 12:28:53 +0000 (14:28 +0200)]
PEP8 fixes in vlmc ext_scripts

11 years agoMore PEP8 fixes
Filippos Giannakos [Fri, 8 Feb 2013 12:22:45 +0000 (14:22 +0200)]
More PEP8 fixes

11 years agoPEP8 fixes
Filippos Giannakos [Fri, 8 Feb 2013 12:16:59 +0000 (14:16 +0200)]
PEP8 fixes

11 years agoAdjust vlmc ext scripts to the new archipelago module
Filippos Giannakos [Fri, 8 Feb 2013 10:02:48 +0000 (12:02 +0200)]
Adjust vlmc ext scripts to the new archipelago module

11 years agovlmc tool: split showmapped in get mapped and show mapped
Filippos Giannakos [Fri, 8 Feb 2013 09:22:35 +0000 (11:22 +0200)]
vlmc tool: split showmapped in get mapped and show mapped

also add is_mapped functionality

11 years agoFix Xseg_ctx spec parsing
Filippos Giannakos [Thu, 7 Feb 2013 16:17:31 +0000 (18:17 +0200)]
Fix Xseg_ctx spec parsing

11 years agoMake vlmc snapshot return snapshot name on success
Filippos Giannakos [Thu, 7 Feb 2013 15:06:06 +0000 (17:06 +0200)]
Make vlmc snapshot return snapshot name on success