Filippos Giannakos [Mon, 15 Apr 2013 08:31:53 +0000 (11:31 +0300)]
Close orig stdout, stderr, when logging to file
(cherry picked from commit
1eeb0111a9d7ac5700d7aedc9f442b748b9cb8e7)
Filippos Giannakos [Mon, 15 Apr 2013 08:30:03 +0000 (11:30 +0300)]
Add wait time while starting a peer.
While starting a peer, wait a few seconds, before checking if the peer is up n
running. In high load conditions, this check was too soon and could fail,
resulting in an overall start failure.
(cherry picked from commit
1b4a1ad9cd39b620b35bd30970edda304fcb92a6)
Filippos Giannakos [Mon, 8 Apr 2013 15:37:23 +0000 (18:37 +0300)]
Remove async unsafe functions from signal handlers
(cherry picked from commit
2d2a24a26ff9f3a7fd651b1c2386fbd31166a5e1)
Filippos Giannakos [Mon, 8 Apr 2013 15:37:00 +0000 (18:37 +0300)]
Fix bug in user xseglog2
(cherry picked from commit
c30767e3f311fd97bb1f5f546b8654911d712072)
Filippos Giannakos [Mon, 8 Apr 2013 10:52:28 +0000 (13:52 +0300)]
Fix user space logrotation bug.
Replace file stream functions in user space logging with their file descriptor
counter parts, which are safe to be called from a signal handler.
(cherry picked from commit
342ac1dc1a5ebc9aef55808b2d42436be3efb88d)
Filippos Giannakos [Tue, 26 Mar 2013 11:44:53 +0000 (13:44 +0200)]
Remove ctypeslib dependency from python-xseg
(cherry picked from commit
93f23c27d9567647a27a035732edd2a4ef66d83c)
Filippos Giannakos [Fri, 22 Mar 2013 11:17:23 +0000 (13:17 +0200)]
Move src dir to archipelago-modules-dkms
(cherry picked from commit
9c39a188e8ff0ae49ecb7f6587b693c5a546289a)
Filippos Giannakos [Thu, 21 Mar 2013 14:14:45 +0000 (16:14 +0200)]
Export KVER if none existing.
Also modulesdir now points to the KVER we are building for.
(cherry picked from commit
201e6354bbe2f6f4a4d921992a425e32444514fa)
Filippos Giannakos [Thu, 21 Mar 2013 13:40:25 +0000 (15:40 +0200)]
Add xseg to src dir.
(cherry picked from commit
0e4ae29f8d36524c61720a7f1dcafcdc9cd6bb03)
Alex Pyrgiotis [Tue, 7 May 2013 12:22:34 +0000 (15:22 +0300)]
peer: Move peerd_loop plug out of peer_reqs loop
The plugging of generic_peerd_loop to peerd_loop was falsely done inside
a loop where peer_reqs were initialized. The plugging has now been moved
out of this loop.
Alex Pyrgiotis [Tue, 23 Apr 2013 12:55:46 +0000 (15:55 +0300)]
bench: Flush stdout when printing progress results
Alex Pyrgiotis [Thu, 18 Apr 2013 15:34:17 +0000 (18:34 +0300)]
bench: Fix issuing of read/write requests
Due to no initialization of struct bench, some "ifs" in the code would
show different behavior depending on the junk values they would check.
Now however, there is proper initialization of struct bench.
Alex Pyrgiotis [Wed, 17 Apr 2013 11:30:02 +0000 (14:30 +0300)]
bench: Include bandwidth and IOPS in results
Alex Pyrgiotis [Wed, 17 Apr 2013 11:24:17 +0000 (14:24 +0300)]
bench: Change prefixes of static functions
In this repo, a static helpder function is prefixed with "__". However,
some static helper functions in bench-utils.c where prefixed with a
single underscore ("_"), so we updated their names to be consistent with
this convention.
Alex Pyrgiotis [Fri, 15 Mar 2013 15:43:03 +0000 (17:43 +0200)]
Add progress report for bench
Progress report is turned on by default. It reports every 5% the current
request status.
Also, we now make sure to terminate only if all peer requests are free.
Alex Pyrgiotis [Thu, 14 Mar 2013 16:29:23 +0000 (18:29 +0200)]
Add a fallback clock for portability reasons
Filippos Giannakos [Thu, 14 Mar 2013 16:00:33 +0000 (18:00 +0200)]
Merge branch 'feature-improvepeer' into develop
Filippos Giannakos [Thu, 14 Mar 2013 13:34:37 +0000 (15:34 +0200)]
Remove unused stuff and reduce warnings
Remove unused pthread_* stuff and modify variable location to reduce warnings
because of the define-based compilation.
Filippos Giannakos [Thu, 14 Mar 2013 11:41:24 +0000 (13:41 +0200)]
Add compilation warning about MT and ST_THREADS
Filippos Giannakos [Thu, 14 Mar 2013 11:03:03 +0000 (13:03 +0200)]
Fix queue size of per thread queue.
Filippos Giannakos [Thu, 14 Mar 2013 09:27:34 +0000 (11:27 +0200)]
Fix peer termination.
Peer checks if all peer requests are free in all thread, and then proceeds to
gracefull shutdown.
Also introduce pthread specific variable, holding the struct thread for each
thread.
Filippos Giannakos [Wed, 13 Mar 2013 11:45:10 +0000 (13:45 +0200)]
Split peer request queues.
Each thread has its own peer request queue, frow which it allocates data. If
this queue is empty, it tries to "steal" from the "next" thread. This greatly
reduces lock congestion to the peer request queue.
Filippos Giannakos [Fri, 8 Mar 2013 17:10:54 +0000 (19:10 +0200)]
Add check to not respond request if none associated.
Conflicts:
xseg/peers/user/peer.c
Filippos Giannakos [Thu, 14 Mar 2013 16:00:04 +0000 (18:00 +0200)]
Merge branch 'feature-improvepthreadsignaling' into develop
Filippos Giannakos [Thu, 14 Mar 2013 09:20:42 +0000 (11:20 +0200)]
Improve pthread signaling.
Remove xpool data type that required a lock on insertion / on seek and replace
it by an array of pids. On the array of pids, each thread place its pid in his
own reserved entry. Any one who wishes to signal a pthread peer, checks that
array for a valid pid.
Filippos Giannakos [Thu, 14 Mar 2013 09:12:26 +0000 (11:12 +0200)]
Disable update stamp during request transfer.
Alex Pyrgiotis [Thu, 14 Mar 2013 15:51:49 +0000 (17:51 +0200)]
Fix clean option in Makefile
Filippos Giannakos [Thu, 14 Mar 2013 10:11:36 +0000 (12:11 +0200)]
Merge branch 'feature-bench-rebased' into develop
Filippos Giannakos [Thu, 14 Mar 2013 10:10:32 +0000 (12:10 +0200)]
Build only the necessary peers.
Alex Pyrgiotis [Wed, 13 Mar 2013 18:14:57 +0000 (20:14 +0200)]
Fix LFSR spin bug
Alex Pyrgiotis [Wed, 13 Mar 2013 16:04:35 +0000 (18:04 +0200)]
Style changes
Alex Pyrgiotis [Wed, 13 Mar 2013 16:00:29 +0000 (18:00 +0200)]
Add option for read/write ops to hit unique objs
Read/write ops can now use the -to option, instead of the -ts option, to
make each read/write request touch a different object.
Alex Pyrgiotis [Wed, 13 Mar 2013 13:12:10 +0000 (15:12 +0200)]
Add request cap
Implemented by popular request, the request cap option allows the
benchmarking peer to stop the benchmark earlier, at a given number
of received requests.
Alex Pyrgiotis [Wed, 13 Mar 2013 11:52:42 +0000 (13:52 +0200)]
Fix verification full mode
Alex Pyrgiotis [Tue, 12 Mar 2013 16:29:54 +0000 (18:29 +0200)]
Add partial support for verification
Adds an option to choose between verification modes:
1. meta: On this mode, we write our signature both at the start and the
end of the chunk size. It should be very fast and moderately safe,
due to the fact that corruptions happen it the chunk's edges.
2. full: Write our a random sequence of numbers throughout the chunk's
data range, which is reproducible given the object name and the
offset inside the object.
At the moment, only meta mode is available, but full will be supported
soon
Alex Pyrgiotis [Tue, 12 Mar 2013 16:16:50 +0000 (18:16 +0200)]
Refactor struct bench and appropriate code paths
Changes:
1. Move flag values (such as pattern, insanity, verify) in a specific
bit-field.
2. Create a struct req_status where the status of requests will be
stored.
3. Tidy up code.
Rationale:
1. For parameters that can have only a few different values, it's an
overkill to use a specific field in struct bench to store them, when
we can just as well store them in a flag. This reduces the size of
struct bench and makes the code easier to grasp.
2. Previously, the requests' statuses would be scattered amongst
different timers, which is counter-intuitive at best. By placing them
in a specific struct, we make the code more intuitive and readable.
Alex Pyrgiotis [Sun, 10 Mar 2013 23:37:52 +0000 (01:37 +0200)]
Add skeleton for verification in write path
Alex Pyrgiotis [Sun, 10 Mar 2013 11:49:25 +0000 (13:49 +0200)]
Improve LFSR implementation
Also, prepare the code for the addition of verification support
Alex Pyrgiotis [Sat, 2 Mar 2013 11:22:14 +0000 (13:22 +0200)]
Change input option to bench
Since -dp is reserved in peer as defer port number, we change the -dp
option in bench (which stood for "destination port") to -tp, which
stands for "target port".
Alex Pyrgiotis [Wed, 27 Feb 2013 18:52:12 +0000 (20:52 +0200)]
Add support for seed values given by the user
A new option has been added (--seed) which allows the user to pre-define
the seed that he/she wants. This has the following effects:
* The target names are prefixed with the first 9 digits of the seed.
* LFSR produces a predictable sequence of "random" numbers
Alex Pyrgiotis [Mon, 25 Feb 2013 16:52:35 +0000 (18:52 +0200)]
Minor fixes
Alex Pyrgiotis [Mon, 25 Feb 2013 16:42:29 +0000 (18:42 +0200)]
Allow LFSR to be used for sizes smaller than 2^3
Alex Pyrgiotis [Sat, 23 Feb 2013 12:42:15 +0000 (14:42 +0200)]
Get correct iodepth
Alex Pyrgiotis [Thu, 21 Feb 2013 15:51:32 +0000 (17:51 +0200)]
Fix incorrect reading of "insanity" parameter
Alex Pyrgiotis [Thu, 21 Feb 2013 12:04:19 +0000 (14:04 +0200)]
Fix identation
Alex Pyrgiotis [Thu, 21 Feb 2013 10:44:46 +0000 (12:44 +0200)]
Make bench snappier when asked to terminate
Alex Pyrgiotis [Thu, 21 Feb 2013 10:34:35 +0000 (12:34 +0200)]
Print more useful results
Alex Pyrgiotis [Wed, 20 Feb 2013 22:16:55 +0000 (00:16 +0200)]
Fix signal handling of peers
Alex Pyrgiotis [Wed, 20 Feb 2013 15:28:23 +0000 (17:28 +0200)]
Tweak performance of bench
Alex Pyrgiotis [Wed, 20 Feb 2013 07:58:43 +0000 (09:58 +0200)]
Fix LFSR behavior
Alex Pyrgiotis [Tue, 19 Feb 2013 17:15:05 +0000 (19:15 +0200)]
Fix flag handling of struct bench
Alex Pyrgiotis [Tue, 19 Feb 2013 16:59:37 +0000 (18:59 +0200)]
Fix target names and chunk offsets
Alex Pyrgiotis [Tue, 19 Feb 2013 15:24:06 +0000 (17:24 +0200)]
Increment completed requests of insane timers
Even if a timer is not used for a benchmark, we need to increment its
completed requests since these are needed to calculate how many requests
are left etc.
Alex Pyrgiotis [Tue, 19 Feb 2013 14:28:43 +0000 (16:28 +0200)]
Implement most semi-finished functionalities
Summary of new functionalities:
* Add support for 4 different xseg operations (read, write, info,
delete).
* Add support for random I/O patterns with a fast Galois LFSR.
* Add insanity levels in timers, which allow the user to exclude obscure
timers from the benchmark (or explicitly allow them)
* Generate target names based on seeds given by the user or produced by
the benchmark (useful for reproducibility of benchmark)
Alex Pyrgiotis [Mon, 18 Feb 2013 19:56:20 +0000 (21:56 +0200)]
Add an LFSR PRNG for bench
The PRNG introduced by this commit is a Galois-XNOR LFSR, which can
generate unique pseudorandom numbers. Also, seeded with the same
seed, it will produce the same sequence.
This implementation is driven by the needs of XSEG for a blazing fast
random generator. Initial tests show that it's capable of producing 2^38
unique pseudorandom numbers in less than 10 minutes (which equals to 2
nanoseconds per number).
Alex Pyrgiotis [Sat, 16 Feb 2013 19:24:24 +0000 (21:24 +0200)]
Add utility functions to header file
Alex Pyrgiotis [Sat, 16 Feb 2013 18:51:10 +0000 (20:51 +0200)]
Design the program's skeleton for new features
Bench peer must be able to test all xseg operations in a synchronous or
asynchronous way. Also, it must have a validation mode to sanity-check
if xseg requests are issued properly. Most of the above are present in
this commit, either in code or TODO form.
WARNING: code is incomplete and won't compile
Alex Pyrgiotis [Fri, 15 Feb 2013 09:40:11 +0000 (11:40 +0200)]
Fix segfault, parameters validation
Alex Pyrgiotis [Thu, 14 Feb 2013 18:00:03 +0000 (20:00 +0200)]
Initial support for xseg ops
Alex Pyrgiotis [Thu, 14 Feb 2013 13:38:51 +0000 (15:38 +0200)]
Alpha release of bench peer
On this release, the bench peer can measure the total elapsed time
of the benchmark. TODO:
* Allow bench peer to be multi-threaded.
* Adjust benchmarking precision, by eliminating or including more
trivial timers.
* Measure mean time, standard deviation of requests.
Alex Pyrgiotis [Tue, 12 Feb 2013 17:39:26 +0000 (19:39 +0200)]
Make bench send requests faster
Alex Pyrgiotis [Tue, 12 Feb 2013 12:02:58 +0000 (14:02 +0200)]
Fix missing function declaration
Alex Pyrgiotis [Tue, 12 Feb 2013 11:28:37 +0000 (13:28 +0200)]
Simplify port binding code
Alex Pyrgiotis [Tue, 12 Feb 2013 07:02:09 +0000 (09:02 +0200)]
Fix segfault
Alex Pyrgiotis [Tue, 12 Feb 2013 06:57:14 +0000 (08:57 +0200)]
Keep naming in peerd_loop consistent
Alex Pyrgiotis [Mon, 11 Feb 2013 16:38:06 +0000 (18:38 +0200)]
Fix timers and add them to critical paths
Fixed incorrect handling of elapsed time and used appropriate timers to
benchmark crtical paths.
Alex Pyrgiotis [Mon, 11 Feb 2013 16:34:50 +0000 (18:34 +0200)]
Complete unification of thread_loop/peerd_loop
Alex Pyrgiotis [Mon, 11 Feb 2013 07:00:08 +0000 (09:00 +0200)]
Fix segfault
Alex Pyrgiotis [Mon, 11 Feb 2013 01:10:12 +0000 (03:10 +0200)]
Unify thread loop with peerd_loop
Since peerd_loop with thread_loop don't have many differences between
them, they can be unified in a single loop. TODO is to choose a unified
name for both threads and peers, as well as fault check the
implementation.
Alex Pyrgiotis [Sat, 9 Feb 2013 11:32:49 +0000 (13:32 +0200)]
Add timer implementation
This an initial timer implementation. It is not very polished at the
momment and some implementation choices will have to be reviewed later
on.
Alex Pyrgiotis [Fri, 8 Feb 2013 09:28:04 +0000 (11:28 +0200)]
Add support for custom peer loop
There is a new field in struct peerd called "custom_peerd_loop". If any
peer wants to use its own loop instead of the generic loop, it can plug
it in that field.
Alex Pyrgiotis [Thu, 7 Feb 2013 11:58:58 +0000 (13:58 +0200)]
Parse and error-check arguments
Alex Pyrgiotis [Wed, 6 Feb 2013 15:54:15 +0000 (17:54 +0200)]
Add initial support for benchmarking xseg
Alex Pyrgiotis [Mon, 4 Feb 2013 10:29:45 +0000 (12:29 +0200)]
Fix grep_xseg to work both with include/exclude
Alex Pyrgiotis [Fri, 1 Feb 2013 18:47:30 +0000 (20:47 +0200)]
Fix case where user is root
In this case, /home/root/... path doesn't exist, so we use a hacky way
to get the scripts' path.
Alex Pyrgiotis [Fri, 1 Feb 2013 11:12:30 +0000 (13:12 +0200)]
Improve arch-scripts
Alex Pyrgiotis [Fri, 1 Feb 2013 10:52:53 +0000 (12:52 +0200)]
Add O_EXCL flag when opening segment for creation
Security fix. One could potentially pass a wrong argument in xseg_create
and destroy an already initialized segment. Ensuring creation
exclusiveness makes xseg less error-prone.
Alex Pyrgiotis [Fri, 1 Feb 2013 10:41:14 +0000 (12:41 +0200)]
Add S_IWUSR permission bit for O_CREAT
In Ubuntu -haven't tested it elsewhere yet- gcc has a global cflag
called "D_FORTIFY_SOURCE=2" that errs when files are not created
with explicit user permissions. Since files are O_WRONLY, I added
the S_IWUSR permission bit.
Alex Pyrgiotis [Fri, 1 Feb 2013 09:28:32 +0000 (11:28 +0200)]
Add simple bash scripts to make and grep xseg
Filippos Giannakos [Wed, 6 Mar 2013 11:27:48 +0000 (13:27 +0200)]
Merge branch 'feature-impovexseg' into develop
Filippos Giannakos [Wed, 27 Feb 2013 16:12:43 +0000 (18:12 +0200)]
Add defer support to peers
Add support to peer skeleton to defer requests to another peer, by forwarding
requests to another port.
Filippos Giannakos [Fri, 22 Feb 2013 16:50:13 +0000 (18:50 +0200)]
Reimplement xseg paths and add xseg_forward
Reimplement xseg paths. Paths are a tool for the administrator to dynamically
alter a running xseg setup, by forcing a request to be submitted to a different
port than the original destination.
xseg_forward, on the other hand, allows a peer to alter the normal path of a
request by changing the effective destination of the request, and thus allows
the dynamic creation of temporal paths based on various criteria.
Also, add CAN_RECEIVE and CAN_ACCEPT port flags, allowing each peer to notify
the segment whether it is able to accept or receive on the specified port.
Filippos Giannakos [Fri, 22 Feb 2013 15:55:59 +0000 (17:55 +0200)]
Change libxseg minor version to 2
Filippos Giannakos [Wed, 27 Feb 2013 10:27:13 +0000 (12:27 +0200)]
Fix permissions in ext storage log directory.
Filippos Giannakos [Tue, 26 Feb 2013 10:47:14 +0000 (12:47 +0200)]
Revert "Add ceph dependency to python archipelago setup.py"
This reverts commit
9c609c9b409d2be8c6559f965770773edf6e22ee.
Filippos Giannakos [Mon, 25 Feb 2013 16:06:29 +0000 (18:06 +0200)]
Add ceph dependency to python archipelago setup.py
Filippos Giannakos [Mon, 25 Feb 2013 16:00:41 +0000 (18:00 +0200)]
Remove dummy, vlmc-xseg, pfiled from peers build.
Filippos Giannakos [Mon, 25 Feb 2013 15:23:48 +0000 (17:23 +0200)]
Bump version.
Bumped version to 0.2next.
Filippos Giannakos [Mon, 25 Feb 2013 15:23:01 +0000 (17:23 +0200)]
Add python xseg dependencies to setup.py
Filippos Giannakos [Mon, 25 Feb 2013 15:22:05 +0000 (17:22 +0200)]
Add python archipelago dependencies to setup.py
Filippos Giannakos [Fri, 22 Feb 2013 15:33:38 +0000 (17:33 +0200)]
Change libxseg minor from 0.1 to 1
Filippos Giannakos [Fri, 22 Feb 2013 13:33:23 +0000 (15:33 +0200)]
Fix xseg, archipelago python cleanup
Filippos Giannakos [Mon, 18 Feb 2013 17:26:21 +0000 (19:26 +0200)]
Fix python package names
Filippos Giannakos [Mon, 18 Feb 2013 16:19:19 +0000 (18:19 +0200)]
Fix xseg dependency for archipelago tool
Filippos Giannakos [Tue, 12 Feb 2013 17:53:45 +0000 (19:53 +0200)]
Make archipelago tool handle updates
Filippos Giannakos [Tue, 12 Feb 2013 17:35:31 +0000 (19:35 +0200)]
Makefile now supports userspace-only build
Filippos Giannakos [Tue, 12 Feb 2013 15:22:42 +0000 (17:22 +0200)]
Merge branch 'feature-segdev' into develop
Filippos Giannakos [Mon, 11 Feb 2013 14:41:09 +0000 (16:41 +0200)]
Remove segdev usercount logging
Filippos Giannakos [Mon, 4 Feb 2013 17:22:56 +0000 (19:22 +0200)]
xseg_segdev user driver: close fd on deallocate
Filippos Giannakos [Mon, 4 Feb 2013 16:57:14 +0000 (18:57 +0200)]
xseg_segdev kernel driver: put segdev on initialization