Statistics
| Branch: | Tag: | Revision:

root / lib / workerpool.py @ 653bc0f1

History | View | Annotate | Download (18.3 kB)

# Date Author Comment
bba69414 11/13/2012 09:20 pm Michael Hanselmann

workerpool: Preserve task number when deferring

When a task is deferred it should receive the same task ID upon being
returned to the pool.

Signed-off-by: Michael Hanselmann <>
Reviewed-by: Bernardo Dal Seno <>

9a2564e7 11/13/2012 09:20 pm Michael Hanselmann

workerpool: Add method to change task's priority

Using the task ID a pending task's priority can be changed. This will be
used to change the priority of jobs in the workerpool.

Signed-off-by: Michael Hanselmann <>
Reviewed-by: Bernardo Dal Seno <>

125b74b2 11/13/2012 09:20 pm Michael Hanselmann

workerpool: Change data structure for priority change

To prepare for the addition of a new function allowing changing a
pending task's priority, the internal data structure is slightly
changed. The (optional) task ID is stored as part of the task entry. A...

c258f110 11/07/2012 03:35 pm Michael Hanselmann

workerpool: Use itertools.count instead of manual counting

Instead of having to explicitely increment the value (“… += 1”), a call
to next() is enough. These numbers should in no case be re-used (they
are used for ordering tasks). Using “itertools.count” is useful here as...

f94779f5 11/06/2012 03:56 pm Michael Hanselmann

workerpool: Don't mask variable in AddManyTasks

The name “priority” is already used.

Signed-off-by: Michael Hanselmann <>
Reviewed-by: Bernardo Dal Seno <>

c69c45a7 11/06/2012 03:56 pm Michael Hanselmann

workerpool: Simplify _WaitForTaskUnlocked

The function in is simplified in its structure and duplicated checks
have been merged.

Signed-off-by: Michael Hanselmann <>
Reviewed-by: Bernardo Dal Seno <>

ef52306a 11/21/2011 09:36 am Michael Hanselmann

workerpool: Export function to check for running tasks

Signed-off-by: Michael Hanselmann <>
Reviewed-by: Iustin Pop <>

27caa993 11/17/2011 03:39 pm Michael Hanselmann

workerpool: Allow processing of new tasks to be stopped

This is different from “Quiesce” in the sense that this function just
changes an internal flag and doesn't wait for the queue to be empty.
Tasks already being processed continue normally, but no new tasks will...

2db05c94 11/17/2011 03:39 pm Michael Hanselmann

workerpool: Use loop to ignore spurious notifications

This saves us from returning to the worker code when there is no
task to be processed.

Signed-off-by: Michael Hanselmann <>
Reviewed-by: Iustin Pop <>

b459a848 08/30/2011 11:24 am Andrea Spadaccini

DeprecationWarning fixes for pylint

In version 0.21, pylint unified all the disable-* (and enable-*)
directives to disable (resp. enable). This leads to a lot of
DeprecationWarning being emitted even if one uses the recommended
version of pylint (0.21.1, as stated in devnotes.rst)....

e1ea54e9 09/16/2010 01:26 pm Michael Hanselmann

workerpool: Fix typo

A call to logging.debug was missing an argument, leading to complaints
on stderr at runtime.

Signed-off-by: Michael Hanselmann <>
Reviewed-by: Iustin Pop <>

c30421e0 08/25/2010 12:56 pm René Nussbaumer

Merge branch 'devel-2.2'

hansmi helped me with merging the conflict. Thanks

Conflicts:
lib/workerpool.py

Signed-off-by: René Nussbaumer <>
Reviewed-by: Iustin Pop <>

52c47e4e 08/24/2010 06:32 pm Michael Hanselmann

workerpool: Add support for task priority

To add job priorities, the worker pool underlying the job queue must
support priorities per task. This patch adds them to the worker pool.

Signed-off-by: Michael Hanselmann <>
Reviewed-by: Iustin Pop <>

daba67c7 08/24/2010 05:27 pm Michael Hanselmann

workerpool: Allow setting task name

With this patch, the task name is added to the thread name and will show up in
logs. Log messages from jobs will look like “pid=578/JobQueue14/Job13 mcpu:289
DEBUG LU locks acquired/cluster/BGL/shared”.

Signed-off-by: Michael Hanselmann <>...

46d0a3d0 08/13/2010 08:25 pm Michael Hanselmann

workerpool: Don't keep reference to task arguments

The workerpool should not keep any reference to task arguments after
they were processed by RunTask. Doing so led to jobs being cached
by the job queue's WeakValueDictionary for longer than they should've...

c1cf1fe5 07/29/2010 04:05 pm Michael Hanselmann

workerpool: Add two additional assertions

Signed-off-by: Michael Hanselmann <>
Reviewed-by: Iustin Pop <>

b2e8a4d9 07/29/2010 04:05 pm Michael Hanselmann

workerpool: Change signature of AddTask function to not use *args

By changing it to a normal parameter, which must be a sequence, we can
start using keyword parameters.

Before this patch all arguments to “AddTask(self, *args)” were passed as
arguments to the worker's “RunTask” method. Priorities, which should be...

21c5ad52 07/29/2010 04:05 pm Michael Hanselmann

workerpool: Move waiting for new tasks for a worker to the pool

This way fewer private variables of the pool are accesssed by the worker.

Signed-off-by: Michael Hanselmann <>
Reviewed-by: Iustin Pop <>

ccedb11b 07/29/2010 04:05 pm Michael Hanselmann

workerpool: Remove unused worker method

HasRunningTask is never used except for an assertion, where we
don't really need the lock.

Signed-off-by: Michael Hanselmann <>
Reviewed-by: Iustin Pop <>

2f4e1516 07/29/2010 04:05 pm Michael Hanselmann

workerpool: Additional check in BaseWorker.ShouldTerminate

Document that it should only be called from within RunTask and
add an assertion for this. This means we can no longer use a
method on the pool and hence remove WorkerPool.ShouldWorkerTerminate.

Signed-off-by: Michael Hanselmann <>...

189d2714 07/29/2010 04:05 pm Michael Hanselmann

workerpool: Use common function to add tasks

Signed-off-by: Michael Hanselmann <>
Reviewed-by: Iustin Pop <>

25e557a5 06/17/2010 08:25 pm Guido Trotter

Workerpool.AddManyTasks: check tasks type

Each task has to be a sequence, or the RunTask call will fail.

Signed-off-by: Guido Trotter <>
Reviewed-by: Michael Hanselmann <>

c2a8e8ba 06/17/2010 01:00 pm Guido Trotter

WorkerPool.AddManyTasks

Useful if we want to add many tasks at once, without contention with the
previous one we added starting.

Signed-off-by: Guido Trotter <>
Reviewed-by: Michael Hanselmann <>

02fc74da 01/13/2010 03:03 pm Michael Hanselmann

workerpool: Simplify log messages

Signed-off-by: Michael Hanselmann <>
Reviewed-by: Iustin Pop <>

d16e6fd9 01/13/2010 03:03 pm Michael Hanselmann

workerpool: Use worker name as thread name

This way it shows up in debug logs.

Signed-off-by: Michael Hanselmann <>
Reviewed-by: Iustin Pop <>

89e2b4d2 01/13/2010 03:02 pm Michael Hanselmann

workerpool: Make worker ID alphanumeric

Having a proper name instead of just a number makes debugging
easier.

Signed-off-by: Michael Hanselmann <>
Reviewed-by: Iustin Pop <>

7260cfbe 01/04/2010 11:15 am Iustin Pop

Add targeted pylint disables

This patch should have only:

- pylint disables
- docstring changes
- whitespace changes

Signed-off-by: Iustin Pop <>
Reviewed-by: Olivier Tharan <>

f1501b3f 10/02/2009 06:52 pm Michael Hanselmann

Replace all xrange() with range()

Signed-off-by: Michael Hanselmann <>
Reviewed-by: Iustin Pop <>

ad1bf20c 12/14/2008 02:04 pm Iustin Pop

cleanup: use _ for unused loop counter

Reviewed-by: amishchenko

805f0c07 12/14/2008 02:04 pm Iustin Pop

cleanup: WorkerPool, wrong variable name

Quoting Michael: "why is this even working?"

Reviewed-by: imsnah,amishchenko

116db7c7 10/28/2008 01:21 am Iustin Pop

Small documentation updates for workerpool.py

Reviewed-by: imsnah

38206f3c 07/30/2008 05:04 pm Iustin Pop

Fix pylint-detected issues

This is mostly:
- whitespace fix (space at EOL in some files, not all, broken
indentation, etc)
- variable names overriding others (one is a real bug in there)
- too-long-lines
- cleanup of most unused imports (not all)...

c0a8eb9e 07/30/2008 11:56 am Michael Hanselmann

workerpool: Log when waiting for a thread

Reviewed-by: iustinp

53b1d12b 07/22/2008 11:17 am Michael Hanselmann

Split conditions in worker pool

This patch splits the single threading.Condition object used in the
worker pool for synchronization into three.

- worker_to_pool: Notified if a worker wants to notify the pool
- pool_to_worker: Notified if the pool wants to notify a single...

b3558df1 07/08/2008 06:03 pm Michael Hanselmann

workerpool: Don't notify if there was no task

Workers have to notify their pool if they finished a task to make
the WorkerPool.Quiesce function work. This is done in the finally:
clause to notify even in case of an exception. However, before
we notified on each run, even if there was no task, thereby creating...

76094e37 07/04/2008 06:34 pm Michael Hanselmann

Add generic worker pool implementation

Reviewed-by: ultrotter