History | View | Annotate | Download (18.3 kB)
workerpool: Preserve task number when deferring
When a task is deferred it should receive the same task ID upon beingreturned to the pool.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
workerpool: Add method to change task's priority
Using the task ID a pending task's priority can be changed. This will beused to change the priority of jobs in the workerpool.
workerpool: Change data structure for priority change
To prepare for the addition of a new function allowing changing apending task's priority, the internal data structure is slightlychanged. The (optional) task ID is stored as part of the task entry. A...
workerpool: Use itertools.count instead of manual counting
Instead of having to explicitely increment the value (“… += 1”), a callto next() is enough. These numbers should in no case be re-used (theyare used for ordering tasks). Using “itertools.count” is useful here as...
workerpool: Don't mask variable in AddManyTasks
The name “priority” is already used.
workerpool: Simplify _WaitForTaskUnlocked
The function in is simplified in its structure and duplicated checkshave been merged.
workerpool: Export function to check for running tasks
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
workerpool: Allow processing of new tasks to be stopped
This is different from “Quiesce” in the sense that this function justchanges an internal flag and doesn't wait for the queue to be empty.Tasks already being processed continue normally, but no new tasks will...
workerpool: Use loop to ignore spurious notifications
This saves us from returning to the worker code when there is notask to be processed.
DeprecationWarning fixes for pylint
In version 0.21, pylint unified all the disable-* (and enable-*)directives to disable (resp. enable). This leads to a lot ofDeprecationWarning being emitted even if one uses the recommendedversion of pylint (0.21.1, as stated in devnotes.rst)....
workerpool: Fix typo
A call to logging.debug was missing an argument, leading to complaintson stderr at runtime.
Merge branch 'devel-2.2'
hansmi helped me with merging the conflict. Thanks
Conflicts: lib/workerpool.py
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
workerpool: Add support for task priority
To add job priorities, the worker pool underlying the job queue mustsupport priorities per task. This patch adds them to the worker pool.
workerpool: Allow setting task name
With this patch, the task name is added to the thread name and will show up inlogs. Log messages from jobs will look like “pid=578/JobQueue14/Job13 mcpu:289DEBUG LU locks acquired/cluster/BGL/shared”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
workerpool: Don't keep reference to task arguments
The workerpool should not keep any reference to task arguments afterthey were processed by RunTask. Doing so led to jobs being cachedby the job queue's WeakValueDictionary for longer than they should've...
workerpool: Add two additional assertions
workerpool: Change signature of AddTask function to not use *args
By changing it to a normal parameter, which must be a sequence, we canstart using keyword parameters.
Before this patch all arguments to “AddTask(self, *args)” were passed asarguments to the worker's “RunTask” method. Priorities, which should be...
workerpool: Move waiting for new tasks for a worker to the pool
This way fewer private variables of the pool are accesssed by the worker.
workerpool: Remove unused worker method
HasRunningTask is never used except for an assertion, where wedon't really need the lock.
workerpool: Additional check in BaseWorker.ShouldTerminate
Document that it should only be called from within RunTask andadd an assertion for this. This means we can no longer use amethod on the pool and hence remove WorkerPool.ShouldWorkerTerminate.
workerpool: Use common function to add tasks
Workerpool.AddManyTasks: check tasks type
Each task has to be a sequence, or the RunTask call will fail.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
WorkerPool.AddManyTasks
Useful if we want to add many tasks at once, without contention with theprevious one we added starting.
workerpool: Simplify log messages
workerpool: Use worker name as thread name
This way it shows up in debug logs.
workerpool: Make worker ID alphanumeric
Having a proper name instead of just a number makes debuggingeasier.
Add targeted pylint disables
This patch should have only:
- pylint disables- docstring changes- whitespace changes
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Olivier Tharan <olive@google.com>
Replace all xrange() with range()
cleanup: use _ for unused loop counter
Reviewed-by: amishchenko
cleanup: WorkerPool, wrong variable name
Quoting Michael: "why is this even working?"
Reviewed-by: imsnah,amishchenko
Small documentation updates for workerpool.py
Reviewed-by: imsnah
Fix pylint-detected issues
This is mostly: - whitespace fix (space at EOL in some files, not all, broken indentation, etc) - variable names overriding others (one is a real bug in there) - too-long-lines - cleanup of most unused imports (not all)...
workerpool: Log when waiting for a thread
Reviewed-by: iustinp
Split conditions in worker pool
This patch splits the single threading.Condition object used in theworker pool for synchronization into three.
- worker_to_pool: Notified if a worker wants to notify the pool- pool_to_worker: Notified if the pool wants to notify a single...
workerpool: Don't notify if there was no task
Workers have to notify their pool if they finished a task to makethe WorkerPool.Quiesce function work. This is done in the finally:clause to notify even in case of an exception. However, beforewe notified on each run, even if there was no task, thereby creating...
Add generic worker pool implementation
Reviewed-by: ultrotter