l.s.t.t.ParallelLimitedTaskConsumer : class documentation

Part of lp.services.twistedsupport.task View In Hierarchy

A consumer that runs tasks with limited parallelism.

Assumes that the task source generates tasks that are nullary callables
that might return `Deferred`s.
Method __init__ Undocumented
Method consume Start consuming tasks from 'task_source'.
Method taskStarted See ITaskConsumer.
Method noTasksFound See ITaskConsumer.
Method taskProductionFailed See ITaskConsumer.
Method _log_state Undocumented
Method _stop Undocumented
Method _taskEnded Handle a task reaching completion.
def __init__(self, worker_limit, logger=None):
def _log_state(self, method, action=''):
def _stop(self):
def consume(self, task_source):
Start consuming tasks from 'task_source'.
Parameterstask_sourceAn ITaskSource provider.
ReturnsA Deferred that fires when the task source is exhausted and we are not running any tasks.
RaisesAlreadyRunningErrorIf 'consume' has already been called on this consumer.
def taskStarted(self, task):
See ITaskConsumer.

Stops the task source when we reach the maximum number of concurrent tasks.

RaisesNotRunningErrorif 'consume' has not yet been called.
def noTasksFound(self):
See ITaskConsumer.

Called when the producer found no tasks. If we are not currently running any workers, exit.

def taskProductionFailed(self, reason):
See ITaskConsumer.

Called by the task source when a failure occurs while producing a task. When this happens, we stop the task source. Any currently running tasks will finish, and each time this happens, we'll ask the task source to start again.

If the source keeps failing, we'll eventually have no tasks running, at which point we stop the source and fire the termination deferred, signalling the end of this run.

This approach allows us to handle intermittent failures gracefully (by retrying the next time a task finishes), and to handle persistent failures well (by shutting down when there are no more tasks left).

RaisesNotRunningErrorif 'consume' has not yet been called.
def _taskEnded(self, ignored):
Handle a task reaching completion.

Reduces the number of concurrent workers. If there are no running workers then we fire the termination deferred, signalling the end of the run.

If there are available workers, we ask the task source to start producing jobs.

API Documentation for Launchpad, generated by pydoctor at 2019-09-17 00:00:34.