lp.services.twistedsupport.task.ParallelLimitedTaskConsumer : API documentation

Method	__init__	Undocumented
Method	consume	Start consuming tasks from 'task_source'.
Method	taskStarted	See `ITaskConsumer`.
Method	noTasksFound	See `ITaskConsumer`.
Method	taskProductionFailed	See `ITaskConsumer`.
Method	_log_state	Undocumented
Method	_stop	Undocumented
Method	_taskEnded	Handle a task reaching completion.

def __init__(self, worker_limit, logger=None):

Undocumented

def _log_state(self, method, action=''):

Undocumented

def _stop(self):

Undocumented

def consume(self, task_source):

Start consuming tasks from 'task_source'.

Parameters	task_source	An `ITaskSource` provider.
Returns	A `Deferred` that fires when the task source is exhausted and we are not running any tasks.
Raises	AlreadyRunningError	If 'consume' has already been called on this consumer.

def taskStarted(self, task):

See ITaskConsumer.

Stops the task source when we reach the maximum number of concurrent tasks.

Raises NotRunningError if 'consume' has not yet been called.

def noTasksFound(self):

See ITaskConsumer.

Called when the producer found no tasks. If we are not currently running any workers, exit.

def taskProductionFailed(self, reason):

See ITaskConsumer.

Called by the task source when a failure occurs while producing a task. When this happens, we stop the task source. Any currently running tasks will finish, and each time this happens, we'll ask the task source to start again.

If the source keeps failing, we'll eventually have no tasks running, at which point we stop the source and fire the termination deferred, signalling the end of this run.

This approach allows us to handle intermittent failures gracefully (by retrying the next time a task finishes), and to handle persistent failures well (by shutting down when there are no more tasks left).

Raises NotRunningError if 'consume' has not yet been called.

def _taskEnded(self, ignored):

Handle a task reaching completion.

Reduces the number of concurrent workers. If there are no running workers then we fire the termination deferred, signalling the end of the run.

If there are available workers, we ask the task source to start producing jobs.

l.s.t.t.ParallelLimitedTaskConsumer : class documentation