l.s.l.LoopTuner : class documentation

Part of lp.services.looptuner View In Hierarchy

Known subclasses: lp.services.looptuner.DBLoopTuner

A loop that tunes itself to approximate an ideal time per iteration.

Use this for large processing jobs that need to be broken down into chunks of such size that processing a single chunk takes approximately a given ideal time. For example, large database operations may have to be performed and committed in a large number of small steps in order to avoid locking out other clients that need to access the same data. Regular commits allow other clients to get their work done.

In such a situation, committing for every step is often far too costly. Imagine inserting a million rows and committing after every row! You could hand-pick a static number of steps per commit, but that takes a lot of experimental guesswork and it will still waste time when things go well, and on the other hand, it will still end up taking too much time per batch when the system slows down for whatever reason.

Instead, define your loop body in an ITunableLoop; parameterize it on the number of steps per batch; say how much time you'd like to spend per batch; and pass it to a LoopTuner. The LoopTuner will execute your loop, dynamically tuning its batch-size parameter to stay close to your time goal. If things go faster than expected, it will ask your loop body to do more work for the next batch. If a batch takes too much time, the next batch will be smaller. There is also some cushioning for one-off spikes and troughs in processing speed.

Method __init__ Initialize a loop, to be run to completion at most once.
Method run Run the loop to completion.
Method _isTimedOut Return True if the task will be timed out in extra_seconds.
Method _coolDown Sleep for self.cooldown_time seconds, if set.
Method _time Monotonic system timer with unit of 1 second.
Method _sleep Sleep.
def __init__(self, operation, goal_seconds, minimum_chunk_size=1, maximum_chunk_size=1000000000, abort_time=None, cooldown_time=None, log=None):

Initialize a loop, to be run to completion at most once.

Parameters:

operation: an object implementing the loop body. It must support the
ITunableLoop interface.
goal_seconds: the ideal number of seconds for any one iteration to
take. The algorithm will vary chunk size in order to stick close to this ideal.
minimum_chunk_size: the smallest chunk size that is reasonable. The
tuning algorithm will never let chunk size sink below this value.
maximum_chunk_size: the largest allowable chunk size. A maximum is
needed even if the ITunableLoop ignores chunk size for whatever reason, since reaching floating-point infinity would seriously break the algorithm's arithmetic.
cooldown_time: time (in seconds, float) to sleep between consecutive
operation runs. Defaults to None for no sleep.
abort_time: abort the loop, logging a WARNING message, if the runtime
takes longer than this many seconds.
log: The log object to use. DEBUG level messages are logged
giving iteration statistics.
def _isTimedOut(self, extra_seconds=0):
Return True if the task will be timed out in extra_seconds.

If this method returns True, all future calls will also return True.

def run(self):
Run the loop to completion.
def _coolDown(self, bedtime):
overridden in lp.services.looptuner.DBLoopTuner
Sleep for `self.cooldown_time` seconds, if set.

Assumes that anything the main LoopTuner loop does apart from
doing a chunk of work or sleeping takes zero time.

:param bedtime: Time the cooldown started, i.e. the time the
chunk of real work was completed.
:return: Time when cooldown completed, i.e. the starting time
for a next chunk of work.
def _time(self):
Monotonic system timer with unit of 1 second.

Overridable so tests can fake processing speeds accurately and without actually waiting.

def _sleep(self, seconds):
Sleep.

If the sleep interval would put us over the tasks timeout, do nothing.

API Documentation for Launchpad, generated by pydoctor at 2022-06-16 00:00:12.