The “Code Import” Pattern For Job Processing

Michael Hudson michael.hudson@canonical.com

Jonathan Lange jonathan.lange@canonical.com

I spent a large fraction of my first year at Canonical designing and implementing a system to replace buildbot for code imports. While the system we came up with is quite complex, it works: it handles the scheduling of jobs whose runtime may range from a few seconds to a couple of weeks, that may fail or get stuck intermittently and in arbitrarily strange ways and may gobble up all the RAM in the machine they're running on, all with no manual intervention.

In this talk I'll explain the architecture of the system, and then I'd like to spark a discussion: there are other Launchpad Code systems which would benefit from this architecture, and it would be interesting to talk about how to do this and how to share as much code as possible. It would also be interesting to find out about similar needs of other teams.

The Problem

Launchpad's code import system provides Bazaar mirrors of remote Subversion and CVS repositories. This talk is not about the process of converting Subversion or CVS data to Bazaar data, but about the scaffolding that schedules, runs and monitors the jobs.

For the purposes of this talk, we may as well think of an import as a python script that:

Should be run regularly for each import branch in LP.

Can be triggered to run now from the webapp.

Should be run with command line arguments that come, ultimately, from the database.

Will produce logging output on stdout or stderr regularly: failure to produce output can be taken as failure to make progress.

Can run for any length of time from a few seconds to a couple of weeks.

Can be expected to occasionally use very large amounts of RAM.

Can be expected to fail intermittently, in a variety of ways.

Any particular run of the script is not that precious – if it failed because of some random network blip, we can assume with some degree of plausibility that the next run will pick up all the relevant changes from the remote repository.

We had some goals and things we wanted to avoid in mind:

For the system to be robust and to run itself.

The system should require little in the way of ongoing maintenance.

There should be no race conditions that result in the system getting tied up or confused about how many jobs should are running.

If a job does consume all memory in a slave and cause thrashing, this should be recovered from without manual intervention.

One failing import should not adversely affect the rest of the system.

The system should be flexible with respect to the priority given different jobs.

It should be possible to kill a job without logging in to the code import slave running the jobs.

Our solution has met all of these goals, so far as we can tell at this point (some parts are not fully implemented, however).

The Solution

For simplicity, some of the aspects of the solution described here are described as how we would like them to be, not how they are currently implemented (as of October 2008).

This paper does not talk about the CodeImportEvent audit trail, as it is not fundamental to the operation of the system and does not seem as relevant to the other use cases for our architecture.

The Data Model

The core object is a row in the Branch table (with branch_type == IMPORTED). The location to perform the import from is specified in a row of the CodeImport table.

Each code import slave – machine that an instance of the code import script can be run on – is represented by a row in the CodeImportMachine table.

A particular future or current run of the import script is run is represented by a row in the CodeImportJob table. The job table has references to the code import to run, who requested the job, if anyone, when the job is due and if the job is running, which machine it is running on and a heartbeat field that is expected to be updated frequently to indicate that the job is still making progress.

When a run of an import completes – successfully or failing – the result is stored in the CodeImportResult table, the row in the job table is deleted and a new one created with a due date an appropriate time in the future.

The Run-Time Model

There is a single data store, the good old launchpad_prod database. All parts of the code import system interact with this via the private XML-RPC, partly to get automatic handling of serialization errors.

There are potentially many code import machines and each machine runs potentially many jobs at once.

On each machine, a code-import-dispatcher cronscript runs frequently and checks with the private XML-RPC server to see if a job is should be started on the machine it is running on. If so, it starts running the code-import-monitor script, with the details of the job – which has now been marked as started in the database – as a command line argument. Then the dispatcher exits; it only ever dispatches one job at once. The monitor script in turn launches the script that performs the input – called the worker – and monitors it: it saves any output the worker makes to a log file, calls an XML-RPC method every 30 seconds to update the heartbeat field in the database and if the worker doesn't make any output for an hour the monitor kills it with a signal.

When the worker completes, the monitor uploads the log to the librarian, informs the XML-RPC server that the job has finished and exits.

More Subtle Issues

As trailed in the abstract, the system I've just described is quite complicated. Here I try to explain that this is not all totally unnecessary...

Killing A Job From The Webapp

The way to kill a job from the webapp, or indeed anywhere that's not the machine its running on (where a signal does the job) is to delete the row representing the job from the database.

The monitor script is written so that if this has happened when it tries to update the heartbeat, it quietly exits.

This is the main reason why a new row is created in the CodeImportJob table for the next run of the import when a job exits rather than updating the due date of the finished job.

Recovering From Thrashing

As mentioned previously, we can occasionally expect the import jobs to consume huge amounts of RAM (importing glibc, for example).

If several of these jobs end up running on the same machine at the same time, this will inevitably result in thrashing and a tremendous slow down in running all of the imports on that machine. This is in part what the heartbeat is for: if the machine is thrashing hard, the monitor processes won't be able to update the heartbeats in a timely manner, allowing the situation to be determined by looking at the database.

In the XML-RPC server, every time a dispatcher asks for a new method, a query runs to see if any job has not updated its heartbeat for too long. For any jobs in this state, the row in the database corresponding to the job will be deleted. Eventually, the monitor processes will notice this and kill the workers, which releases the memory and enables the system to run normally again.

This is the theory anyway: in practice, the main reason for the heartbeat not being updated in a timely manner is that the monitor process failed to start up for some reason, and the LOSAs keep far too good an eye on the machines for much thrashing to occur.

Avoiding Race Conditions

The code import system contains much concurrency – for example, what if the code import dispatcher determines that a job's heartbeat hasn't be up for too long at the same time as the monitor process finally tries to update the heartbeat?

The main strategy is the usual protection afforded by the serializable transaction isolation level, mediated through the internal XML-RPC server, which basically means that one of the simultaneous events “wins” and it appears that the other happened afterwards.

There are some other details: the “get a job to run” XML-RPC method marks the job as started so that the very next call doesn't get the same job to start, and deleting the job row when a job finishes means that where a slow running import cannot fail to notice that it has been killed.

Shutting Down A Machine

Simply pulling the plug on or restarting a code import slave will obviously kill all the imports running on the machine at the time.

This situation can be avoided by putting the machine into a “quiescing” state where no new jobs are assigned to this machine.

In practice, I don't think we bother with this.

Monitoring

Having all the data in the database means that we can plot graphs of the number of running imports and the number of due-but-not-yet-running imports, which helps us spot problems in the system, sometimes before they become serious (for example, we are starting to seriously need to get a new code import slave).

Responsiveness

A problem with the system is that it is not very event-driven: when an import becomes due or is requested, you have to wait until the dispatcher cronjob runs before the job is started, and when the import is completed you have to wait until the import mirror-puller script runs before you can get at the freshly imported revisions and then wait until the scanner runs before you can see the new revisions in the web UI.

I'm not really sure what to do about this; having all the state in the database brings definite advantages. It might be possible to use database triggers to send simple “hey, now might be a time to look for a job” messages to the slave machines.

Other Problems, Similar Solutions?

Given that we've built a good deal of infrastructure and it works in practice, it is a natural thought to make the architecture more generic and try to use it to solve other, similar, problems.

The Branch Puller

The branch puller is responsible for getting the branch data from a remote or staging area into the area from which it served to read-only users.

Currently, it runs as a cronjob which processes all branches that needed to be pulled when the puller starts. This means that a single long-running mirror job can prevent all other branches from being mirrored – which is unfortunate.

For hosted branches, we request a mirror when the branch is unlocked by a client. We mirror 'mirrored' (i.e. internet-hosted) branches every 6 hours. The code import system as stands doesn't support a job that only needs to be run when requested, but this is not a big deal. In other respects, it's a very similar problem.

Generating Diffs For Merge Proposals (And Other Things)

The number 1 requested feature for Launchpad's new code review system is to show a diff of the changes the proposed branch would introduce. We can't compute this as part of the merge proposal creation, because at least in some cases producing the diff is a fairly slow process and also because the app servers do not have access to the branch data.

We'd like to implement a system where the diffs are generated by jobs that would be created by the webapp when a branch is proposed for merging and run on a machine that has access to the branches as soon as possible.

In addition the branch scanner, which is responsible for importing branch metadata into the Launchpad database, currently can be dramatically slowed down if a user subscribes to a large branch (for example, the gcc code import) and asks for diffs. We'd like to change the scanner to use the same diff-generating job architecture as code reviews.

Other Branch Tasks

There are some other tasks that we'd like to trigger from the webapp that need access to or even modify branches:

Upgrading the branch format.

Running reconcile on the copy of the branch in the mirrored area

Running 'bzr check' on the branch.

Deleting the branch data from the file system when the branch is deleted from the database.

None of these tasks have the periodic nature of code imports, which changes some of the considerations slightly (in particular, it means we lose the “no single run of a job is that important” property of code imports), but many of the details are the same.

Soyuz?

Soyuz currently works rather differently: as I understand it, a cronjob builds a queue of jobs in priority order and tells the free builders which builds to perform next. But it could use a similar slave-asks scheme and share the infrastructure and monitoring code we use for the above tasks could, hopefully, be re-used.

Anything Else?

Use cases we don't yet know about are hard to include in this paper, for obvious reasons. We'd love to hear about them though!