canonical.librarian.librariangc : API documentation

def confirm_no_clock_skew(con):

Raise an exception if there is significant clock skew between the database and this machine.

It is theoretically possible to lose data if there is more than several hours of skew.

def delete_expired_blobs(con):

Remove expired TemporaryBlobStorage entries and their corresponding LibraryFileAlias entries.

We delete the LibraryFileAliases here as the default behavior of the garbage collector could leave them hanging around indefinitely.

We also delete any linked ApportJob and Job records here.

def merge_duplicates(con):

Merge duplicate LibraryFileContent rows

This is the first step in a full garbage collection run. We assume files are identical if their sha1 hashes and filesizes are identical. For every duplicate detected, we make all LibraryFileAlias entries point to one of them and delete the unnecessary duplicates from the filesystem and the database.

def expire_aliases(con):

Invoke ExpireLibraryFileAliases.

def delete_unreferenced_aliases(con):

Run the UnreferencedLibraryFileAliasPruner.

def delete_unreferenced_content(con):

Invoke UnreferencedContentPruner.

def _walk(top, topdown=True, onerror=None, followlinks=False):

Directory tree generator.

For each directory in the directory tree rooted at top (including top
itself, but excluding '.' and '..'), yields a 3-tuple

    dirpath, dirnames, filenames

dirpath is a string, the path to the directory.  dirnames is a list of
the names of the subdirectories in dirpath (excluding '.' and '..').
filenames is a list of the names of the non-directory files in dirpath.
Note that the names in the lists are just names, with no path components.
To get a full path (which begins with top) to a file or directory in
dirpath, do os.path.join(dirpath, name).

If optional arg 'topdown' is true or not specified, the triple for a
directory is generated before the triples for any of its subdirectories
(directories are generated top down).  If topdown is false, the triple
for a directory is generated after the triples for all of its
subdirectories (directories are generated bottom up).

When topdown is true, the caller can modify the dirnames list in-place
(e.g., via del or slice assignment), and walk will only recurse into the
subdirectories whose names remain in dirnames; this can be used to prune
the search, or to impose a specific order of visiting.  Modifying
dirnames when topdown is false is ineffective, since the directories in
dirnames have already been generated by the time dirnames itself is
generated.

By default errors from the os.listdir() call are ignored.  If
optional arg 'onerror' is specified, it should be a function; it
will be called with one argument, an os.error instance.  It can
report the error to continue with the walk, or raise the exception
to abort the walk.  Note that the filename is available as the
filename attribute of the exception object.

By default, os.walk does not follow symbolic links to subdirectories on
systems that support them.  In order to get this functionality, set the
optional argument 'followlinks' to true.

Caution:  if you pass a relative pathname for top, don't change the
current working directory between resumptions of walk.  walk never
changes the current directory, and assumes that the client doesn't
either.

Example:

import os
from os.path import join, getsize
for root, dirs, files in os.walk('python/Lib/email'):
    print root, "consumes",
    print sum([getsize(join(root, name)) for name in files]),
    print "bytes in", len(files), "non-directory files"
    if 'CVS' in dirs:
        dirs.remove('CVS')  # don't visit CVS directories

def delete_unwanted_files(con):

Delete files found on disk that have no corresponding record in the database.

Files will only be deleted if they were created more than one day ago to avoid deleting files that have just been uploaded but have yet to have the database records committed.

def get_file_path(content_id):

Return the physical file path to the matching LibraryFileContent id.

def get_storage_root():

Return the path to the root of the Librarian storage area.

Performs some basic sanity checking to avoid accidents.

Function	confirm_no_clock_skew	Raise an exception if there is significant clock skew between the
Function	delete_expired_blobs	Remove expired TemporaryBlobStorage entries and their corresponding
Function	merge_duplicates	Merge duplicate LibraryFileContent rows
Class	ExpireAliases	Expire expired LibraryFileAlias records.
Function	expire_aliases	Invoke ExpireLibraryFileAliases.
Class	UnreferencedLibraryFileAliasPruner	Delete unreferenced LibraryFileAliases.
Function	delete_unreferenced_aliases	Run the UnreferencedLibraryFileAliasPruner.
Class	UnreferencedContentPruner	Delete LibraryFileContent entries and their disk files that are
Function	delete_unreferenced_content	Invoke UnreferencedContentPruner.
Function	delete_unwanted_files	Delete files found on disk that have no corresponding record in the
Function	get_file_path	Return the physical file path to the matching LibraryFileContent id.
Function	get_storage_root	Return the path to the root of the Librarian storage area.
Function	_walk	Directory tree generator.

c.l.librariangc : module documentation