l.s.l.librariangc : module documentation

Part of lp.services.librarianserver

Librarian garbage collection routines
Function file_exists True if the file exists either on disk or in Swift.
Function open_stream Return an open file for the given content_id.
Function sha1_file Undocumented
Function confirm_no_clock_skew Raise an exception if there is significant clock skew between the
Function delete_expired_blobs Remove expired TemporaryBlobStorage entries and their corresponding
Function merge_duplicates Merge duplicate LibraryFileContent rows
Class ExpireAliases Expire expired LibraryFileAlias records.
Function expire_aliases Invoke ExpireLibraryFileAliases.
Class UnreferencedLibraryFileAliasPruner Delete unreferenced LibraryFileAliases.
Function delete_unreferenced_aliases Run the UnreferencedLibraryFileAliasPruner.
Class UnreferencedContentPruner Delete LibraryFileContent entries and their disk files that are
Function delete_unreferenced_content Invoke UnreferencedContentPruner.
Function delete_unwanted_files Undocumented
Function delete_unwanted_disk_files Delete files found on disk that have no corresponding record in the
Function swift_files Generate the (container, name) of all files stored in Swift.
Function delete_unwanted_swift_files Delete files found in Swift that have no corresponding db record.
Function get_file_path Return the physical file path to the matching LibraryFileContent id.
Function get_storage_root Return the path to the root of the Librarian storage area.
Function _utcnow Undocumented
def file_exists(content_id):
True if the file exists either on disk or in Swift.

Swift is only checked if the librarian.swift.enabled feature flag is set.

def _utcnow():
Undocumented
def open_stream(content_id):
Return an open file for the given content_id.

Returns None if the file cannot be found.

def sha1_file(content_id):
Undocumented
def confirm_no_clock_skew(store):
Raise an exception if there is significant clock skew between the database and this machine.

It is theoretically possible to lose data if there is more than several hours of skew.

def delete_expired_blobs(con):
Remove expired TemporaryBlobStorage entries and their corresponding LibraryFileAlias entries.

We delete the LibraryFileAliases here as the default behaviour of the garbage collector could leave them hanging around indefinitely.

We also delete any linked ApportJob and Job records here.

def merge_duplicates(con):
Merge duplicate LibraryFileContent rows

This is the first step in a full garbage collection run. We assume files are identical if their sha1 hashes and filesizes are identical. For every duplicate detected, we make all LibraryFileAlias entries point to one of them and delete the unnecessary duplicates from the filesystem and the database.

def expire_aliases(con):
Invoke ExpireLibraryFileAliases.
def delete_unreferenced_aliases(con):
Run the UnreferencedLibraryFileAliasPruner.
def delete_unreferenced_content(con):
Invoke UnreferencedContentPruner.
def delete_unwanted_files(con):
Undocumented
def delete_unwanted_disk_files(con):
Delete files found on disk that have no corresponding record in the database.

Files will only be deleted if they were created more than one day ago to avoid deleting files that have just been uploaded but have yet to have the database records committed.

def swift_files(max_lfc_id):
Generate the (container, name) of all files stored in Swift.

Results are yielded in numerical order.

def delete_unwanted_swift_files(con):
Delete files found in Swift that have no corresponding db record.
def get_file_path(content_id):
Return the physical file path to the matching LibraryFileContent id.
def get_storage_root():
Return the path to the root of the Librarian storage area.

Performs some basic sanity checking to avoid accidents.

API Documentation for Launchpad, generated by pydoctor at 2022-06-16 00:00:12.