Part of lp.services.librarianserver
|Function||file_exists||True if the file exists either on disk or in Swift.|
|Function||open_stream||Return an open file for the given content_id.|
|Function||confirm_no_clock_skew||Raise an exception if there is significant clock skew between the|
|Function||delete_expired_blobs||Remove expired TemporaryBlobStorage entries and their corresponding|
|Function||merge_duplicates||Merge duplicate LibraryFileContent rows|
|Class||ExpireAliases||Expire expired LibraryFileAlias records.|
|Class||UnreferencedLibraryFileAliasPruner||Delete unreferenced LibraryFileAliases.|
|Function||delete_unreferenced_aliases||Run the UnreferencedLibraryFileAliasPruner.|
|Class||UnreferencedContentPruner||Delete LibraryFileContent entries and their disk files that are|
|Function||delete_unwanted_disk_files||Delete files found on disk that have no corresponding record in the|
|Function||swift_files||Generate the (container, name) of all files stored in Swift.|
|Function||delete_unwanted_swift_files||Delete files found in Swift that have no corresponding db record.|
|Function||get_file_path||Return the physical file path to the matching LibraryFileContent id.|
|Function||get_storage_root||Return the path to the root of the Librarian storage area.|
Swift is only checked if the librarian.swift.enabled feature flag is set.
Returns None if the file cannot be found.
It is theoretically possible to lose data if there is more than several hours of skew.
We delete the LibraryFileAliases here as the default behaviour of the garbage collector could leave them hanging around indefinitely.
We also delete any linked ApportJob and Job records here.
This is the first step in a full garbage collection run. We assume files are identical if their sha1 hashes and filesizes are identical. For every duplicate detected, we make all LibraryFileAlias entries point to one of them and delete the unnecessary duplicates from the filesystem and the database.
Files will only be deleted if they were created more than one day ago to avoid deleting files that have just been uploaded but have yet to have the database records committed.
Results are yielded in numerical order.