b.i.GraphIndex(object) : class documentation

Part of bzrlib.index View In Hierarchy

An index for data with embedded graphs.

The index maps keys to a list of key reference lists, and a value. Each node has the same number of key reference lists. Each key reference list can be empty or an arbitrary length. The value is an opaque NULL terminated string without any newlines. The storage of the index is hidden in the interface: keys and key references are always tuples of bytestrings, never the internal representation (e.g. dictionary offsets).

It is presumed that the index will not be mutated - it is static data.

Successive iter_all_entries calls will read the entire index each time. Additionally, iter_entries calls will read the index linearly until the desired keys are found. XXX: This must be fixed before the index is suitable for production use. :XXX

Method __init__ Open an index called name on transport.
Method __eq__ Equal when self and other were created with the same parameters.
Method __ne__ Undocumented
Method __repr__ Undocumented
Method clear_cache Clear out any cached/memoized values.
Method external_references Return references that are not present in this index.
Method iter_all_entries Iterate over all keys within the index.
Method iter_entries Iterate over keys within the index.
Method iter_entries_prefix Iterate over keys within the index using prefix matching.
Method key_count Return an estimate of the number of keys in this index.
Method validate Validate that everything in the index can be accessed.
Method _buffer_all Buffer all the index data.
Method _get_nodes_by_key Undocumented
Method _read_prefix Undocumented
Method _resolve_references Return the resolved key references for references.
Method _find_index Helper for the _parsed_*_index calls.
Method _parsed_byte_index Return the index of the entry immediately before offset.
Method _parsed_key_index Return the index of the entry immediately before key.
Method _is_parsed Returns True if offset has been parsed.
Method _iter_entries_from_total_buffer Iterate over keys when the entire index is parsed.
Method _find_ancestors See BTreeIndex._find_ancestors.
Method _lookup_keys_via_location Public interface for implementing bisection.
Method _parse_header_from_bytes Parse the header from a region of bytes.
Method _parse_region Parse node data returned from a readv operation.
Method _parse_segment Parse one segment of data.
Method _parse_lines Undocumented
Method _parsed_bytes Mark the bytes from start to end as parsed.
Method _read_and_parse Read the ranges and parse the resulting data.
Method _signature The file signature for this index type.
def __init__(self, transport, name, size, unlimited_cache=False, offset=0):
Open an index called name on transport.
ParameterstransportA bzrlib.transport.Transport.
nameA path to provide to transport API calls.
sizeThe size of the index in bytes. This is used for bisection logic to perform partial index reads. While the size could be obtained by statting the file this introduced an additional round trip as well as requiring stat'able transports, both of which are avoided by having it supplied. If size is None, then bisection support will be disabled and accessing the index will just stream all the data.
offsetInstead of starting the index data at offset 0, start it at an arbitrary offset.
def __eq__(self, other):
Equal when self and other were created with the same parameters.
def __ne__(self, other):
Undocumented
def __repr__(self):
Undocumented
def _buffer_all(self, stream=None):
Buffer all the index data.

Mutates self._nodes and self.keys_by_offset.

def clear_cache(self):
Clear out any cached/memoized values.

This can be called at any time, but generally it is used when we have extracted some information, but don't expect to be requesting any more from this index.

def external_references(self, ref_list_num):
Return references that are not present in this index.
def _get_nodes_by_key(self):
Undocumented
def iter_all_entries(self):
Iterate over all keys within the index.
ReturnsAn iterable of (index, key, value) or (index, key, value, reference_lists). The former tuple is used when there are no reference lists in the index, making the API compatible with simple key:value index types. There is no defined order for the result iteration - it will be in the most efficient order for the index.
def _read_prefix(self, stream):
Undocumented
def _resolve_references(self, references):
Return the resolved key references for references.

References are resolved by looking up the location of the key in the _keys_by_offset map and substituting the key name, preserving ordering.

ParametersreferencesAn iterable of iterables of key locations. e.g. [[123, 456], [123]]
ReturnsA tuple of tuples of keys.
def _find_index(self, range_map, key):
Helper for the _parsed_*_index calls.

Given a range map - [(start, end), ...], finds the index of the range in the map for key if it is in the map, and if it is not there, the immediately preceeding range in the map.

def _parsed_byte_index(self, offset):
Return the index of the entry immediately before offset.

e.g. if the parsed map has regions 0,10 and 11,12 parsed, meaning that there is one unparsed byte (the 11th, addressed as[10]). then: asking for 0 will return 0 asking for 10 will return 0 asking for 11 will return 1 asking for 12 will return 1

def _parsed_key_index(self, key):
Return the index of the entry immediately before key.

e.g. if the parsed map has regions (None, 'a') and ('b','c') parsed, meaning that keys from None to 'a' inclusive, and 'b' to 'c' inclusive have been parsed, then: asking for '' will return 0 asking for 'a' will return 0 asking for 'b' will return 1 asking for 'e' will return 1

def _is_parsed(self, offset):
Returns True if offset has been parsed.
def _iter_entries_from_total_buffer(self, keys):
Iterate over keys when the entire index is parsed.
def iter_entries(self, keys):
Iterate over keys within the index.
ParameterskeysAn iterable providing the keys to be retrieved.
ReturnsAn iterable as per iter_all_entries, but restricted to the keys supplied. No additional keys will be returned, and every key supplied that is in the index will be returned.
def iter_entries_prefix(self, keys):
Iterate over keys within the index using prefix matching.

Prefix matching is applied within the tuple of a key, not to within the bytestring of each key element. e.g. if you have the keys ('foo', 'bar'), ('foobar', 'gam') and do a prefix search for ('foo', None) then only the former key is returned.

WARNING: Note that this method currently causes a full index parse unconditionally (which is reasonably appropriate as it is a means for thunking many small indices into one larger one and still supplies iter_all_entries at the thunk layer).

ParameterskeysAn iterable providing the key prefixes to be retrieved. Each key prefix takes the form of a tuple the length of a key, but with the last N elements 'None' rather than a regular bytestring. The first element cannot be 'None'.
ReturnsAn iterable as per iter_all_entries, but restricted to the keys with a matching prefix to those supplied. No additional keys will be returned, and every match that is in the index will be returned.
def _find_ancestors(self, keys, ref_list_num, parent_map, missing_keys):
See BTreeIndex._find_ancestors.
def key_count(self):
Return an estimate of the number of keys in this index.

For GraphIndex the estimate is exact.

def _lookup_keys_via_location(self, location_keys):
Public interface for implementing bisection.

If _buffer_all has been called, then all the data for the index is in memory, and this method should not be called, as it uses a separate cache because it cannot pre-resolve all indices, which buffer_all does for performance.

Parameterslocation_keysA list of location(byte offset), key tuples.
ReturnsA list of (location_key, result) tuples as expected by bzrlib.bisect_multi.bisect_multi_bytes.
def _parse_header_from_bytes(self, bytes):
Parse the header from a region of bytes.
ParametersbytesThe data to parse.
ReturnsAn offset, data tuple such as readv yields, for the unparsed data. (which may length 0).
def _parse_region(self, offset, data):
Parse node data returned from a readv operation.
ParametersoffsetThe byte offset the data starts at.
dataThe data to parse.
def _parse_segment(self, offset, data, end, index):
Parse one segment of data.
ParametersoffsetWhere 'data' begins in the file.
dataSome data to parse a segment of.
endWhere data ends
indexThe current index into the parsed bytes map.
Returnshigh_parsed_byte, last_segment. high_parsed_byte is the location of the highest parsed byte in this segment, last_segment is True if the parsed segment is the last possible one in the data block.
def _parse_lines(self, lines, pos):
Undocumented
def _parsed_bytes(self, start, start_key, end, end_key):
Mark the bytes from start to end as parsed.

Calling self._parsed_bytes(1,2) will mark one byte (the one at offset 1) as parsed.

ParametersstartThe start of the parsed region.
endThe end of the parsed region.
def _read_and_parse(self, readv_ranges):
Read the ranges and parse the resulting data.
Parametersreadv_rangesA prepared readv range list.
def _signature(self):
The file signature for this index type.
def validate(self):
Validate that everything in the index can be accessed.
API Documentation for Bazaar, generated by pydoctor at 2019-08-17 00:32:45.