l.h.s.h.SubmissionParser(object) : class documentation

Part of lp.hardwaredb.scripts.hwdbsubmissions View In Hierarchy

Known subclasses: lp.hardwaredb.scripts.tests.test_hwdb_submission_parser.SubmissionParserTestParseSoftware, lp.hardwaredb.scripts.tests.test_hwdb_submission_parser.SubmissionParserTestParseSoftwareNoPackagesNode, lp.hardwaredb.scripts.tests.test_hwdb_submission_parser.SubmissionParserTestParseSoftwareNoXorgNode, lp.hardwaredb.scripts.tests.test_hwdb_submission_parser.TestHWDBSubmissionParser.MockSubmissionParser, lp.hardwaredb.scripts.tests.test_hwdb_submission_parser.TestHWDBSubmissionParser.MockSubmissionParserMainParserTest, lp.hardwaredb.scripts.tests.test_hwdb_submission_parser.TestHWDBSubmissionParser.MockSubmissionParserParseHardwareTest, lp.hardwaredb.scripts.tests.test_hwdb_submission_parser.TestHWDBSubmissionParser.UdevTestSubmissionParser

A Parser for the submissions to the hardware database.
Method __init__ Undocumented
Method fixFrequentErrors Fixes for frequent formal errors in the submissions.
Method parseMainSections Undocumented
Method parseSubmission Parse the data of a HWDB submission.
Method findDuplicateIDs Return the set of duplicate IDs.
Method findInvalidIDReferences Return the set of invalid references to IDs.
Method getUDIDeviceMap Return a dictionary which maps UDIs to HAL devices.
Method getUDIChildren Build lists of all children of a UDI.
Method checkHALDevicesParentChildConsistency Ensure that HAL devices are represented in exactly one tree.
Method checkUdevDictsHavePathKey Ensure that each udev dictionary has a 'P' key.
Method checkUdevPciProperties Validation of udev PCI devices.
Method checkUdevUsbProperties Validation of udev USB devices.
Method checkUdevScsiProperties Validation of udev SCSI devices.
Method checkUdevDmiData Consistency check for DMI data.
Method checkConsistentUdevDeviceData Consistency checks for udev data.
Method checkConsistency Run consistency checks on the submitted data.
Method buildDeviceList Create a list of devices from a submission.
Method buildHalDeviceList Create a list of devices from the HAL data of a submission.
Method buildUdevDeviceList Create a list of devices from the udev data of a submission.
Method kernel_package_name The kernel package name for the submission.
Method processSubmission Process a submisson.
Method root_device The HALDevice of UdevDevice node of the root device.
Method _logError Log message for an error in submission submission_key`.
Method _logWarning Log message for a warning in submission submission_key`.
Method _getValidatedEtree Create an etree doc from the XML string submission and validate it.
Method _getValueAttributeAsBoolean Return the value of the attribute "value" as a boolean.
Method _getValueAttributeAsString Return the value of the attribute "value".
Method _getValueAttributeAsDateTime Convert a "value" attribute into a datetime object.
Method _getClientData Parse the <client> node in the <summary> section.
Method _parseSummary Parse the <summary> part of a submission.
Method _getValueAndType Return (value, type) of a <property> or <value> node.
Method _parseProperty Parse a <property> node.
Method _parseProperties Parse <property> sub-nodes of properties_node.
Method _parseDevice Parse a HAL <device> node.
Method _parseHAL Parse the <hal> section of a submission.
Method _parseProcessors Parse the <processors> node.
Method _parseAliases Parse the <aliases> node.
Method _parseUdev Parse the <udev> node.
Method _parseDmi Parse the <dmi> node.
Method _parseSysfsAttributes Parse the <sysfs-attributes> node.
Method _setHardwareSectionParsers Undocumented
Method _parseHardware Parse the <hardware> part of a submission.
Method _parseLSBRelease Parse the <lsb_release> part of a submission.
Method _parsePackages Parse the <packages> part of a submission.
Method _parseXOrg Parse the <xorg> part of a submission.
Method _setSoftwareSectionParsers Undocumented
Method _parseSoftware Parse the <software> section of a submission.
Method _parseQuestions Parse the <questions> part of a submission.
Method _parseContext Parse the <context> part of a submission.
Method _setMainSectionParsers Undocumented
Method _findDuplicates Search for duplicate elements in test_ids.
Method _getIDMap Return a dictionary ID -> devices, processors and packages.
Method _getIDUDIMaps Return two mappings describing the relation between IDs and UDIs.
Method _removeChildren Remove recursively all children of the device named udi.
def __init__(self, logger=None, record_warnings=True):
Undocumented
def _logError(self, message, submission_key, create_oops=True):
Log message for an error in submission submission_key`.
def _logWarning(self, message, warning_id=None):
Log message for a warning in submission submission_key`.
def fixFrequentErrors(self, submission):
Fixes for frequent formal errors in the submissions.
def _getValidatedEtree(self, submission, submission_key):
Create an etree doc from the XML string submission and validate it.
Returnsan lxml.etree instance representation of a valid submission or None for invalid submissions.
def _getValueAttributeAsBoolean(self, node):
Return the value of the attribute "value" as a boolean.
def _getValueAttributeAsString(self, node):
Return the value of the attribute "value".
def _getValueAttributeAsDateTime(self, time_node):
Convert a "value" attribute into a datetime object.
def _getClientData(self, client_node):
Parse the <client> node in the <summary> section.
ReturnsA dictionary with keys 'name', 'version', 'plugins'. Name and version describe the client program that produced the submission. Pugins is a list with one entry per client plugin; each entry is dictionary with the keys 'name' and 'version'.
def _parseSummary(self, summary_node):
Parse the <summary> part of a submission.
ReturnsA dictionary with the keys 'live_cd', 'system_id', 'distribution', 'distroseries', 'architecture', 'private', 'contactable', 'date_created', 'client'. See the sample XML file tests/hardwaretest.xml for detailed description of the values.
def _getValueAndType(self, node):
Return (value, type) of a <property> or <value> node.
def _parseProperty(self, property_node):
Parse a <property> node.
Returns(name, (value, type)) of a property.
def _parseProperties(self, properties_node):
Parse <property> sub-nodes of properties_node.
ReturnsA dictionary, where each key is the name of a property; the values are the tuples (value, type) of a property.
def _parseDevice(self, device_node):
Parse a HAL <device> node.
ReturnsA dictionary d with the keys 'id', 'udi', 'parent', 'properties'. d['id'] is an ID of the device d['udi'] is the HAL UDI of the device; d['properties'] is a dictionary with the properties of the device (see _parseProperties for details).
def _parseHAL(self, hal_node):
Parse the <hal> section of a submission.
ReturnsA list, where each entry is the result of a _parseDevice call.
def _parseProcessors(self, processors_node):
Parse the <processors> node.
ReturnsA list of dictionaries, where each dictionary d contains the data of a <processor> node. The dictionary keys are 'id', 'name', 'properties'. d['id'] is an ID of a <processor> node, d['name'] its name, and d['properties'] contains the properties of a processor (see _parseProperties for details).
def _parseAliases(self, aliases_node):
Parse the <aliases> node.
ReturnsA list of dictionaries, where each dictionary d has the keys 'id', 'vendor', 'model'. d['id'] is the ID of a HAL device; d['vendor'] is an alternative vendor name of the device; d['model'] is an alternative model name.

See tests/hardwaretest.xml more more details.

def _parseUdev(self, udev_node):
Parse the <udev> node.

The <udev> node contains the output produced by "udevadm info --export-db". Each entry of the dictionaries represents the data of the key:value pairs as they appear in this data. The value of d['S'] is a list of strings, the value s['E'] is a dictionary containing the key=value pairs of the "E:" lines.

ReturnsA list of dictionaries, where each dictionary describes a udev device.
def _parseDmi(self, dmi_node):
Parse the <dmi> node.
ReturnsA dictionary containing the key:value pairs of the DMI data.
def _parseSysfsAttributes(self, sysfs_node):
Parse the <sysfs-attributes> node.

A sample of the input data:

P: /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 A: modalias=input:b0019v0000p0001e0000-e0,1,k74,ramlsfw A: uniq= A: phys=LNXPWRBN/button/input0 A: name=Power Button

P: /devices/LNXSYSTM:00/device:00/PNP0A08:00/device:03/input/input8 A: modalias=input:b0019v0000p0006e0000-e0,1,kE0,E1,E3,F0,F1 A: uniq= A: phys=/video/input0 A: name=Video Bus

Data for different devices is separated by empty lines. The data for each device starts with a line 'P: /devices/LNXSYSTM...', specifying the sysfs path of a device, followed by zero or more lines of the form 'A: key=value'

ReturnsA dictionary {path: attrs, ...} where path is the path is the path of a sysfs directory, and where attrs is a dictionary containing attribute names and values.
def _setHardwareSectionParsers(self):
Undocumented
def _parseHardware(self, hardware_node):
Parse the <hardware> part of a submission.
ReturnsA dictionary with the keys 'hal', 'processors', 'aliases', where the values are the parsing results of _parseHAL, _parseProcessors, _parseAliases.
def _parseLSBRelease(self, lsb_node):
Parse the <lsb_release> part of a submission.
ReturnsA dictionary with the content of the <properta> nodes within the <lsb> node. See tests/hardwaretest.xml for details.
def _parsePackages(self, packages_node):
Parse the <packages> part of a submission.
ReturnsA dictionary with one entry per <package> sub-node. The key is the package name, the value a dictionary containing the content of the <property> nodes within <package>. See tests/hardwaretest.xml for more details.
def _parseXOrg(self, xorg_node):
Parse the <xorg> part of a submission.
ReturnsA dictionary with the keys 'version' and 'drivers'. d['version'] is the xorg version; d['drivers'] is a dictionary with one entry for each <driver> sub-node, where the key is the driver name, the value is a dictionary containing the attributes of the <driver> node. See tests/hardwaretest.xml for more details.
def _setSoftwareSectionParsers(self):
Undocumented
def _parseSoftware(self, software_node):
Parse the <software> section of a submission.
ReturnsA dictionary with the keys 'lsbrelease', 'packages', 'xorg', containing the parsing results of the respective sub-nodes. The key 'lsbrelease' exists always; 'xorg' and 'packages' are optional. See _parseLSBRelease, _parsePackages, _parseXOrg for more details.
def _parseQuestions(self, questions_node):
Parse the <questions> part of a submission.

:return: A list, where each entry is a dictionary containing
         the parsing result of the <question> sub-nodes.

         Content of a list entry d (see tests/hardwaretest.xml
         for a more detailed description):
         d['name']:
                The name of a question. (Always present)
         d['plugin']:
                The name of the client plugin which is
                "responsible" for the question. (Optional)
         d['targets']:
                A list, where each entry is a dicitionary
                describing a target device for this question.
                This list is always present, but may be empty.

                The contents of each list entry t is:

                t['id']:
                        The ID of a HAL <device> node of a
                        target device.
                t['drivers']:
                        A list of driver names, possibly empty.
         d['answer']:
                The answer to this question. The value is a
                dictionary a:
                a['value']:
                        The value of the answer. (Always present)

                        For questions of type muliple_choice,
                        the value should match one of the
                        entries of the answer_choices list,

                        For questions of type measurement, the
                        value is a numerical value.
                a['type']:
                        This is either 'multiple_choice' or
                        'measurement'. (Always present)
                a['unit']:
                        The unit of a measurement value.
                        (Optional)
         d['answer_choices']:
                A list of choices from which the user can select
                an answer. This list is always present, but should
                be empty for questions of type measurement.
         d['command']:
                The command line of a test script which was
                run for this question. (Optional)
         d['comment']:
                A comment the user has typed when running the
                client. (Optional)

         A consistency check of the content of d is done in
         method _checkSubmissionConsistency.
def _parseContext(self, context_node):
Parse the <context> part of a submission.

We don't do anything real right now, but simply log a warning that this submission contains a <context> section, so that we can parse it again later, once we have the SQL tables needed to store the data.

def _setMainSectionParsers(self):
Undocumented
def parseMainSections(self, submission_doc):
Undocumented
def parseSubmission(self, submission, submission_key):
Parse the data of a HWDB submission.
ReturnsA dictionary with the keys 'summary', 'hardware', 'software', 'questions'. See _parseSummary, _parseHardware, _parseSoftware, _parseQuestions for the content.
def _findDuplicates(self, all_ids, test_ids):
Search for duplicate elements in test_ids.

:return: A set of those elements in the sequence test_ids that
are elements of the set all_ids or that appear more than once
in test_ids.

all_ids is updated with test_ids.
def findDuplicateIDs(self, parsed_data):
Return the set of duplicate IDs.

The IDs of devices, processors and software packages should be unique; this method returns a list of duplicate IDs found in a submission.

def _getIDMap(self, parsed_data):
Return a dictionary ID -> devices, processors and packages.
def findInvalidIDReferences(self, parsed_data):
Return the set of invalid references to IDs.

The sub-tag <target> of <question> references a device, processor of package node by its ID; the submission must contain a <device>, <processor> or <software> tag with this ID. This method returns a set of those IDs mentioned in <target> nodes that have no corresponding device or processor node.

def getUDIDeviceMap(self, devices):
Return a dictionary which maps UDIs to HAL devices.

Also check, if a UDI is used more than once.

Generally, a duplicate UDI indicates a bad or bogus submission, but we have some UDIs where the duplicate UDI is caused by a bug in HAL, see http://lists.freedesktop.org/archives/hal/2009-April/013250.html In these cases, we simply remove the duplicates, otherwise, a ValueError is raised.

def _getIDUDIMaps(self, devices):
Return two mappings describing the relation between IDs and UDIs.
Returnstwo dictionaries id_to_udi and udi_to_id, where id_2_udi has IDs as keys and UDI as values, and where udi_to_id has UDIs as keys and IDs as values.
def getUDIChildren(self, udi_device_map):
Build lists of all children of a UDI.

If any info.parent property points to a non-existing existing device, a ValueError is raised.

ReturnsA dictionary that maps UDIs to lists of children.
def _removeChildren(self, udi, udi_test):
Remove recursively all children of the device named udi.
def checkHALDevicesParentChildConsistency(self, udi_children):
Ensure that HAL devices are represented in exactly one tree.

HAL devices "know" their parent device; each device has a parent, except the root element. This means that it is possible to traverse all existing devices, beginning at the root node.

Several inconsistencies are possible:

  1. we may have more than one root device (i.e., one without a parent)
  2. we may have no root element
  3. circular parent/child references may exist.

(a) and (b) are already checked in _getUDIChildren; this method implements (c),

ReturnsA list of those UDIs that are not "connected" to the root node /org/freedesktop/Hal/devices/computer
def checkUdevDictsHavePathKey(self, udev_nodes):
Ensure that each udev dictionary has a 'P' key.

The 'P' (path) key identifies a device.

def checkUdevPciProperties(self, udev_data):
Validation of udev PCI devices.

Each PCI device must have the properties PCI_CLASS, PCI_ID, PCI_SUBSYS_ID, PCI_SLOT_NAME. Non-PCI devices must not have them.

The value of PCI class must be a 24 bit integer in hexadecimal representation.

The values of PCI_ID and PCI_SUBSYS_ID must be two 16 bit integers, separated by a ':'.

Parametersudev_dataA list of dicitionaries describing udev devices, as returned by _parseUdev()
ReturnsTrue if all checks pass, else False.
def checkUdevUsbProperties(self, udev_data):
Validation of udev USB devices.

USB devices must either have the three properties DEVTYPE (value 'usb_device' or 'usb_interface'), PRODUCT and TYPE, or they must have none of them.

PRODUCT must be a tuple of three integers in hexadecimal representation, separates by '/'. TYPE must be a a tuple of three integers in decimal representation, separated by '/'. usb_interface nodes must additionally have a property INTERFACE, containing three integers in the same format as TYPE.

def checkUdevScsiProperties(self, udev_data, sysfs_data):
Validation of udev SCSI devices.

Each udev node where SUBSYSTEM is 'scsi' should have the property DEVTYPE; nodes where DEVTYPE is 'scsi_device' should have a corresponding sysfs node, and this node should define the attributes 'vendor', 'model', 'type'.

def checkUdevDmiData(self, dmi_data):
Consistency check for DMI data.

All keys of the dictionary dmi_data should start with '/sys/class/dmi/id/'.

def checkConsistentUdevDeviceData(self, udev_data, sysfs_data, dmi_data):
Consistency checks for udev data.
def checkConsistency(self, parsed_data):
Run consistency checks on the submitted data.
Parametersparsed_data: parsed submission data, as returned by parseSubmission
ReturnsTrue, if the data looks consistent, otherwise False.
def buildDeviceList(self, parsed_data):
Create a list of devices from a submission.
def buildHalDeviceList(self, parsed_data):
Create a list of devices from the HAL data of a submission.
def buildUdevDeviceList(self, parsed_data):
Create a list of devices from the udev data of a submission.
@cachedproperty
def kernel_package_name(self):
The kernel package name for the submission.
def processSubmission(self, submission):
Process a submisson.
ParameterssubmissionAn IHWSubmission instance.
ReturnsTrue, if the submission could be sucessfully processed, otherwise False.
@property
def root_device(self):
The HALDevice of UdevDevice node of the root device.
API Documentation for Launchpad, generated by pydoctor at 2020-07-11 00:00:15.