Enriching Test Equipment Analytics with Structured Logging
Automated test equipment plays a large role in the manufacturing process at the Kansas City National Security Campus (KCNSC). Every test run generates data which is used to determine if a part meets its requirements or to troubleshoot failures if they occur. Among this data are log events which contain unique information about each test run. Ordinarily, these events are rendered to a string and routed to a file, database, or status window. Strings are used because they are easy to create and store, are human-readable, and are capable of encoding many data types.
When troubleshooting failures, these logs are a critical resource. Log files can be browsed manually to find key information; however, it can quickly become tedious to correlate a specific log event across multiple runs. At the KCNSC, tester software now utilizes an approach known as structured logging. A structured log event message is still defined using a string but one which is annotated to indicate the parts that were derived from data. This enhances each event, allowing it to be treated as a collection of properties rather than a simple string.
Using this approach to define log events sacrifices no flexibility but highly promotes data accessibility. With access to this data, tester teams can quickly find answers to their questions about tester or part performance. For example, one tester exhibited an issue with crosstalk between digitizer channels. Using log aggregation software, the structured logs were queried and used to produce plots indicating on which channels the issue most frequently appeared. This minimized the effort required for inspecting signal routing through the tester. Analysis like this is critical to reducing downtime during troubleshooting and also allows for proactive monitoring in a way which is not feasible with conventional unstructured logs.