Disclosed herein is a method and apparatus for hardware-accelerating various data quality checking operations. Incoming data streams can be processed with respect to a plurality of data quality check operations using offload engines (e.g., reconfigurable logic such as field programmable gate arrays (FPGAs)). Accelerated data quality checking can be highly advantageous for use in connection with Extract, Transfer, and Load (ETL) systems.

Method and apparatus for accelerated data quality checking

Various methods and apparatuses are described for performing high speed format translations of incoming data, where the incoming data is arranged in a delimited data format. As an example, the data in the delimited data format can be translated to a fixed field format using pipelined operations. As another example, the data in the delimited data format can be translated to a mapped variable field format using pipelined operations. A reconfigurable logic device can be used in exemplary embodiments as a platform for the format translation.

Method and apparatus for accelerated format translation of data in a delimited data format

Various methods and apparatuses are described for performing high speed translations of data. In an example embodiment, record layout detection can be performed for data. In another example embodiment, data pivoting prior to field-specific data processing can be performed.

Method and apparatus for record pivoting to accelerate processing of data fields

Improved computer technology is disclosed for enabling high performance stream processing on data such as complex, hierarchical data. In an example embodiment, a dynamic field schema specifies a dynamic field format for expressing the incoming data. An incoming data stream is then translated according to the dynamic field schema into an outgoing data stream in the dynamic field format. Stream processing, including field-specific stream processing, can then be performed on the outgoing data stream.

Dynamic Field Data Translation to Support High Performance Stream Data Processing

A method of matching flight data from multiple sources, including receiving a parametric flight data set for a given flight where the parametric flight data set includes a number of data features, receiving operational flight data sets related to a number of flights where the operational flight data sets include a number of data features, and determining a matching operational flight data set for the parametric flight data set.

Jinan Lou

Papers

Generating data records based on parsing