Discussed herein are embodiments of methods and systems which allow engineers or administrators to create modular plugins which represent the logic for various fault detection tests that can be performed on data pipelines and shared among different software deployments. In some cases, the modular plugins each define a particular test to be executed against data received from the pipeline in addition to one or more configuration points. The configuration points represent configurable arguments, such as variables and/or functions, referenced by the instructions which implement the tests and that can be set according to the specific operation environment of the monitored pipeline.

Pluggable fault detection tests for data pipelines

Computer implemented systems and methods are disclosed for importing data from electronic data files. In accordance with some embodiments, source electronic data files are received at a data importation system and managed by the data importation system. The data importation system may apply detector/transformer plugins to the received source electronic data files to transform the files for importation into one or more data analysis systems and/or databases. The data importation system may also receive user inputs for mapping source electronic data files to transformation templates. The inputs may include, for example, an assignment of a file format to the source electronic data file, identification of a file type identifier associated with the source electronic data file, and a mapping of a the source electronic data file to a transformation template. The data importation system may store the received inputs as a file type profile in a database.

Systems and methods for importing data from electronic data files

The invention discloses a linked data oriented entity classification method and system which are aimed at the problem of entity classification of linked data. The entity classification method includes pretreatment, statistical classification and post treatment. Pretreatment: word segmentation is performed on text description information in an entity page; and an attribute name of an information box and word information obtained by segmentation form an entity page character. Statistical classification: a statistical classification model is trained through various segmentation granularities to classify the entity page, and then a primary prediction result of entity class can be obtained. Post treatment: the entity statistical classification result is corrected; and a combined entity class is corrected through model combination, language knowledge, linkage information and class associate attribute information. The method and the system is easy to implement and debug, is high in efficiency, is good in accuracy, can be used for performing knowledge management on the linked data, and achieve high-precision classification of the entity.

Linked data oriented entity classification method and system

Techniques for easy and secure deployment of a software package from a server to a customer-controlled computing device are described. In an embodiment, a deployment engine running on a server can be used to generate a unique URL for deployment of the software package. The unique URL may include a restricted use token. The restricted use token may be generated based on a combination of a random selection of one or more dictionary words, numbers, and/or symbols. The restricted token is easily readable given the combination of dictionary words, numbers, and/or symbols that make up the restricted use token. The unique URL may then be entered into a customer-controlled computing device via a curl command. The curl command will use the unique URL to generate a secure channel to the deployment engine and automatically download the software package onto the customer-controlled computing device. Upon downloading and/or installing the software package on the customer-controlled computing device, the unique URL is invalidated so that it may not be reused.

Secure deployment of a software package

A method and system for data pipeline monitoring receives an event data object and a current status data object from one or more subsystems of a pipeline. The system analyzes the event data object and the current status data object to determine a first and second validation value. The system, in response to determining that either the first or second validation value is not valid, sends a notification.

Data pipeline monitoring

Techniques are disclosure for a data enrichment system that enables declarative external data source importation and exportation. A user can specify via a user interface input for identifying different data sources from which to obtain input data. The data enrichment system is configured to import and export various types of sources storing resources such as URL-based resources and HDFS-based resources for high-speed bi-directional metadata and data interchange. Connection metadata (e.g., credentials, access paths, etc.) can be managed by the data enrichment system in a declarative format for managing and visualizing the connection metadata.

Douglas C. Savolainen

Papers

Declarative external data source importation, exportation, and metadata reflection utilizing http and hdfs protocols