Neural language representation models such as BERT pre-trained on large-scale corpora can well capture rich semantic patterns from plain text, and be fine-tuned to consistently improve the performance of various NLP tasks. However, the existing pre-trained language models rarely consider incorporating knowledge graphs (KGs), which can provide rich structured knowledge facts for better language understanding. We argue that informative entities in KGs can enhance language representation with external knowledge. In this paper, we utilize both large-scale textual corpora and KGs to train an enhanced language representation model (ERNIE), which can take full advantage of lexical, syntactic, and knowledge information simultaneously. The experimental results have demonstrated that ERNIE achieves significant improvements on various knowledge-driven tasks, and meanwhile is comparable with the state-of-the-art model BERT on other common NLP tasks. The code and datasets will be available in the future.

ERNIE: Enhanced Language Representation with Informative Entities

Learning from data has led to paradigm shifts in a multitude of disciplines, including web, text and image search, speech recognition, as well as bioinformatics. Can machine learning enable similar breakthroughs in understanding quantum many-body systems? Here we develop an efficient deep learning approach that enables spatially and chemically resolved insights into quantum-mechanical observables of molecular systems. We unify concepts from many-body Hamiltonians with purpose-designed deep tensor neural networks, which leads to size-extensive and uniformly accurate (1 kcal mol−1) predictions in compositional and configurational chemical space for molecules of intermediate size. As an example of chemical relevance, the model reveals a classification of aromatic rings with respect to their stability. Further applications of our model for predicting atomic energies and local chemical potentials in molecules, reliable isomer energies, and molecules with peculiar electronic structure demonstrate the potential of machine learning for revealing insights into complex quantum-chemical systems.

/pdf/quantum-chemical-insights-from-deep-tensor-neural-networks-266u885pa4.pdf

Quantum-Chemical Insights from Deep Tensor Neural Networks

We address the problem of adversarial attacks on text classification, which is rarely studied comparing to attacks on image classification. The challenge of this task is to generate adversarial examples that maintain lexical correctness, grammatical correctness and semantic similarity. Based on the synonyms substitution strategy, we introduce a new word replacement order determined by both the word saliency and the classification probability, and propose a greedy algorithm called probability weighted word saliency (PWWS) for text adversarial attack. Experiments on three popular datasets using convolutional as well as LSTM models show that PWWS reduces the classification accuracy to the most extent, and keeps a very low word substitution rate. A human evaluation study shows that our generated adversarial examples maintain the semantic similarity well and are hard for humans to perceive. Performing adversarial training using our perturbed datasets improves the robustness of the models. At last, our method also exhibits a good transferability on the generated adversarial examples.

/pdf/generating-natural-language-adversarial-examples-through-1ftt7y125j.pdf

Generating Natural Language Adversarial Examples through Probability Weighted Word Saliency.

Deep learning--based models have surpassed classical machine learning--based approaches in various text classification tasks, including sentiment analysis, news categorization, question answering, and natural language inference. In this article, we provide a comprehensive review of more than 150 deep learning--based models for text classification developed in recent years, and we discuss their technical contributions, similarities, and strengths. We also provide a summary of more than 40 popular datasets widely used for text classification. Finally, we provide a quantitative analysis of the performance of different deep learning models on popular benchmarks, and we discuss future research directions.

Deep Learning--based Text Classification: A Comprehensive Review

The combination of modern scientific computing with electronic structure theory can lead to an unprecedented amount of data amenable to intelligent data analysis for the identification of meaningful, novel, and predictive structure-property relationships. Such relationships enable high-throughput screening for relevant properties in an exponentially growing pool of virtual compounds that are synthetically accessible. Here, we present a machine learning (ML) model, trained on a data base of \textit{ab initio} calculation results for thousands of organic molecules, that simultaneously predicts multiple electronic ground- and excited-state properties. The properties include atomization energy, polarizability, frontier orbital eigenvalues, ionization potential, electron affinity, and excitation energies. The ML model is based on a deep multi-task artificial neural network, exploiting underlying correlations between various molecular properties. The input is identical to \emph{ab initio} methods, \emph{i.e.} nuclear charges and Cartesian coordinates of all atoms. For small organic molecules the accuracy of such a "Quantum Machine" is similar, and sometimes superior, to modern quantum-chemical methods---at negligible computational cost.

Machine Learning of Molecular Electronic Properties in Chemical Compound Space

In the drive toward sub-10-nm semiconductor devices, manufacturers have been developing advanced lithography technologies such as extreme ultraviolet lithography and multiple patterning. However, these technologies can cause unexpected defects, and a high-speed inspection is thus required to cover the entire surface of a wafer. A Die-to-Database (D2DB) inspection is commonly known as a high-speed inspection. The D2DB inspection compares an inspection image with a design layout, so it does not require a reference image for comparing with the inspection image, unlike a die-to-die inspection, thereby achieving a high-speed inspection. However, conventional D2DB inspections suffer from erroneous detection because the manufacturing processes deform the circuit pattern from the design layout, and such deformations will be detected as defects. To resolve this issue, we propose a deep-learning-based D2DB inspection that can distinguish a defect deformation from a normal deformation by learning the luminosity distribution in normal images. Our inspection detects outliers of the learned luminosity distribution as defects. Because our inspection requires only normal images, we can train the model without defect images, which are difficult to obtain with enough variety. In this way, our inspection can detect unseen defects. Through experiments, we show that our inspection can detect only the defect region on an inspection image.

A trainable die-to-database for fast e-Beam inspection: learning normal images to detect defects

We propose a simple and accurate model for coordination boundary identification. Our model decomposes the task into three sub-tasks during training; finding a coordinator, identifying inside boundaries of a pair of conjuncts, and selecting outside boundaries of it. For inference, we make use of probabilities of coordinators and conjuncts in the CKY parsing to find the optimal combination of coordinate structures. Experimental results demonstrate that our model achieves state-of-the-art results, ensuring that the global structure of coordinations is consistent.

Decomposed Local Models for Coordinate Structure Parsing.

Biomedical Named Entity Recognition (BioNER) is a crucial step for analyzing Biomedical texts, which aims at extracting biomedical named entities from a given text. Different supervised machine learning algorithms have been applied for BioNER by various researchers. The main requirement of these approaches is an annotated dataset used for learning the parameters of machine learning algorithms. Segment Representation (SR) models comprise of different tag sets used for representing the annotated data, such as IOB2, IOE2 and IOBES. In this paper, we propose an extension of IOBES model to improve the performance of BioNER. The proposed SR model, FROBES, improves the representation of multi-word entities. We used Bidirectional Long Short-Term Memory (BiLSTM) network; an instance of Recurrent Neural Networks (RNN), to design a baseline system for BioNER and evaluated the new SR model on two datasets, i2b2/VA 2010 challenge dataset and JNLPBA 2004 shared task dataset. The proposed SR model outperforms other models for multi-word entities with length greater than two. Further, the outputs of different SR models have been combined using majority voting ensemble method which outperforms the baseline models performance.

Improving Multi-Word Entity Recognition for Biomedical Texts.

The new measuring method that we developed executes a contour shape analysis that is based on the pattern edge information from a SEM image. This analysis helps to create a highly precise quantification of every circuit pattern shape by comparing the contour extracted from the SEM image using a CD measurement algorithm and the ideal circuit pattern. The developed method, in the next phase, can generate four shape indices by using the analysis mass measurement data. When the shape index measured using the developed method is compared the CD, the difference of the shape index and the CD is negligibly small for the quantification of the circuit pattern shape. In addition, when the 2D patterns on a FEM wafer are measured using the developed method, the tendency for shape deformations is precisely caught by the four shape indices. This new method and the evaluation results will be presented in detail in this paper.

SEM-contour shape analysis method for advanced semiconductor devices

In this paper, we describe TextEnt, a neural network model that learns distributed representations of entities and documents directly from a knowledge base (KB). Given a document in a KB consisting of words and entity annotations, we train our model to predict the entity that the document describes and map the document and its target entity close to each other in a continuous vector space. Our model is trained using a large number of documents extracted from Wikipedia. The performance of the proposed model is evaluated using two tasks, namely fine-grained entity typing and multiclass text classification. The results demonstrate that our model achieves state-of-the-art performance on both tasks. The code and the trained representations are made available online for further academic research.

Hiroyuki Shindo

Papers

A trainable die-to-database for fast e-Beam inspection: learning normal images to detect defects

Decomposed Local Models for Coordinate Structure Parsing.

Improving Multi-Word Entity Recognition for Biomedical Texts.

SEM-contour shape analysis method for advanced semiconductor devices

Representation Learning of Entities and Documents from Knowledge Base Descriptions