The development of cognitive robotics brings an attractive scenario where humans and robots cooperate to accomplish specific tasks. To facilitate this scenario, cognitive robots are expected to have the ability to interact with humans with natural language, which depends on natural language understanding (NLU) technologies. As one core task in NLU, sentence semantic matching (SSM) has widely existed in various interaction scenarios. Recently, deep learning–based methods for SSM have become predominant due to their outstanding performance. However, each sentence consists of a sequence of words, and it is usually viewed as one-dimensional (1D) text, leading to the existing available neural models being restricted into 1D sequential networks. A few researches attempt to explore the potential of 2D or 3D neural models in text representation. However, it is hard for their works to capture the complex features in texts, and thus the achieved performance improvement is quite limited. To tackle this challenge, we devise a novel 3D CNN-based SSM (3DSSM) method for human–robot language interaction. Specifically, first, a specific architecture called feature cube network is designed to transform a 1D sentence into a multi-dimensional representation named as semantic feature cube. Then, a 3D CNN module is employed to learn a semantic representation for the semantic feature cube by capturing both the local features embedded in word representations and the sequential information among successive words in a sentence. Given a pair of sentences, their representations are concatenated together to feed into another 3D CNN to capture the interactive features between them to generate the final matching representation. Finally, the semantic matching degree is judged with the sigmoid function by taking the learned matching representation as the input. Extensive experiments on two real-world datasets demonstrate that 3DSSM is able to achieve comparable or even better performance over the state-of-the-art competing methods.

Sentence Semantic Matching Based on 3D CNN for Human–Robot Language Interaction

The rapid development of Internet of Things (IoT) brings an urgent requirement on intelligent human–device interactions using natural language, which are critical for facilitating people to use IoT devices. The efficient interactive approaches depend on various natural language understanding technologies. Among them, sentence pair modeling (SPM) is essential, where neural networks have achieved great success in SPM area due to their powerful abilities in feature extraction and representation. However, as sentences are one-dimensional (1D) texts, the available neural networks are usually limited to 1D sequential models, which prevents the performance improvement of SPM task. To address this gap, in this paper, we propose a novel neural architecture for sentence pair modeling, which utilizes 1D sentences to construct multi-dimensional feature maps similar to images containing multiple color channels. Based on the feature maps, more kinds of neural models become applicable on SPM task, including 2D CNN. In the proposed model, first, the sentence on a specific granularity is encoded with BiLSTM to generate the representation on this granularity, which is viewed as a special channel of the sentence. The representations from different granularity are merged together to construct semantic feature map of the input sentence. Then, 2D CNN is employed to encode the feature map to capture the deeper semantic features contained in the sentence. Next, another 2D CNN is utilized to capture the interactive matching features between sentences, followed by 2D max-pooling and attention mechanism to generate the final matching representation. Finally, the matching degree of sentences are judged with a sigmoid function according to the matching representation. Extensive experiments are conducted on two real-world data sets. In comparison with benchmarks, the proposed model achieved remarkable results, and performed better or comparably with BERT-based models. Our work is beneficial to building a more powerful humanized interaction system with IoT devices.

Sentence pair modeling based on semantic feature map for human interaction with IoT devices

Membrane fouling diagnosis of membrane components based on multi-feature information fusion

Textbooks play a vital role in any educational system, despite their clarity and information, students tend to use community question answers (cQA) forums to acquire more knowledge. Due to the high data volume, the quality of Question-Answers (QA) of cQA forums can differ greatly, so it takes additional effort to go through all possible QA pairs for a better insight. This paper proposes an “sentence-level text enrichment system” where the fine-tuned BERT (Bidirectional Encoder Representations from Transformers) summarizer understands the given text, picks out the important sentence, and then rearranged them to give the overall summary of the text document. For each important sentence, we recommend the relevant QA pairs from cQA to make the learning more effective. In this work, we fine-tuned the pre-trained BERT model to extract the relevant QA sets that are most relevant for enriching important sentences of the textbook. We notice that fine-tuning the BERT model significantly improves the performance for QA selection and find that it outperforms existing RNN-based models for such tasks. We also investigate the effectiveness of our fine-tuned BERT\(_\mathrm{Large}\) model on three cQA datasets for the QA selection task and observed a maximum improvement of 19.72% compared to the previous models. Experiments have been carried out on NCERT (Grade IX and X) Textbooks from India and “Pattern Recognition and Machine Learning” Textbook. The extensive evaluation methods demonstrate that the proposed model offers more precise and relevant recommendations in comparison to the state-of-the-art methods.

A Finetuned Language Model for Recommending cQA-QAs for Enriching Textbooks

Answer selection (AS), as one of the hottest topics in the field of natural language processing, has developed rapidly with outstanding performances reported, especially with the emergency of pretrained model (e.g., BERT). However, the current BERT based AS methods applied BERT only by fine-tuning or stacking other modules such as CNN and RNN, but ignored to exploit the discrimination embedded inside the BERT. In this paper, we proposed a novel method LMPF-IE, i.e., Lightweight Multiple Perspective Fusion with Information Enriching. The method can mine and fuse the multi-layer discrimination inside different layers of BERT and can use Question Category and Name Entity Recognition to enrich the information which can help BERT better understand the relationship between questions and answers. We test the proposed BERT layer-wised attention model in 5 benchmark datasets of answer selection task. The experimental results clearly verify better performances than the baseline models can be achieved by our method.

Lightweight Multiple Perspective Fusion with Information Enriching for BERT-Based Answer Selection.

This paper develops a novel soft fault diagnosis approach for analog circuits. The proposed method employs the backward difference strategy to process the data, and a novel variant of convolutional neural network, i.e., convolutional neural network with global average pooling (CNN-GAP) is taken for feature extraction and fault classification. Specifically, the measured raw domain response signals are firstly processed by the backward difference strategy and the first-order and the second-order backward difference sequences are generated, which contain the signal variation and the rate of variation characteristics. Then, based on the one-dimensional convolutional neural network, the CNN-GAP is developed by introducing the global average pooling technical. Since global average pooling calculates each input vector’s mean value, the designed CNN-GAP could deal with different lengths of input signals and be applied to diagnose different circuits. Additionally, the first-order and the second-order backward difference sequences along with the raw domain response signals are directly fed into the CNN-GAP, in which the convolutional layers automatically extract and fuse multi-scale features. Finally, fault classification is performed by the fully connected layer of the CNN-GAP. The effectiveness of our proposal is verified by two benchmark circuits under symmetric and asymmetric fault conditions. Experimental results prove that the proposed method outperforms the existing methods in terms of diagnosis accuracy and reliability.

A Novel Analog Circuit Soft Fault Diagnosis Method Based on Convolutional Neural Network and Backward Difference

In open-domain question answering system, the granularities of the answers vary with different types of questions. For example, for the questions asking about locations (Location type questions), their answers are usually short phrases. While for the questions asking about reasons (Description type questions), their answers are usually long clauses or sentences. This insight can be used to improve the performance of answer sentence selection, which is a crucial component of the open-domain QA system. In this paper, we propose a novel Multi-Granularity Neural Network (MGNN) model to better evaluate the semantic matching of questions and answers. First, MGNN has three classes of channels with each computing the similarity of question and answer pairs from one of the three granularity levels: clause level, phrase level and ngram level. Then, MGNN uses a parametrization weighting scheme which considers question types to combine these different granularity channels. We carry out experiments on a public available benchmark dataset for question answering. Empirical results show that our method outperforms state-of-the-art methods.

A Multi-granularity Neural Network for Answer Sentence Selection

Knowledge base question answering (KBQA) can be divided into two types according to the type of complexity: questions with constraints and questions with multiple hops of relationships. Previous work on knowledge base question answering have mostly focused on entities and relations. In a multihop question, it is insufficient to focus solely on topic entities and their relations since the relation between words also contains some important information. In addition, because the question contains constraints or multiple relationships, the information is difficult to capture, or the constraints are missed. In this paper, we applied a dependency structure to questions that capture relation information (e.g., constraint) between the words in question through a graph convolution network. The captured relation information is integrated into the question for re-encoding, and the information is used to generate and rank query graphs. Compared with existing sequence models and query graph generation models, our approach achieves a 0.8–3% improvement on two benchmark datasets.

/pdf/graph-convolution-network-over-dependency-structure-improve-1uwud7h1.pdf

Graph Convolution Network over Dependency Structure Improve Knowledge Base Question Answering

The face super-resolution (FSR) task is to reconstruct high-resolution face images from low-resolution inputs. Re-cent works have achieved success on this task by utilizing facial priors such as facial landmarks. Most existing methods pay more attention to global shape and structure information, but less to local texture information, which makes them cannot recover local details well. In this pa-per, we propose a novel recurrent convolutional network based framework for face super-resolution, which progres-sively introduces both global shape and local texture infor-mation. We take full advantage of the intermediate outputs of the recurrent network, and landmarks information and facial action units (AUs) information are extracted in the output of the first and second steps respectively, rather than low-resolution input. Moreover, we introduced AU clas-sification results as a novel quantitative metric for facial details restoration. Extensive experiments show that our proposed method significantly outperforms state-of-the-art FSR methods in terms of image quality and facial details restoration.

Chenggong Zhang

Papers

A Novel Analog Circuit Soft Fault Diagnosis Method Based on Convolutional Neural Network and Backward Difference

A Multi-granularity Neural Network for Answer Sentence Selection

Graph Convolution Network over Dependency Structure Improve Knowledge Base Question Answering

Face Super-Resolution with Progressive Embedding of Multi-scale Face Priors