Vision-and-language reasoning requires an understanding of visual concepts, language semantics, and, most importantly, the alignment and relationships between these two modalities. We thus propose the LXMERT (Learning Cross-Modality Encoder Representations from Transformers) framework to learn these vision-and-language connections. In LXMERT, we build a large-scale Transformer model that consists of three encoders: an object relationship encoder, a language encoder, and a cross-modality encoder. Next, to endow our model with the capability of connecting vision and language semantics, we pre-train the model with large amounts of image-and-sentence pairs, via five diverse representative pre-training tasks: masked language modeling, masked object prediction (feature regression and label classification), cross-modality matching, and image question answering. These tasks help in learning both intra-modality and cross-modality relationships. After fine-tuning from our pre-trained parameters, our model achieves the state-of-the-art results on two visual question answering datasets (i.e., VQA and GQA). We also show the generalizability of our pre-trained cross-modality model by adapting it to a challenging visual-reasoning task, NLVR2, and improve the previous best result by 22% absolute (54% to 76%). Lastly, we demonstrate detailed ablation studies to prove that both our novel model components and pre-training strategies significantly contribute to our strong results. Code and pre-trained models publicly available at: https://github.com/airsplay/lxmert

/pdf/lxmert-learning-cross-modality-encoder-representations-from-23xkklrxtq.pdf

LXMERT: Learning Cross-Modality Encoder Representations from Transformers

Joint image-text embedding is the bedrock for most Vision-and-Language (V+L) tasks, where multimodality inputs are simultaneously processed for joint visual and textual understanding. In this paper, we introduce UNITER, a UNiversal Image-TExt Representation, learned through large-scale pre-training over four image-text datasets (COCO, Visual Genome, Conceptual Captions, and SBU Captions), which can power heterogeneous downstream V+L tasks with joint multimodal embeddings. We design four pre-training tasks: Masked Language Modeling (MLM), Masked Region Modeling (MRM, with three variants), Image-Text Matching (ITM), and Word-Region Alignment (WRA). Different from previous work that applies joint random masking to both modalities, we use conditional masking on pre-training tasks (i.e., masked language/region modeling is conditioned on full observation of image/text). In addition to ITM for global image-text alignment, we also propose WRA via the use of Optimal Transport (OT) to explicitly encourage fine-grained alignment between words and image regions during pre-training. Comprehensive analysis shows that both conditional masking and OT-based WRA contribute to better pre-training. We also conduct a thorough ablation study to find an optimal combination of pre-training tasks. Extensive experiments show that UNITER achieves new state of the art across six V+L tasks (over nine datasets), including Visual Question Answering, Image-Text Retrieval, Referring Expression Comprehension, Visual Commonsense Reasoning, Visual Entailment, and NLVR\(^2\) (Code is available at https://github.com/ChenRocks/UNITER.).

UNITER: UNiversal Image-TExt Representation Learning

We propose VisualBERT, a simple and flexible framework for modeling a broad range of vision-and-language tasks. VisualBERT consists of a stack of Transformer layers that implicitly align elements of an input text and regions in an associated input image with self-attention. We further propose two visually-grounded language model objectives for pre-training VisualBERT on image caption data. Experiments on four vision-and-language tasks including VQA, VCR, NLVR2, and Flickr30K show that VisualBERT outperforms or rivals with state-of-the-art models while being significantly simpler. Further analysis demonstrates that VisualBERT can ground elements of language to image regions without any explicit supervision and is even sensitive to syntactic relationships, tracking, for example, associations between verbs and image regions corresponding to their arguments.

VisualBERT: A Simple and Performant Baseline for Vision and Language.

Large-scale pre-training methods of learning cross-modal representations on image-text pairs are becoming popular for vision-language tasks. While existing methods simply concatenate image region features and text features as input to the model to be pre-trained and use self-attention to learn image-text semantic alignments in a brute force manner, in this paper, we propose a new learning method Oscar (Object-Semantics Aligned Pre-training), which uses object tags detected in images as anchor points to significantly ease the learning of alignments. Our method is motivated by the observation that the salient objects in an image can be accurately detected, and are often mentioned in the paired text. We pre-train an Oscar model on the public corpus of 6.5 million text-image pairs, and fine-tune it on downstream tasks, creating new state-of-the-arts on six well-established vision-language understanding and generation tasks.

Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks

This paper presents a detailed study of improving visual representations for vision language (VL) tasks and develops an improved object detection model to provide object-centric representations of images. Compared to the most widely used bottom-up and top-down model [2], the new model is bigger, better-designed for VL tasks, and pre-trained on much larger training corpora that combine multiple public annotated object detection datasets. Therefore, it can generate representations of a richer collection of visual objects and concepts. While previous VL research focuses mainly on improving the vision-language fusion model and leaves the object detection model improvement untouched, we show that visual features matter significantly in VL models. In our experiments we feed the visual features generated by the new object detection model into a Transformer-based VL fusion model OSCAR [20], and utilize an improved approach OSCAR+ to pre-train the VL model and fine-tune it on a wide range of downstream VL tasks. Our results show that the new visual features significantly improve the performance across all VL tasks, creating new state-of-the-art results on seven public benchmarks. Code, models and pre-extracted features are released at https://github.com/pzzhang/VinVL.

/pdf/vinvl-revisiting-visual-representations-in-vision-language-118kt9l8uf.pdf

VinVL: Revisiting Visual Representations in Vision-Language Models

We introduce a new dataset for joint reasoning about natural language and images, with a focus on semantic diversity, compositionality, and visual reasoning challenges. The data contains 107,292 examples of English sentences paired with web photographs. The task is to determine whether a natural language caption is true about a pair of photographs. We crowdsource the data using sets of visually rich images and a compare-and-contrast task to elicit linguistically diverse language. Qualitative analysis shows the data requires compositional joint reasoning, including about quantities, comparisons, and relations. Evaluation using state-of-the-art visual reasoning methods shows the data presents a strong challenge.

/pdf/a-corpus-for-reasoning-about-natural-language-grounded-in-4u0mg9x2g0.pdf

A Corpus for Reasoning about Natural Language Grounded in Photographs

A Corpus for Reasoning About Natural Language Grounded in Photographs.

Diesel engines have a wide range of functions in the industrial and military fields. An urgent problem to be solved is how to diagnose and identify their faults effectively and timely. In this paper, a diesel engine acoustic fault diagnosis method based on variational modal decomposition mapping Mel frequency cepstral coefficients (MFCC) and long-short-term memory network is proposed. Variational mode decomposition (VMD) is used to remove noise from the original signal and differentiate the signal into multiple modes. The sound pressure signals of different modes are mapped to the Mel filter bank in the frequency domain, and then the Mel frequency cepstral coefficients of the respective mode signals are calculated in the mapping range of frequency domain, and the optimized Mel frequency cepstral coefficients are used as the input of long and short time memory network (LSTM) which is trained and verified, and the fault diagnosis model of the diesel engine is obtained. The experimental part compares the fault diagnosis effects of different feature extraction methods, different modal decomposition methods and different classifiers, finally verifying the feasibility and effectiveness of the method proposed in this paper, and providing solutions to the problem of how to realise fault diagnosis using acoustic signals.

/pdf/combination-of-vmd-mapping-mfcc-and-lstm-a-new-acoustic-3sh20dta.pdf

Combination of VMD Mapping MFCC and LSTM: A New Acoustic Fault Diagnosis Method of Diesel Engine

Extracting features manually and employing preeminent knowledge is overly utilized in methods to conduct fault diagnosis. A diagnosis approach utilizing intelligent methods of the optimized variational mode decomposition and deep transfer learning is proposed in this manuscript to deal with fault diagnosis. Firstly, the variational mode decomposition is optimized by K values of the dispersion entropy to realize an adaptive decomposition and reduce the noise of the signal. Secondly, an image with two dimensions is generated by a vibration signal with one dimension utilizing a short-time Fourier transform, after conducting noise reduction. Then, the ResNet18 network model is used to pre-train the model. Finally, the model transfer method is used to detect faults of a diesel engine. The results show that the proposed method outperforms the deep learning methods available in the literature. Besides, better noise reduction ability and higher diagnostic accuracy are attained.

/pdf/combination-of-optimized-variational-mode-decomposition-and-1zshi9bv.pdf

Combination of Optimized Variational Mode Decomposition and Deep Transfer Learning: A Better Fault Diagnosis Approach for Diesel Engines

The safe operation of diesel engines performs a vital function in industrial production and life. Because diesel engines often work in harsh environmental conditions, they are prone to failure. Therefore, this paper proposes a fault analysis method based on a combination of optimized variational mode decomposition (VMD) and improved convolutional neural networks (CNN) to address the necessary need for preventive maintenance of diesel engines. The authentic vibration sign is first decomposed by using the (VMD) algorithm, then the greatest range of decomposition layers is decided by using scattering entropy and the useful components are preferentially chosen for reconstruction. The continuous wavelet transform (CWT) records preprocessing method is then delivered to radically change the noise-reduced vibration sign into a time-frequency map, which is fed into the CNN for model coaching and extraction of fault features. Finally, fault classification is realized by support vector machine (SVM) with excellent classification performance. Through preset fault experiments on diesel engines, it is established that the technique proposed in this paper can successfully identify fault states, and the classification accuracy is higher than alternative methods.

/pdf/diesel-engine-fault-diagnosis-method-based-on-optimized-vmd-gepbd4oj.pdf

Huajun Bai

Papers

A Corpus for Reasoning about Natural Language Grounded in Photographs

A Corpus for Reasoning About Natural Language Grounded in Photographs.

Combination of VMD Mapping MFCC and LSTM: A New Acoustic Fault Diagnosis Method of Diesel Engine

Combination of Optimized Variational Mode Decomposition and Deep Transfer Learning: A Better Fault Diagnosis Approach for Diesel Engines

Diesel Engine Fault Diagnosis Method Based on Optimized VMD and Improved CNN