“Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告

Pre-trained Transformer-based models have achieved state-of-the-art performance for various Natural Language Processing (NLP) tasks. However, these models often have billions of parameters, and, thus, are too resource-hungry and computation-intensive to suit low-capability devices or applications with strict latency requirements. One potential remedy for this is model compression, which has attracted a lot of research attention. Here, we summarize the research in compressing Transformers, focusing on the especially popular BERT model. In particular, we survey the state of the art in compression for BERT, we clarify the current best practices for compressing large-scale Transformer models, and we provide insights into the workings of various methods. Our categorization and analysis also shed light on promising future research directions for achieving lightweight, accurate, and generic NLP models.

Compressing Large-Scale Transformer-Based Models: A Case Study on BERT

The number of scientific publications in the literature is steadily growing, containing our knowledge in the biomedical, health, and clinical sciences. Since there is currently no automatic archiving of the obtained results, much of this information remains buried in textual details not readily available for further usage or analysis. For this reason, natural language processing (NLP) and text mining methods are used for information extraction from such publications. In this paper, we review practices for Named Entity Recognition (NER) and Relation Detection (RD), allowing, e.g., to identify interactions between proteins and drugs or genes and diseases. This information can be integrated into networks to summarize large-scale details on a particular biomedical or clinical problem, which is then amenable for easy data management and further analysis. Furthermore, we survey novel deep learning methods that have recently been introduced for such tasks.

/pdf/named-entity-recognition-and-relation-detection-for-2d1hp93ahp.pdf

Named Entity Recognition and Relation Detection for Biomedical Information Extraction

Knowledge graphs (KGs) have been widely used in the field of artificial intelligence, such as in information retrieval, natural language processing, recommendation systems, etc. However, the open nature of KGs often implies that they are incomplete, having self-defects. This creates the need to build a more complete knowledge graph for enhancing the practical utilization of KGs. Link prediction is a fundamental task in knowledge graph completion that utilizes existing relations to infer new relations so as to build a more complete knowledge graph. Numerous methods have been proposed to perform the link-prediction task based on various representation techniques. Among them, KG-embedding models have significantly advanced the state of the art in the past few years. In this paper, we provide a comprehensive survey on KG-embedding models for link prediction in knowledge graphs. We first provide a theoretical analysis and comparison of existing methods proposed to date for generating KG embedding. Then, we investigate several representative models that are classified into five categories. Finally, we conducted experiments on two benchmark datasets to report comprehensive findings and provide some new insights into the strengths and weaknesses of existing models.

A Survey on Knowledge Graph Embeddings for Link Prediction

Pre-trained Transformer-based models have achieved state-of-the-art performance for various Natural Language Processing (NLP) tasks. However, these models often have billions of parameters, and thus are too resource-hungry and computation-intensive to suit low-capability devices or applications with strict latency requirements. One potential remedy for this is model compression, which has attracted a lot of research attention. Here, we summarize the research in compressing Transformers, focusing on the especially popular BERT model. In particular, we survey the state of the art in compression for BERT, we clarify the current best practices for compressing large-scale Transformer models, and we provide insights into the workings of various methods. Our categorization and analysis also shed light on promising future research directions for achieving lightweight, accurate, and generic NLP models.

/pdf/compressing-large-scale-transformer-based-models-a-case-2mgb7x45uj.pdf

The representational basis for downstream natural language processing tasks is word embeddings, which capture lexical semantics in numerical form to handle the abstract semantic concept of words. Recently, the word embeddings approaches, represented by deep learning, has attracted extensive attention and widely used in many tasks, such as text classification, knowledge mining, question-answering, smart Internet of Things systems and so on. These neural networks-based models are based on the distributed hypothesis while the semantic association between words can be efficiently calculated in low-dimensional space. However, the expressed semantics of most models are constrained by the context distribution of each word in the corpus while the logic and common knowledge are not better utilized. Therefore, how to use the massive multi-source data to better represent natural language and world knowledge still need to be explored. In this paper, we introduce the recent advances of neural networks-based word embeddings with their technical features, summarizing the key challenges and existing solutions, and further give a future outlook on the research and application.

A survey of word embeddings based on deep learning

Cluster adaptation networks for unsupervised domain adaptation

Domain adaptation algorithm is a powerful tool for transferring the knowledge of source domain with sufficient annotations for target tasks. Recently, adversarial learning is embedded into deep networks to reduce domain shift between source and target domains for learning transferable features. Existing adversarial domain adaptation methods aim at reducing the source and target domain discrepancy ignoring the class discrepancy between source and target domains. This paper proposes a novel Multiple Adversarial Networks (MAN) for unsupervised domain adaptation. MAN utilizes a pair of classifiers to minimize inter-domain discrepancy and embeds a domain discriminator for each category for intra-class discrepancy. Furthermore, we extend our MAN as improved MAN (iMAN) by utilizing a feature norm term to regularize the task-specific features, which can improve model generalization and help for minimizing intra-class discrepancy. We conduct extensive experiments on two real-world datasets Office–Home and ImageCLEF-DA, and experiment results show the effectiveness and superiority of our methods compared with several state-of-the-art unsupervised domain adaptation methods.

Multiple adversarial networks for unsupervised domain adaptation

Clustering methods based on deep neural networks have been extensively studied and applied in data mining. Most existing unsupervised deep embedded clustering methods jointly learn feature representations and cluster assignments with deep neural networks. However, to the best of our knowledge, most of them ignore the relevance between representation learning and clustering, so that fail to couple feature representation learning and cluster assignments effectively, which leads to non-representative features for clustering and this in turn hurts clustering performance. In this paper, we propose Adversarial Deep Embedded Clustering (ADEC), a novel unsupervised clustering method based on adversarial auto-encoder (AAE) and k -means clustering method. Specifically, ADEC matches distribution of feature representations with the given prior distribution to preserve the data structure with AAE, and k -means based on distribution distance metrics is conducted for clustering with these feature representations, simultaneously. ADEC optimizes a clustering objective iteratively with backpropagation algorithm in learning AAE from data space to feature space. Extensive experiments have demonstrated the effectiveness of our proposed method on images clustering compared to several strong baseline methods.

Deep Embedded Clustering With Adversarial Distribution Adaptation

Domain adaptation is a powerful tool for transferring the knowledge of the source domain with sufficient annotations for target tasks. However, most existing domain adaptation methods focus on the single-source–single-target adaptation setting, which ignore that existing different but related multiple source domains are available for target tasks. In this paper, we propose a novel theoretical generalization bound for multiple source domain adaptation, which averages the multiple source hypothesis risks by using H Δ H -distance with an assumption that the target hypothesis is the combination of all the source hypotheses. Following this theoretical result, we propose a new algorithm called Duplex Adversarial Networks for Multi-source Adaptation (DAN_MA): one adversarial network is utilized for feature generator to learn domain invariant representations between source and target domains with a domain classifier for each pair of source and target; the other is utilized to eliminate the domain characteristics among different sources by making target samples close to each source task-specific decision boundaries. The shared feature generator aims to learn target representations which are invariant to the multiple domain shifts while still close to the source decision boundaries. Extensive experiments show that our method outperforms other domain adaptation methods on three public datasets.

Wen’an Zhou

Papers

A survey of word embeddings based on deep learning

Cluster adaptation networks for unsupervised domain adaptation

Multiple adversarial networks for unsupervised domain adaptation

Deep Embedded Clustering With Adversarial Distribution Adaptation

Duplex adversarial networks for multiple-source domain adaptation