Human knowledge provides a formal understanding of the world. Knowledge graphs that represent structural relations between entities have become an increasingly popular research direction toward cognition and human-level intelligence. In this survey, we provide a comprehensive review of the knowledge graph covering overall research topics about: 1) knowledge graph representation learning; 2) knowledge acquisition and completion; 3) temporal knowledge graph; and 4) knowledge-aware applications and summarize recent breakthroughs and perspective directions to facilitate future research. We propose a full-view categorization and new taxonomies on these topics. Knowledge graph embedding is organized from four aspects of representation space, scoring function, encoding models, and auxiliary information. For knowledge acquisition, especially knowledge graph completion, embedding methods, path inference, and logical rule reasoning are reviewed. We further explore several emerging topics, including metarelational learning, commonsense reasoning, and temporal knowledge graphs. To facilitate future research on knowledge graphs, we also provide a curated collection of data sets and open-source libraries on different tasks. In the end, we have a thorough outlook on several promising research directions.

A Survey on Knowledge Graphs: Representation, Acquisition and Applications

With the emerging branch of incorporating factual knowledge into pre-trained language models such as BERT, most existing models consider shallow, static, and separately pre-trained entity embeddings, which limits the performance gains of these models. Few works explore the potential of deep contextualized knowledge representation when injecting knowledge. In this paper, we propose the Contextualized Language and Knowledge Embedding (CoLAKE), which jointly learns contextualized representation for both language and knowledge with the extended MLM objective. Instead of injecting only entity embeddings, CoLAKE extracts the knowledge context of an entity from large-scale knowledge bases. To handle the heterogeneity of knowledge context and language context, we integrate them in a unified data structure, word-knowledge graph (WK graph). CoLAKE is pre-trained on large-scale WK graphs with the modified Transformer encoder. We conduct experiments on knowledge-driven tasks, knowledge probing tasks, and language understanding tasks. Experimental results show that CoLAKE outperforms previous counterparts on most of the tasks. Besides, CoLAKE achieves surprisingly high performance on our synthetic task called word-knowledge graph completion, which shows the superiority of simultaneously contextualizing language and knowledge representation.

/pdf/colake-contextualized-language-and-knowledge-embedding-42moehcsqf.pdf

CoLAKE: Contextualized Language and Knowledge Embedding

This paper presents a comprehensive survey of ChatGPT and GPT-4, state-of-the-art large language models (LLM) from the GPT series, and their prospective applications across diverse domains. Indeed, key innovations such as large-scale pre-training that captures knowledge across the entire world wide web, instruction fine-tuning and Reinforcement Learning from Human Feedback (RLHF) have played significant roles in enhancing LLMs' adaptability and performance. We performed an in-depth analysis of 194 relevant papers on arXiv, encompassing trend analysis, word cloud representation, and distribution analysis across various application domains. The findings reveal a significant and increasing interest in ChatGPT/GPT-4 research, predominantly centered on direct natural language processing applications, while also demonstrating considerable potential in areas ranging from education and history to mathematics, medicine, and physics. This study endeavors to furnish insights into ChatGPT's capabilities, potential implications, ethical concerns, and offer direction for future advancements in this field.

Summary of ChatGPT/GPT-4 Research and Perspective Towards the Future of Large Language Models

This paper examines the challenging problem of learning representations of entities and relations in a complex multi-relational knowledge graph. We propose HittER, a Hierarchical Transformer model to jointly learn Entity-relation composition and Relational contextualization based on a source entity's neighborhood. Our proposed model consists of two different Transformer blocks: the bottom block extracts features of each entity-relation pair in the local neighborhood of the source entity and the top block aggregates the relational information from the outputs of the bottom block. We further design a masked entity prediction task to balance information from the relational context and the source entity itself. Evaluated on the task of link prediction, our approach achieves new state-of-the-art results on two standard benchmark datasets FB15K-237 and WN18RR.

HittER: Hierarchical Transformers for Knowledge Graph Embeddings

Hyper-relational knowledge graphs (KGs) (e.g., Wikidata) enable associating additional key-value pairs along with the main triple to disambiguate, or restrict the validity of a fact. In this work, we propose a message passing based graph encoder - StarE capable of modeling such hyper-relational KGs. Unlike existing approaches, StarE can encode an arbitrary number of additional information (qualifiers) along with the main triple while keeping the semantic roles of qualifiers and triples intact. We also demonstrate that existing benchmarks for evaluating link prediction (LP) performance on hyper-relational KGs suffer from fundamental flaws and thus develop a new Wikidata-based dataset - WD50K. Our experiments demonstrate that StarE based LP model outperforms existing approaches across multiple benchmarks. We also confirm that leveraging qualifiers is vital for link prediction with gains up to 25 MRR points compared to triple-based representations.

Message Passing for Hyper-Relational Knowledge Graphs

Knowledge graph embedding, which projects symbolic entities and relations into continuous vector spaces, is gaining increasing attention. Previous methods allow a single static embedding for each entity or relation, ignoring their intrinsic contextual nature, i.e., entities and relations may appear in different graph contexts, and accordingly, exhibit different properties. This work presents Contextualized Knowledge Graph Embedding (CoKE), a novel paradigm that takes into account such contextual nature, and learns dynamic, flexible, and fully contextualized entity and relation embeddings. Two types of graph contexts are studied: edges and paths, both formulated as sequences of entities and relations. CoKE takes a sequence as input and uses a Transformer encoder to obtain contextualized representations. These representations are hence naturally adaptive to the input, capturing contextual meanings of entities and relations therein. Evaluation on a wide variety of public benchmarks verifies the superiority of CoKE in link prediction and path query answering. It performs consistently better than, or at least equally well as current state-of-the-art in almost every case, in particular offering an absolute improvement of 21.0% in H@10 on path query answering. Our code is available at \url{this https URL}.

CoKE: Contextualized Knowledge Graph Embedding.

Information extraction is an important foundation for knowledge graph construction, as well as many natural language understanding applications. Similar to many other artificial intelligence tasks, high quality annotated datasets are essential to train a high-performance information extraction system. Existing datasets, however, are mostly built for English. To promote research in Chinese information extraction and evaluate the performance of related systems, we build a large-scale high-quality dataset, named DuIE, and make it publicly available. We design an efficient coarse-to-fine procedure including candidate generation and crowdsourcing annotation, in order to achieve high data quality at a large data scale. DuIE contains 210,000 sentences and 450,000 instances covering 49 types of commonly used relations, reflecting the real-world scenario. We also hosted an open competition based on DuIE, which attracted 1,896 participants. The competition results demonstrated the potential of this dataset in promoting information extraction research.

DuIE: A Large-Scale Chinese Dataset for Information Extraction

The invention provides a mixed annotation learning neural network model, and a training method and device of the model. The model comprises a first sub neural network model and a second sub neural network model, wherein the first sub neural network model is used for encoding and decoding input data to be processed to generate initial result representation corresponding to the data to be processed,and the second sub neural network model is used for correcting the initial result representation corresponding to the data to be processed according to a vector corresponding to an acquired processing standard to generate target result representation corresponding to the data to be processed. According to the mixed annotation learning neural network model, training corpuses with different annotation standards are integrally used, the adaptability of the neural network model to the difference between the training corpuses with different annotation standards is improved, universality of the neural network model is enhanced, and processing performance is improved.

Mixed annotation learning neural network model, and training method and device of model

The invention discloses a knowledge graph completion method and a device, electronic equipment and a storage medium, and relates to the technical field of big data. According to the specific implementation scheme, the method comprises the steps of obtaining view instances of a triple at different view angles; inputting the view instances of different view angles into a target discrimination classification model, and obtaining a comprehensive classification result of the triad at all view angles; wherein the comprehensive classification result is used for representing the effectiveness of the triad. Therefore, the effectiveness index of one triad can be measured from a plurality of different perspectives so as to more comprehensively measure the effectiveness of the triad.

Knowledge graph completion method and device, electronic equipment and storage medium

The invention provides a method for obtaining a reading understanding material, and the method comprises the steps: obtaining a subject-predicate-object triad which comprises a corresponding subject,a predicate and an object; according to a preset question template, constructing at least one question including a subject and a predicate of the subject-predicate-object triad, and taking an object of the subject-predicate-object triad as an answer corresponding to the question; performing searching to obtain at least one text comprising a subject and an object of the subject-predicate-object triad; and constructing a reading understanding material according to the answer, the question and the text. The invention further provides a device for obtaining the reading understanding material, electronic equipment and a computer readable medium.

Jiang Wenbin

Papers

CoKE: Contextualized Knowledge Graph Embedding.

DuIE: A Large-Scale Chinese Dataset for Information Extraction

Mixed annotation learning neural network model, and training method and device of model

Knowledge graph completion method and device, electronic equipment and storage medium

Method and device for obtaining reading understanding material, electronic equipment and readable medium