Deep learning is a form of machine learning that enables computers to learn from experience and understand the world in terms of a hierarchy of concepts. Because the computer gathers knowledge from experience, there is no need for a human computer operator to formally specify all the knowledge that the computer needs. The hierarchy of concepts allows the computer to learn complicated concepts by building them out of simpler ones; a graph of these hierarchies would be many layers deep. This book introduces a broad range of topics in deep learning. The text offers mathematical and conceptual background, covering relevant concepts in linear algebra, probability theory and information theory, numerical computation, and machine learning. It describes deep learning techniques used by practitioners in industry, including deep feedforward networks, regularization, optimization algorithms, convolutional networks, sequence modeling, and practical methodology; and it surveys such applications as natural language processing, speech recognition, computer vision, online recommendation systems, bioinformatics, and videogames. Finally, the book offers research perspectives, covering such theoretical topics as linear factor models, autoencoders, representation learning, structured probabilistic models, Monte Carlo methods, the partition function, approximate inference, and deep generative models. Deep Learning can be used by undergraduate or graduate students planning careers in either industry or research, and by software engineers who want to begin using deep learning in their products or platforms. A website offers supplementary material for both readers and instructors.

Deep Learning

다중혈관 관상동맥 환자에서 y-문합을 이용하여 양쪽 내흉동맥만을 사용한 우회술의 조기 성적

Machine Learning is the study of methods for programming computers to learn. Computers are applied to a wide range of tasks, and for most of these it is relatively easy for programmers to design and implement the necessary software. However, there are many tasks for which this is difficult or impossible. These can be divided into four general categories. First, there are problems for which there exist no human experts. For example, in modern automated manufacturing facilities, there is a need to predict machine failures before they occur by analyzing sensor readings. Because the machines are new, there are no human experts who can be interviewed by a programmer to provide the knowledge necessary to build a computer system. A machine learning system can study recorded data and subsequent machine failures and learn prediction rules. Second, there are problems where human experts exist, but where they are unable to explain their expertise. This is the case in many perceptual tasks, such as speech recognition, hand-writing recognition, and natural language understanding. Virtually all humans exhibit expert-level abilities on these tasks, but none of them can describe the detailed steps that they follow as they perform them. Fortunately, humans can provide machines with examples of the inputs and correct outputs for these tasks, so machine learning algorithms can learn to map the inputs to the outputs. Third, there are problems where phenomena are changing rapidly. In finance, for example, people would like to predict the future behavior of the stock market, of consumer purchases, or of exchange rates. These behaviors change frequently, so that even if a programmer could construct a good predictive computer program, it would need to be rewritten frequently. A learning program can relieve the programmer of this burden by constantly modifying and tuning a set of learned prediction rules. Fourth, there are applications that need to be customized for each computer user separately. Consider, for example, a program to filter unwanted electronic mail messages. Different users will need different filters. It is unreasonable to expect each user to program his or her own rules, and it is infeasible to provide every user with a software engineer to keep the rules up-to-date. A machine learning system can learn which mail messages the user rejects and maintain the filtering rules automatically. Machine learning addresses many of the same research questions as the fields of statistics, data mining, and psychology, but with differences of emphasis. Statistics focuses on understanding the phenomena that have generated the data, often with the goal of testing different hypotheses about those phenomena. Data mining seeks to find patterns in the data that are understandable by people. Psychological studies of human learning aspire to understand the mechanisms underlying the various learning behaviors exhibited by people (concept learning, skill acquisition, strategy change, etc.).

Machine learning

[신간의 별자리x] 우리/미술, 그리고 ‘슬픔의 박물관’

Continuous word representations, trained on large unlabeled corpora are useful for many natural language processing tasks. Popular models to learn such representations  ignore the morphology of words, by assigning a distinct vector to each word. This is a limitation, especially for languages with large vocabularies and many rare words. In this paper, we propose a new approach based on the skipgram model, where each word is represented as a bag of character n-grams. A vector representation is associated to each character n-gram, words being represented as the sum of these representations. Our method is fast, allowing to train models on large corpora quickly and allows to compute word representations for words that did not appear in the training data. We evaluate our word representations on nine different languages, both on word similarity and analogy tasks. By comparing to recently proposed morphological word representations, we show that our vectors achieve state-of-the-art performance on these tasks.

/pdf/enriching-word-vectors-with-subword-information-22sqq6w40j.pdf

Enriching Word Vectors with Subword Information

The diverse relationships among real-world events, including coreference, temporal, causal, and subevent relations, are fundamental to understanding natural languages. However, two drawbacks of existing datasets limit event relation extraction (ERE) tasks: (1) Small scale. Due to the annotation complexity, the data scale of existing datasets is limited, which cannot well train and evaluate data-hungry models. (2) Absence of unified annotation. Different types of event relations naturally interact with each other, but existing datasets only cover limited relation types at once, which prevents models from taking full advantage of relation interactions. To address these issues, we construct a unified large-scale human-annotated ERE dataset MAVEN-ERE with improved annotation schemes. It contains 103,193 event coreference chains, 1,216,217 temporal relations, 57,992 causal relations, and 15,841 subevent relations, which is larger than existing datasets of all the ERE tasks by at least an order of magnitude. Experiments show that ERE on MAVEN-ERE is quite challenging, and considering relation interactions with joint learning can improve performances. The dataset and source codes can be obtained from https://github.com/THU-KEG/MAVEN-ERE.

MAVEN-ERE: A Unified Large-scale Dataset for Event Coreference, Temporal, Causal, and Subevent Relation Extraction

Pre-trained language models (PLMs) may fail in giving reliable estimates of their predictive uncertainty. We take a close look into this problem, aiming to answer two questions: (1) Do PLMs learn to become calibrated in the training process? (2) How effective are existing calibration methods? For the first question, we conduct fine-grained control experiments to study the dynamic change in PLMs’ calibration performance in training. We consider six factors as control variables, including dataset difficulty, available training samples, training steps, the number of tunable parameters, model scale, and pretraining. We observe a consistent change in calibration performance across six factors. We find that PLMs don’t learn to become calibrated in training, evidenced by the continual increase in confidence, no matter whether the predictions are correct or not. We highlight that our finding somewhat contradicts two established conclusions: (a) Larger PLMs are more calibrated; (b) Pretraining improves model calibration. Next, we study the effectiveness of existing calibration methods in mitigating the overconfidence issue. Besides unlearnable calibration methods (e.g., label smoothing), we adapt and extend two recently proposed learnable methods that directly collect data to train models to have reasonable confidence estimations. Experimental results show that learnable methods significantly reduce PLMs’ confidence in wrong predictions.

A Close Look into the Calibration of Pre-trained Language Models

Previous research often failed to prepared polymer-derived SiOC ceramic components with dense and thick monoliths/skeletons based on additive manufacturing processes, due to the porosities and cracks generated during the pyrolysis process. Herein, we report a simple solution which combines the introduction of phenolic resin (PR) as an additive and the use of controlled strategy of pyrolysis, for the vat photopolymerization additively manufactured SiOC ceramics. As a result, crack-free dense monolith and lattice skeleton structural polymer-derived SiOC ceramics are successfully prepared despite the low ceramic yield of 19.3 wt% and large linear shrinkage of 50% generated after pyrolysis. The prepared dense monoliths possess a thickness of over 5 mm and the lattices show attainable maximum dense skeletons thickness of ~2 mm. The maximum dense thickness of the samples after pyrolysis is over 7 times that of other samples prepared based on additive manufacturing reported in the literature. As an indicator of dimensional retention, the specific dense thickness to ceramic yield ratio (T/Y) is deduced, resulting maximum values of 101.9 µm/(wt%) for lattice skeletons and 254.7 µm/(wt%) for monoliths. The former is even larger than that for dense monolith SiOC structures prepared by other conventional gel casting and isostatic pressing techniques with much higher ceramic yields of > 80 wt%, and is about 6.1 times the largest T/Y ratio (=16.7 µm/(wt%)) achievable in the AM-based works reported in the literature. This shows predominance of the current work over other types of polymer-derived ceramic structures prepared by additive manufacturing and even conventional methods. The lattice structures’ apparent density and specific strength reached 0.55 g/cm 3 and 6.6 × 10 4 N∙m/kg, respectively. The mechanical performance surpasses other structural polymer-derived ceramic structures with similar apparent density reported in the literature. Through the advanced characterization and analysis of material phase and morphology evolution during the pyrolysis process, the mechanism of defect-free densification is revealed. That is, the introduction of PR enables a highly smooth gas-releasing process due to the generation of ordered gas-releasing channels in the sample bodies during the pyrolysis process. As a result, the generation of cracks and defects was successfully prevented. Moreover, due to the high carbon residue of the PR, the samples containing 15 wt% PR in the preceramic state still possessed amorphous SiOC ceramics after pyrolysis at 1400 ℃, while 0 wt% PR samples showed SiC crystallization. Compared with similar materials reported, the ceramics prepared in this study exhibit enhanced structural and mechanical performance. The strategy investigated in the study can be a viable route for the additive manufacturing of high-performance crack-free and thick monolith and lattice skeleton SiOC ceramic structures based on vat photopolymerization. 

3D printing of crack-free dense polymer-derived ceramic monoliths and lattice skeletons with improved thickness and mechanical performance

Entity Linking is to link a name string from plain-text documents to the corresponding entry in given knowledge base. In this paper we demonstrate our entity linking system for TAC KBP 2011 Track. Our system implements pairwise and listwise learning to rank methods to create a ranking list of candidates with several kinds of features, including context similarity, term frequency, key entity extraction and WikiPage information. We participate in entity linking and cross-lingual entity linking task. We use random forest to validate the top 1 candidate recommended by our system. We achieve a performance of 72.9% F1 measure for both two tasks.

/pdf/thunlp-at-tac-kbp-2011-in-entity-linking-3ezdlhtkxs.pdf

THUNLP at TAC KBP 2011 in Entity Linking

Conceptual knowledge is fundamental to human cognition and knowledge bases. However, existing knowledge probing works only focus on evaluating factual knowledge of pre-trained language models (PLMs) and ignore conceptual knowledge. Since conceptual knowledge often appears as implicit commonsense behind texts, designing probes for conceptual knowledge is hard. Inspired by knowledge representation schemata, we comprehensively evaluate conceptual knowledge of PLMs by designing three tasks to probe whether PLMs organize entities by conceptual similarities, learn conceptual properties, and conceptualize entities in contexts, respectively. For the tasks, we collect and annotate 24k data instances covering 393 concepts, which is COPEN, a COnceptual knowledge Probing bENchmark. Extensive experiments on different sizes and types of PLMs show that existing PLMs systematically lack conceptual knowledge and suffer from various spurious correlations. We believe this is a critical bottleneck for realizing human-like cognition in PLMs. COPEN and our codes are publicly released at https://github.com/THU-KEG/COPEN.

Zhiyuan Liu

Papers

MAVEN-ERE: A Unified Large-scale Dataset for Event Coreference, Temporal, Causal, and Subevent Relation Extraction

A Close Look into the Calibration of Pre-trained Language Models

3D printing of crack-free dense polymer-derived ceramic monoliths and lattice skeletons with improved thickness and mechanical performance

THUNLP at TAC KBP 2011 in Entity Linking

COPEN: Probing Conceptual Knowledge in Pre-trained Language Models