Machine Learning is the study of methods for programming computers to learn. Computers are applied to a wide range of tasks, and for most of these it is relatively easy for programmers to design and implement the necessary software. However, there are many tasks for which this is difficult or impossible. These can be divided into four general categories. First, there are problems for which there exist no human experts. For example, in modern automated manufacturing facilities, there is a need to predict machine failures before they occur by analyzing sensor readings. Because the machines are new, there are no human experts who can be interviewed by a programmer to provide the knowledge necessary to build a computer system. A machine learning system can study recorded data and subsequent machine failures and learn prediction rules. Second, there are problems where human experts exist, but where they are unable to explain their expertise. This is the case in many perceptual tasks, such as speech recognition, hand-writing recognition, and natural language understanding. Virtually all humans exhibit expert-level abilities on these tasks, but none of them can describe the detailed steps that they follow as they perform them. Fortunately, humans can provide machines with examples of the inputs and correct outputs for these tasks, so machine learning algorithms can learn to map the inputs to the outputs. Third, there are problems where phenomena are changing rapidly. In finance, for example, people would like to predict the future behavior of the stock market, of consumer purchases, or of exchange rates. These behaviors change frequently, so that even if a programmer could construct a good predictive computer program, it would need to be rewritten frequently. A learning program can relieve the programmer of this burden by constantly modifying and tuning a set of learned prediction rules. Fourth, there are applications that need to be customized for each computer user separately. Consider, for example, a program to filter unwanted electronic mail messages. Different users will need different filters. It is unreasonable to expect each user to program his or her own rules, and it is infeasible to provide every user with a software engineer to keep the rules up-to-date. A machine learning system can learn which mail messages the user rejects and maintain the filtering rules automatically. Machine learning addresses many of the same research questions as the fields of statistics, data mining, and psychology, but with differences of emphasis. Statistics focuses on understanding the phenomena that have generated the data, often with the goal of testing different hypotheses about those phenomena. Data mining seeks to find patterns in the data that are understandable by people. Psychological studies of human learning aspire to understand the mechanisms underlying the various learning behaviors exhibited by people (concept learning, skill acquisition, strategy change, etc.).

Machine learning

Probability Distributions.- Linear Models for Regression.- Linear Models for Classification.- Neural Networks.- Kernel Methods.- Sparse Kernel Machines.- Graphical Models.- Mixture Models and EM.- Approximate Inference.- Sampling Methods.- Continuous Latent Variables.- Sequential Data.- Combining Models.

Pattern Recognition and Machine Learning

The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.

Attention Is All You Need

We propose a unified neural network architecture and learning algorithm that can be applied to various natural language processing tasks including part-of-speech tagging, chunking, named entity recognition, and semantic role labeling. This versatility is achieved by trying to avoid task-specific engineering and therefore disregarding a lot of prior knowledge. Instead of exploiting man-made input features carefully optimized for each task, our system learns internal representations on the basis of vast amounts of mostly unlabeled training data. This work is then used as a basis for building a freely available tagging system with good performance and minimal computational requirements.

/pdf/natural-language-processing-almost-from-scratch-osj5vem4f3.pdf

Natural Language Processing (Almost) from Scratch

We describe a single convolutional neural network architecture that, given a sentence, outputs a host of language processing predictions: part-of-speech tags, chunks, named entity tags, semantic roles, semantically similar words and the likelihood that the sentence makes sense (grammatically and semantically) using a language model. The entire network is trained jointly on all these tasks using weight-sharing, an instance of multitask learning. All the tasks use labeled data except the language model which is learnt from unlabeled text and represents a novel form of semi-supervised learning for the shared tasks. We show how both multitask learning and semi-supervised learning improve the generalization of the shared tasks, resulting in state-of-the-art-performance.

/pdf/a-unified-architecture-for-natural-language-processing-deep-qlgoz5mygc.pdf

A unified architecture for natural language processing: deep neural networks with multitask learning

This paper introduces tree transducers as a unifying theory for semantic parsing models based on tree transformations. Many existing models use tree transformations, but implement specialized training and smoothing methods, which makes it difficult to modify or extend the models. By connecting to the rich literature on tree automata, we show how semantic parsing models can be developed using completely general estimation methods. We demonstrate the approach by reframing and extending one state-of-the-art model as a tree automaton. Using a variant of the inside-outside algorithm with variational Bayesian estimation, our generative model achieves higher raw accuracy than existing generative and discriminative approaches on a standard data set.

/pdf/formalizing-semantic-parsing-with-tree-transducers-w3wlo3fff6.pdf

Formalizing Semantic Parsing with Tree Transducers

There are many different ways in which external information might be used in an NLP task. This paper investigates how external syntactic information can be used most effectively in the Semantic Role Labeling (SRL) task. We evaluate three different ways of encoding syntactic parses and three different ways of injecting them into a state-of-the-art neural ELMo-based SRL sequence labelling model. We show that using a constituency representation as input features improves performance the most, achieving a new state-of-the-art for non-ensemble SRL models on the in-domain CoNLL'05 and CoNLL'12 benchmarks.

How to best use Syntax in Semantic Role Labelling

Cross-linguistic studies on unsupervised word segmentation have consistently shown that English is easier to segment than other languages. In this paper, we propose an explanation of this finding based on the notion of segmentation ambiguity. We show that English has a very low segmentation ambiguity compared to Japanese and that this difference correlates with the segmentation performance in a unigram model. We suggest that segmentation ambiguity is linked to a trade-off between syllable structure complexity and word length distribution.

Why is English so easy to segment

Probabilistic grammars define a set of well-formed or grammatical linguistic structures, just as all grammars do. But in addition probabilistic grammars also define probability distributions over these structures, which a statistical inference procedure can exploit. This paper describes two simple statistical inference procedures for probabilistic grammars and discusses some of the challenges involved in generalising them to more realistic settings. The first three sections of this paper review important ideas from statistical estimation, namely the Maximum Likelihood Principle and Bayesian estimation, and show how they can be applied to learn the parameters of a Probabilistic Context-Free Grammar from surface strings alone in an idealized “toy” scenario. We show that they succeed in learning both the phrase structure and lexical entries without explicit negative evidence in a situation that seems highly challenging for any “staged” learner, i.e., a learner that learns either lexical entries or phrase structure rules first. Then we discuss attempts to generalize these kinds of methods to more realistic scenarios, and point out some of the challenges that are involved in doing this. One of the main themes of this paper is how little we know about the the linguistic implications of the various approximations made by current statistical inference algorithms. In part because of this, much of the work in this field takes an experimental, “try it and see” approach. The presentation in this paper is deliberately informal: for a more mathematical presentation of much of this material see e.g., Geman and Johnson (2004). The software used to produce the results below is available from http://web.science.mq.edu.au/ mjohnson.

/pdf/language-acquisition-as-statistical-inference-5ezgwxbrc7.pdf

Language Acquisition as Statistical Inference

English-speaking children have acquired phrasefinal lengthening by the age of 2, but other aspects of prosodic organization appear to be later acquired. This study investigated 2-year-olds’ prosodic organization of function words in an elicited imitation task. In particular, we wanted to know if children would prosodify pronouns 1) as part of a trochaic foot with the preceding word, or 2) as a separate prosodic word. The results showed that the function word was produced as an independent prosodic unit, in contrast to the adult model being imitated. Implications for a developmental model of speech planning and production are discussed.

/pdf/prosodic-structure-in-child-speech-planning-and-production-236rba095a.pdf

Mark Johnson

Papers

Formalizing Semantic Parsing with Tree Transducers

How to best use Syntax in Semantic Role Labelling

Why is English so easy to segment

Language Acquisition as Statistical Inference

Prosodic structure in child speech planning and production