Structure and interpretation of computer programs

© Gulwani, S. et al. | ACM 2015. This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in Communications of the ACM, http://dx.doi.org/10.1145/2736282

/pdf/inductive-programming-meets-the-real-world-3o03oneyse.pdf

Inductive programming meets the real world

We develop a technique for generalising from data in which models are samplers represented as program text. We establish encouraging empirical results that suggest that Markov chain Monte Carlo probabilistic programming inference techniques coupled with higher-order probabilistic programming languages are now sufficiently powerful to enable successful inference of this kind in nontrivial domains. We also introduce a new notion of probabilistic program compilation and show how the same machinery might be used in the future to compile probabilistic programs for efficient reusable predictive inference.

Learning Probabilistic Programs.

We describe an approach to automatic discovery of samplers in the form of human interpretable probabilistic programs. Specifically, we learn the procedure code of samplers for one-dimensional distributions. We formulate a Bayesian approach to this problem by specifying an adaptor grammar prior over probabilistic program code, and use approximate Bayesian computation to learn a program whose execution generates samples that match observed data or analytical characteristics of a distribution of interest. In our experiments we leverage the probabilistic programming system Anglican to perform Markov chain Monte Carlo sampling over the space of programs. Our results are competive relative to state-of-the-art genetic programming methods and demonstrate that we can learn approximate and even exact samplers.

Automatic Sampler Discovery via Probabilistic Programming and Approximate Bayesian Computation

I design a machine learning system capable of ‘cumulative learning’, which means that it automatically acquires the knowledge necessary for solving harder problems through experience of solving easier ones Working within the learning framework of inductive programming, I propose that the technique of abstraction, familiar from software engineering, is a suitable mechanism for accumulating knowledge In abstraction, syntactic patterns in solutions to past problems are isolated as re-usable units and added to background knowledge For the system’s knowledge representation language, I argue that lambda calculus is a more suitable choice than first-order logic because lambda calculus supports abstraction readily However, more mature and theoretically well-founded base inference techniques are available within first-order Inductive Logic Programming (ILP) Therefore, my approach is to adapt ILP inference techniques to lambda calculus Central to ILP is the idea of ‘generality’, and I show that a suitable concept of generality in lambda calculus arises from its standard denotational semantics Consequently, notions of entailment, subsumption, refinement, and inverse deduction have direct analogues in the lambda calculus setting I argue that the conventional ‘compression’ measure used in ILP is inflexible in capturing prior assumptions, particularly in the context of an expanding background knowledge Instead I introduce a non-parametric Bayesian prior over hypotheses and background knowledge I then design an inductive inference algorithm for the lambda calculus setting based on refinement and proof-directed search I give a formal proof of correctness of this algorithm To enable automatic invention of abstractions, I design two algorithms The first is a heuristic search that uses anti-unification to discovering opportunities for abstraction within a corpus of knowledge I give a formal characterisation of its search space The second algorithm performs inverse deduction in order to refactor knowledge in terms of an abstraction I prove that this refactoring process is semantics-preserving

Cumulative Learning in the Lambda Calculus

Inductive programming systems characteristically exhibit an exponential explosion in search time as one increases the size of the programs to be generated. As a way of overcoming this, we introduce incremental learning, a process in which an inductive programming system automatically modifies its inductive bias towards some domain through solving a sequence of gradually more difficult problems in that domain. We demonstrate a simple form of incremental learning in which a system incorporates solution programs into its background knowledge as it progresses through a sequence of problems. Using a search-based inductive functional programming system modelled on the MagicHaskeller system of Katayama [4], we perform a set of experiments comparing the performance of inductive programming with and without incremental learning. Incremental learning is shown to produce a performance improvement of at least a factor of thirty on each of the four problem sequences tested. We describe how, given some assumptions, inductive programming with incremental learning can be shown to have a polynomial, rather than exponential, time complexity with respect to the size of the program to be generated. We discuss the difficulties involved in constructing suitable problem sequences for our incremental learning system, and consider what improvements can be made to overcome these difficulties.

/pdf/incremental-learning-in-inductive-programming-3pxruadyl2.pdf

Incremental Learning in Inductive Programming

We investigate how new elements of background knowledge can be abstracted automatically from patterns in programs. The approach is implemented in the KANDINSKY system using an algorithm that searches for common subterms over sets of functional programs. We demonstrate that KANDINSKY can invent higher-order functions such as map, fold, and sumBy from small sets of input programs. An experiment shows that KANDINSKY can find high-compression abstractions efficiently, with low settings of its input parameters. Finally we compare our approach with related work in the inductive logic programming and functional programming literature, and suggest directions for further work.

/pdf/automatic-invention-of-functional-abstractions-4vho97opmr.pdf

Robert J. Henderson

Papers

Incremental Learning in Inductive Programming

Automatic Invention of Functional Abstractions.