Machine Learning is the study of methods for programming computers to learn. Computers are applied to a wide range of tasks, and for most of these it is relatively easy for programmers to design and implement the necessary software. However, there are many tasks for which this is difficult or impossible. These can be divided into four general categories. First, there are problems for which there exist no human experts. For example, in modern automated manufacturing facilities, there is a need to predict machine failures before they occur by analyzing sensor readings. Because the machines are new, there are no human experts who can be interviewed by a programmer to provide the knowledge necessary to build a computer system. A machine learning system can study recorded data and subsequent machine failures and learn prediction rules. Second, there are problems where human experts exist, but where they are unable to explain their expertise. This is the case in many perceptual tasks, such as speech recognition, hand-writing recognition, and natural language understanding. Virtually all humans exhibit expert-level abilities on these tasks, but none of them can describe the detailed steps that they follow as they perform them. Fortunately, humans can provide machines with examples of the inputs and correct outputs for these tasks, so machine learning algorithms can learn to map the inputs to the outputs. Third, there are problems where phenomena are changing rapidly. In finance, for example, people would like to predict the future behavior of the stock market, of consumer purchases, or of exchange rates. These behaviors change frequently, so that even if a programmer could construct a good predictive computer program, it would need to be rewritten frequently. A learning program can relieve the programmer of this burden by constantly modifying and tuning a set of learned prediction rules. Fourth, there are applications that need to be customized for each computer user separately. Consider, for example, a program to filter unwanted electronic mail messages. Different users will need different filters. It is unreasonable to expect each user to program his or her own rules, and it is infeasible to provide every user with a software engineer to keep the rules up-to-date. A machine learning system can learn which mail messages the user rejects and maintain the filtering rules automatically. Machine learning addresses many of the same research questions as the fields of statistics, data mining, and psychology, but with differences of emphasis. Statistics focuses on understanding the phenomena that have generated the data, often with the goal of testing different hypotheses about those phenomena. Data mining seeks to find patterns in the data that are understandable by people. Psychological studies of human learning aspire to understand the mechanisms underlying the various learning behaviors exhibited by people (concept learning, skill acquisition, strategy change, etc.).

Machine learning

Neural networks are both computationally intensive and memory intensive, making them difficult to deploy on embedded systems with limited hardware resources. To address this limitation, we introduce "deep compression", a three stage pipeline: pruning, trained quantization and Huffman coding, that work together to reduce the storage requirement of neural networks by 35x to 49x without affecting their accuracy. Our method first prunes the network by learning only the important connections. Next, we quantize the weights to enforce weight sharing, finally, we apply Huffman coding. After the first two steps we retrain the network to fine tune the remaining connections and the quantized centroids. Pruning, reduces the number of connections by 9x to 13x; Quantization then reduces the number of bits that represent each connection from 32 to 5. On the ImageNet dataset, our method reduced the storage required by AlexNet by 35x, from 240MB to 6.9MB, without loss of accuracy. Our method reduced the size of VGG-16 by 49x from 552MB to 11.3MB, again with no loss of accuracy. This allows fitting the model into on-chip SRAM cache rather than off-chip DRAM memory. Our compression method also facilitates the use of complex neural networks in mobile applications where application size and download bandwidth are constrained. Benchmarked on CPU, GPU and mobile GPU, compressed network has 3x to 4x layerwise speedup and 3x to 7x better energy efficiency.

https://forum.stanford.edu/events/posterslides/DeepCompressionCompressingDeepNeuralNetworkswithPruning,TrainedQuantizationandHuffmanCoding.pdf

Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding

Planning algorithms are impacting technical disciplines and industries around the world, including robotics, computer-aided design, manufacturing, computer graphics, aerospace applications, drug design, and protein folding. This coherent and comprehensive book unifies material from several sources, including robotics, control theory, artificial intelligence, and algorithms. The treatment is centered on robot motion planning but integrates material on planning in discrete spaces. A major part of the book is devoted to planning under uncertainty, including decision theory, Markov decision processes, and information spaces, which are the “configuration spaces” of all sensor-based planning problems. The last part of the book delves into planning under differential constraints that arise when automating the motions of virtually any mechanical system. Developed from courses taught by the author, the book is intended for students, engineers, and researchers in robotics, artificial intelligence, and control theory as well as computer graphics, algorithms, and computational biology.

Planning Algorithms: Introductory Material

Model Checking

A classic introduction to artificial intelligence intended to bridge the gap between theory and practice, "Principles of Artificial Intelligence" describes fundamental AI ideas that underlie applications such as natural language processing, automatic programming, robotics, machine vision, automatic theorem proving, and intelligent data retrieval. Rather than focusing on the subject matter of the applications, the book is organized around general computational concepts involving the kinds of data structures used, the types of operations performed on the data structures, and the properties of the control strategies used. "Principles of Artificial Intelligence"evolved from the author's courses and seminars at Stanford University and University of Massachusetts, Amherst, and is suitable for text use in a senior or graduate AI course, or for individual study.

Principles of Artificial Intelligence

Many IT-systems behave very differently from classical machine models: they interact with an unpredictable environment, they never terminate, and their behavior changes over time. Wegner [25,26] (see also [28]) recently argued that the power of interaction goes beyond the Church-Turing thesis. To explore interaction from a computational viewpoint, we describe a generic model of an 'interactive machine' which interacts with the environment using single streams of input and output signals over a simple alphabet. The model uses ingredients from the theory of ω-automata. Viewing the interactive machines as transducers of infinite streams of signals, we show that their interactive recognition and generation capabilities are identical. It is also shown that, in the given model, all interactively computable functions are limit-continuous.

On Algorithms and Interaction

In this paper, we investigate which processor networks allow k-label Interval Routing Schemes, under the assumption that costs of edges may vary. We show that for each fixed k≥1, the class of graphs allowing such routing schemes is closed under minor-taking in the domain of connected graphs, and hence has a linear time recognition algorithm. This result connects the theory of compact routing with the theory of graph minors and treewidth.

/pdf/on-interval-routing-schemes-and-treewidth-4is0chii6f.pdf

On Interval Routing Schemes and Treewidth

Constraints in Computational Logics

Motivated by an elementary programming system for formal language generation, we propose a generalized a pushdown acceptor which uses both a checking stack and a pushdown store in original manner. In the analysis of the machine-model parallel rewriting appears to be a fundamental tool for breaking the implicit recursion-structure. Variations of the machine-model (in the sense of Cook) lead to a natural hierarchy of machines defining DLOG, NLOG, P, NP, and PSPACE, showing different ways in which one class may be considered a restriction of another (as anticipated also in Galil's work on hierarchies of complete problems). Various applications in language theory and in the study of complexity classes are discussed.

Variations of a new machine model

Several methods exist for routing messages in a network without using complete routing tables (compact routing). In k-interval routing schemes (k-IRS), nodes assign up to k intervals to each incident link. A message is routed over a link if its destination belongs to one of the intervals of the link. We give some results for the necessary value of k in order to achieve shortest path routing. Even though for very structured networks low values of k suffice, we show that for ‘general graphs’ interval routing cannot significantly reduce the space-requirements for shortest path routing. In particular, for any δ>0, there exist classes of random graphs G n, p for all n sufficiently large such that with high probability an optimal k-IRS for a graph G ∈ G n, p requires k = Ω(n1−δ).

/pdf/the-complexity-of-interval-routing-on-random-graphs-1v9vwtw7i4.pdf

Jan van Leeuwen

Papers

On Algorithms and Interaction

On Interval Routing Schemes and Treewidth

Constraints in Computational Logics

Variations of a new machine model

The Complexity of Interval Routing on Random Graphs