Machine Learning is the study of methods for programming computers to learn. Computers are applied to a wide range of tasks, and for most of these it is relatively easy for programmers to design and implement the necessary software. However, there are many tasks for which this is difficult or impossible. These can be divided into four general categories. First, there are problems for which there exist no human experts. For example, in modern automated manufacturing facilities, there is a need to predict machine failures before they occur by analyzing sensor readings. Because the machines are new, there are no human experts who can be interviewed by a programmer to provide the knowledge necessary to build a computer system. A machine learning system can study recorded data and subsequent machine failures and learn prediction rules. Second, there are problems where human experts exist, but where they are unable to explain their expertise. This is the case in many perceptual tasks, such as speech recognition, hand-writing recognition, and natural language understanding. Virtually all humans exhibit expert-level abilities on these tasks, but none of them can describe the detailed steps that they follow as they perform them. Fortunately, humans can provide machines with examples of the inputs and correct outputs for these tasks, so machine learning algorithms can learn to map the inputs to the outputs. Third, there are problems where phenomena are changing rapidly. In finance, for example, people would like to predict the future behavior of the stock market, of consumer purchases, or of exchange rates. These behaviors change frequently, so that even if a programmer could construct a good predictive computer program, it would need to be rewritten frequently. A learning program can relieve the programmer of this burden by constantly modifying and tuning a set of learned prediction rules. Fourth, there are applications that need to be customized for each computer user separately. Consider, for example, a program to filter unwanted electronic mail messages. Different users will need different filters. It is unreasonable to expect each user to program his or her own rules, and it is infeasible to provide every user with a software engineer to keep the rules up-to-date. A machine learning system can learn which mail messages the user rejects and maintain the filtering rules automatically. Machine learning addresses many of the same research questions as the fields of statistics, data mining, and psychology, but with differences of emphasis. Statistics focuses on understanding the phenomena that have generated the data, often with the goal of testing different hypotheses about those phenomena. Data mining seeks to find patterns in the data that are understandable by people. Psychological studies of human learning aspire to understand the mechanisms underlying the various learning behaviors exhibited by people (concept learning, skill acquisition, strategy change, etc.).

Machine learning

In Distributed Algorithms, Nancy Lynch provides a blueprint for designing, implementing, and analyzing distributed algorithms. She directs her book at a wide audience, including students, programmers, system designers, and researchers.



Distributed Algorithms contains the most significant algorithms and impossibility results in the area, all in a simple automata-theoretic setting. The algorithms are proved correct, and their complexity is analyzed according to precisely defined complexity measures. The problems covered include resource allocation, communication, consensus among distributed processes, data consistency, deadlock detection, leader election, global snapshots, and many others.



The material is organized according to the system model-first by the timing model and then by the interprocess communication mechanism. The material on system models is isolated in separate chapters for easy reference.



The presentation is completely rigorous, yet is intuitive enough for immediate comprehension. This book familiarizes readers with important problems, algorithms, and impossibility results in the area: readers can then recognize the problems when they arise in practice, apply the algorithms to solve them, and use the impossibility results to determine whether problems are unsolvable. The book also provides readers with the basic mathematical tools for designing new algorithms and proving new impossibility results. In addition, it teaches readers how to reason carefully about distributed algorithms-to model them formally, devise precise specifications for their required behavior, prove their correctness, and evaluate their performance with realistic measures.


Table of Contents

1 Introduction 
2 Modelling I; Synchronous Network Model 
3 Leader Election in a Synchronous Ring 
4 Algorithms in General Synchronous Networks 
5 Distributed Consensus with Link Failures 
6 Distributed Consensus with Process Failures 
7 More Consensus Problems 
8 Modelling II: Asynchronous System Model 
9 Modelling III: Asynchronous Shared Memory Model 
10 Mutual Exclusion 
11 Resource Allocation 
12 Consensus 
13 Atomic Objects 
14 Modelling IV: Asynchronous Network Model 
15 Basic Asynchronous Network Algorithms 
16 Synchronizers 
17 Shared Memory versus Networks 
18 Logical Time 
19 Global Snapshots and Stable Properties 
20 Network Resource Allocation 
21 Asynchronous Networks with Process Failures 
22 Data Link Protocols 
23 Partially Synchronous System Models 
24 Mutual Exclusion with Partial Synchrony 
25 Consensus with Partial Synchrony

Distributed algorithms

The book is outstanding and admirable in many respects. ... is necessary reading for all kinds of readers from undergraduate students to top authorities in the field. Journal of Symbolic Logic Written by two experts in the field, this is the only comprehensive and unified treatment of the central ideas and their applications of Kolmogorov complexity. The book presents a thorough treatment of the subject with a wide range of illustrative applications. Such applications include the randomness of finite objects or infinite sequences, Martin-Loef tests for randomness, information theory, computational learning theory, the complexity of algorithms, and the thermodynamics of computing. It will be ideal for advanced undergraduate students, graduate students, and researchers in computer science, mathematics, cognitive sciences, philosophy, artificial intelligence, statistics, and physics. The book is self-contained in that it contains the basic requirements from mathematics and computer science. Included are also numerous problem sets, comments, source references, and hints to solutions of problems. New topics in this edition include Omega numbers, KolmogorovLoveland randomness, universal learning, communication complexity, Kolmogorov's random graphs, time-limited universal distribution, Shannon information and others.

An Introduction to Kolmogorov Complexity and Its Applications

We introduce the concept of unreliable failure detectors and study how they can be used to solve Consensus in asynchronous systems with crash failures. We characterise unreliable failure detectors in terms of two properties—completeness and accuracy. We show that Consensus can be solved even with unreliable failure detectors that make an infinite number of mistakes, and determine which ones can be used to solve Consensus despite any number of crashes, and which ones require a majority of correct processes. We prove that Consensus and Atomic Broadcast are reducible to each other in asynchronous systems with crash failures; thus, the above results also apply to Atomic Broadcast. A companion paper shows that one of the failure detectors introduced here is the weakest failure detector for solving Consensus [Chandra et al. 1992].

/pdf/unreliable-failure-detectors-for-reliable-distributed-4o7ni5mkch.pdf

Unreliable failure detectors for reliable distributed systems

In this chapter we survey the theory of two-party communication complexity. This field of theoretical computer science aims at studying the following, seemingly very simple, scenario: There are two players Alice who holds an n-bit string x and Bob who holds an n-bit string y. Their goal is to communicate in order to compute the value of some boolean function f(x, y), while exchanging a number of bits which is as small as possible. In the first part of this survey we present, mainly by giving examples, some of the results (and techniques) developed as part of this theory. We put an emphasis on proving lower bounds on the amount of communication that must be exchanged in the above scenario for certain functions f . In the second part of this survey we will exemplify the wide applicability of the results proved in the first part to other areas of computer science. While it is obvious that there are many applications of the results to problems in which communication is involved (e.g., in distributed systems), we concentrate on applications in which communication does not appear explicitly in the statement of the problems. In particular, we present results regarding the following models of computation: • Finite automata • Turing machines • Decision trees • Ordered binary decision diagrams (OBDDs) • VLSI chips • Networks of threshold gates We provide references to many other issues and applications of communication complexity which are not discussed in this survey.

Communication Complexity

This paper is concerned with the solvability of the problem of processor renaming in unreliable, completely asynchronous distributed systems. Fischer et al. prove in [8] that “nontrivial consensus” cannot be attained in such systems, even when only a single, benign processor failure is possible. In contrast, this paper shows that problems of processor renaming can be solved even in the presence of up to t

Renaming in an asynchronous environment

Byzantine Agreement has become increasingly important in establishing distributed properties when errors may exist in the systems. Recent polynomial algorithms for reaching Byzantine Agreement provide us with feasible solutions for obtaining coordination and synchronization in distributed systems. In this paper the amount of information exchange necessary to ensure Byzantine Agreement is studied. This is measured by the total number of messages the participating processors have to send in the worst case. In algorithms that use a signature scheme, the number of signatures appended to messages are also counted.First it is shown that O(nt) is a lower bound for the number of signatures for any algorithm using authentication, where n denotes the number of processors and t the upper bound on the number of faults the algorithm is supposed to handle. For algorithms that reach Byzantine Agreement without using authentication this is even a lower bound for the total number of messages. If n is large compared to t, these bounds match the upper bounds from previously known algorithms. For the number of messages in the authenticated case we prove the lower bound O(n + t2). Finally algorithms that achieve this bound are presented.

/pdf/bounds-on-information-exchange-for-byzantine-agreement-5cty9szqqz.pdf

Bounds on information exchange for Byzantine agreement

Probabilistic parallel algorithms are described to sort n keys and to select the k-smallest element among them. For each problem we construct a probabilistic parallel decision tree. The tree for selection finishes with high probability in constant time and the sorting tree in time $O(\log n)$. The same time bound for sorting can also be achieved by a probabilistic parallel machine consisting of n RAMs, each with small private memory, and a common memory of size $O(n)$. These algorithms meet the information theoretic lower bounds.

Probabilistic Parallel Algorithms for Sorting and Selection

Every alternating t(n)-time bounded multitape Turing machine can be simulated by an alternating t(n)-time bounded 1-tape Turing machine. Every nondeterministic t(n)-time bounded 1-tape Turing machine can be simulated by an alternating (n + (t(n)) 1/2)-timebounded 1-tape Turing machine. For wellbehaved functions t(n) every nondeterministic t(n)-time bounded 1-tape Turing machine can be simulated by a deterministic ((nlogn)1/2 + (t(n))1/2)-tape bounded off-line Turing machine. These results improve or extend results by Chandra-Stockmeyer, Lipton-Tarjan and Paterson.

On alternation

We define a new model for algorithms to reach Byzantine Agreement. It allows to measure the complexity more accurately, to differentiate between processor faults and to include communication link failures. A deterministic algorithm is presented that exhibits early stopping by phase 2f+4 in the worst case, where f is the actual number of faults, under less stringent conditions than the ones of previous algorithms. Also its average performance can easily be analysed making realistic assumptions on random distributions of faults. We show that it stops with high probability after a small number of phases.

Rüdiger Reischuk

Papers

Renaming in an asynchronous environment

Bounds on information exchange for Byzantine agreement

Probabilistic Parallel Algorithms for Sorting and Selection

On alternation

A new solution for the byzantine generals problem