Machine Learning is the study of methods for programming computers to learn. Computers are applied to a wide range of tasks, and for most of these it is relatively easy for programmers to design and implement the necessary software. However, there are many tasks for which this is difficult or impossible. These can be divided into four general categories. First, there are problems for which there exist no human experts. For example, in modern automated manufacturing facilities, there is a need to predict machine failures before they occur by analyzing sensor readings. Because the machines are new, there are no human experts who can be interviewed by a programmer to provide the knowledge necessary to build a computer system. A machine learning system can study recorded data and subsequent machine failures and learn prediction rules. Second, there are problems where human experts exist, but where they are unable to explain their expertise. This is the case in many perceptual tasks, such as speech recognition, hand-writing recognition, and natural language understanding. Virtually all humans exhibit expert-level abilities on these tasks, but none of them can describe the detailed steps that they follow as they perform them. Fortunately, humans can provide machines with examples of the inputs and correct outputs for these tasks, so machine learning algorithms can learn to map the inputs to the outputs. Third, there are problems where phenomena are changing rapidly. In finance, for example, people would like to predict the future behavior of the stock market, of consumer purchases, or of exchange rates. These behaviors change frequently, so that even if a programmer could construct a good predictive computer program, it would need to be rewritten frequently. A learning program can relieve the programmer of this burden by constantly modifying and tuning a set of learned prediction rules. Fourth, there are applications that need to be customized for each computer user separately. Consider, for example, a program to filter unwanted electronic mail messages. Different users will need different filters. It is unreasonable to expect each user to program his or her own rules, and it is infeasible to provide every user with a software engineer to keep the rules up-to-date. A machine learning system can learn which mail messages the user rejects and maintain the filtering rules automatically. Machine learning addresses many of the same research questions as the fields of statistics, data mining, and psychology, but with differences of emphasis. Statistics focuses on understanding the phenomena that have generated the data, often with the goal of testing different hypotheses about those phenomena. Data mining seeks to find patterns in the data that are understandable by people. Psychological studies of human learning aspire to understand the mechanisms underlying the various learning behaviors exhibited by people (concept learning, skill acquisition, strategy change, etc.).

Machine learning

From the Publisher:
With this text, you gain an understanding of the fundamental concepts of algorithms, the very heart of computer science. It introduces the basic data structures and programming techniques often used in efficient algorithms. Covers use of lists, push-down stacks, queues, trees, and graphs. Later chapters go into sorting, searching and graphing algorithms, the string-matching algorithms, and the Schonhage-Strassen integer-multiplication algorithm. Provides numerous graded exercises at the end of each chapter.


0201000296B04062001

The Design and Analysis of Computer Algorithms

A digital computer is generally believed to be an efficient universal computing device; that is, it is believed able to simulate any physical computing device with an increase in computation time by at most a polynomial factor. This may not be true when quantum mechanics is taken into consideration. This paper considers factoring integers and finding discrete logarithms, two problems which are generally thought to be hard on a classical computer and which have been used as the basis of several proposed cryptosystems. Efficient randomized algorithms are given for these two problems on a hypothetical quantum computer. These algorithms take a number of steps polynomial in the input size, e.g., the number of digits of the integer to be factored.

Polynomial-Time Algorithms for Prime Factorization and Discrete Logarithms on a Quantum Computer

Our growing dependence on increasingly complex computer and software systems necessitates the development of formalisms, techniques, and tools for assessing functional properties of these systems. One such technique that has emerged in the last twenty years is model checking, which systematically (and automatically) checks whether a model of a given system satisfies a desired property such as deadlock freedom, invariants, and request-response properties. This automated technique for verification and debugging has developed into a mature and widely used approach with many applications. Principles of Model Checking offers a comprehensive introduction to model checking that is not only a text suitable for classroom use but also a valuable reference for researchers and practitioners in the field. The book begins with the basic principles for modeling concurrent and communicating systems, introduces different classes of properties (including safety and liveness), presents the notion of fairness, and provides automata-based algorithms for these properties. It introduces the temporal logics LTL and CTL, compares them, and covers algorithms for verifying these logics, discussing real-time systems as well as systems subject to random phenomena. Separate chapters treat such efficiency-improving techniques as abstraction and symbolic manipulation. The book includes an extensive set of examples (most of which run through several chapters) and a complete set of basic results accompanied by detailed proofs. Each chapter concludes with a summary, bibliographic notes, and an extensive list of exercises of both practical and theoretical nature.

/pdf/principles-of-model-checking-249iimickz.pdf

Principles of Model Checking

SPIN is an efficient verification system for models of distributed software systems. It has been used to detect design errors in applications ranging from high-level descriptions of distributed algorithms to detailed code for controlling telephone exchanges. The paper gives an overview of the design and structure of the verifier, reviews its theoretical foundation, and gives an overview of significant practical applications.

/pdf/the-model-checker-spin-1bgo4jtefp.pdf

The model checker SPIN

computer model in our later work. I personally was deeply impressed with Shannon's communication theory [12]. Shannon's theory gave precise quantitative laws of how much inforlnation can be \"reliably\" transmitted over a noisy channel in terms of the channel capacity and the entropy of the inforlnation source, l loved physics for its beautifully precise laws that govern and explaiu the behavior of the physical world. In Shannon's work, ti)r the first time, 1 saw precise quantitative laws that governed the behavior of the abstract entity of information. For an ex-physicist the idea that there could be quantitative laws governing such abstract entities as information and its transmission was surprising and imlnensely fascinating. Shannon had given a beautiful example of quantitative laws for intbrmation which by its nature is not directly constrained by physical laws. This raised the question whether there could be precise quantitative laws that govern the abstract process of computing, which again was not directly constrained by physical laws. Could there be quantitative laws that de termine for each problem how much computing effort (work) is required for its solution and how to measure and determine it? From these and other considerations grew our deep conviction that there lnust be quantitative laws that govern the behavior of information and computing. The results of this research eftbrt were summarized in our illSt paper on this topic, which also named this new research area, \"On the computat ional complexity of algorithms\" [5]. To capture the quantitative behavior of the comput ing effort and to classify computations by their intrinsic computational complexity, which we were seeking, we needed a robust computing model and an intuitively satisfying classification of the complexity of problems. The Tur ing machine was ideally suited for the computer lnodel, and we modified it to the multi-tape version. To classify computations (or problems) we introduced the key concept of a complexity class in terms of the Tur ing machines with bounded computat ional resources. A complexity class, for example, C,,e in our original notation, consists of all problems whose instances of length n can be solved in n 2 steps on a multi-tape Turing machine. In contemporary notation, C,,z = TIME[n2]. Today, complexity classes are central objects of study, and many results and problems in complexity theory are expressed in terms of complexity classes. A considerable part of our early work on complexity theory was dedicated to showing that we had defined a meaningful classification of problems according to their computat ional difficulty and deriving results about it. We showed that our classification was robust and was not essentially altered by minor changes in the model and that the complexity classification indeed captured the intuitive ideas about the complexity of numbers and functions. We explored how computat ion speed changed by going from one-tape to multi-tape machines and even to multidimensional tapes and derived bounds tbl these \"speed8 October 19941Vol.37, No.10 COMMUN|¢ATIOIHUG OP THIil ACM ups.\" Somewhat later, Manuel Blum, in his Ph.D. dissertation at MIT [1 ], developed an axiomatic theory of computational complexity and, among many other results, showed that all complexity measures are recursively related. Our speed-up results were special cases of this relationship. For us it was a delight to meet Manuel while he was writing his dissertation and to exchange ideas about computational complexity. Similarly, we were impressed and influenced by H. Yamada's work on real-time computations in his dissertation at the University of Pennsylvania [15] under the supervision of Robert McNaughton. We also proved Hierarchy Theorelns that asserted that a slight increase in computation time (bounds) permits sohltion of new problelns. More explicity: if T(n) and U(n) are \"nice\" functions and T(n) ~ lim = 0 ' , >~ U ( n ) then complexity class TIME[T(n)] is properly contained in TIME[U(n)]. These results showed that there are problems with very sharp, intrinsic computational complexity bounds. No lnatter what method and computational algorithm was used, the problem solution required, say n 2, operation for problem instance of size n. Blum in his dissertation showed that this is not the case for all problems and that there can exist exotic problems with less sharply defined bounds. To relate our classification of the real numbers by their computation complexity to the classical concepts, we showed that all algebraic numbers are in the low complexity class TIME[n\"] and found, to our surprise, some transcendental numbers that were real-time computable (i.e., they were in TIME[n]). Since we could not prove that any irrational algebraic numbers were in TIME[n], we conjectured that all real-time computable numbers are either rational or transcendental. This is still an open problem 30 years later and only gradually did we realize its mathematical depth and the profound consequences its p roof would have in mathematics. Toward the end of the introduction of our first paper on complexity theory [5], we state: \"The final section is devoted to open questions and problem areas. It is our conviction that numbers and functions have an intrinsic computational nature according to which they can be classified, as shown in this paper, and that there is a good opportuni ty here for further research.\" Indeed there was! We had opened a new computer science area of research and given it a name. At the GE Research Laboratory, Phil Louis jo ined us to explore tape(or memory) bounded computations that yielded many interesting results and established computational space as another major computational resource measure [7, 13]. We showed that all context-free languages could be recognized on (log n)2-tape. This result led Savitch [10] to his elegant result about the relation between deterministic and nondeterministic tapebounded computations: for \"nice\" functions F(n), NTAPE[F(n)] is contained in TAPE[F(n)2]. Our colleague at the Laboratory, Daniel Younger [16], showed that context-free languages were contained in TIME[n:~]. Soon many others jo ined the exploration of the complexity of computation, and COlnputational complexity theory grew into a major research area with deep and interesting results and some of the most notorious open problems in computer science. Looking at all of computer science and its history, 1 am very impressed by the scientific and technological achievements, and they far exceed what I had expected. Computer science has grown into an important science with rich intellectual achievements, an impressive arsenal of practical results and exciting fllture challenges. Equally impressive are the unprecedented technological developments in computing power and communication capacity that have amplified the scientific achievements and have given computing and computer science a central role in our scientific, intellectual and commercial activities. 1 personally believe that computer science is not only a rapidly maturing science, but that it is more. Computer science differs so basically f iom the other sciences that it has to be viewed as a new species among the sciences, and it must be so understood. Computer science deals with intbrmation, its creation and processing, and with the systems that perform it, much of which is not directly restrained and governed by physical laws. Thus computer science is laying the foundations and developing the research paradigms and scientific methods for the exploration of the world of informat ion and intellectual processes that are not directly governed by physical laws. This is what sets it apart from the other sciences and what we vaguely perceived and found fascinating in our early exploration of computational complexity. One of the defining characteristics of computer science is the immense difference in scale of the phenomena computer science deals with. From the individual bits of programs and data in the computers to billions of operations per second on this infbrmation by the highly complex machines, their operat ing systems and the various languages in which the problems are described, the scale changes through many orders of magnitude. Donald Knuth 2 puts it nicely: Computer Science and Engineering is a field that attracts a different kind of thinker. I believe that one who is a natural computer scientist thinks algorithmically. Such people are especiaUy good at dealing with situations where different rules apply in different cases; they are individuals who can rapidly change levels of abstraction, simultaneously seeing thin U \"in the large\" and \"in the small.\" The computer scientist has to create many levels of abstractions to deal with these problems. One has to create intellectual tools to conceive, design, control, program, and reason about the most complicated of human creations. Fur thermore , this has to be done with unprecedented precision. The underlying hardware that executes the computations are universal machines and therefore they are chaotic systems: the slightest change in their instructions or data can result in arbitrarily large differences in the results. This, as we well know, is an inherent prop2Personal conmmnication. March 10, 1992 letter. COMMUNICAT IONSOFTHE ACM Octobcr 1994/Vo1.37, No.lO ~

Turing Award lecture on computational complexity and the nature of computer science

One of the Grand Challenges to computer science is to understand what is and is not feasibly computable. Recursive function theory clarified what is and is not effectively computable and in the process extended our understanding of Goedel incompleteness results about the limits of the power of formal mathematical methods. Since computing is universal and encompasses the power of mathematics, the understanding of the limits of the feasibly computable could give a deeper understanding of the limits of rational intellectual processes and insights into the power and limits of scientific theories. The search for what is and is not feasibly computable has two distinct aspects. The first is to determine (estimate) how much and what kind of computing power will be available in the foreseeable future. The other problem is to determine what kind of problems can be solved with these available computing resources. The first problem is a technological assessment of the existing and potential computing technologies to estimate what kind and how much computing work our machines will be able to render. The other problem leads us to the central questions of complexity theory: what is the intrinsic complexity of important classes of problems we wish to solve. Clearly, the P=NP=PSPACE? problems are among the best known in this area. In all these considerations, the exponential function seems to give a crude upper limit for the feasibly computable. May it be time, memory or weight requirements, if they grow exponentially, then the computations are not feasible. We do not know what computations require exponential amounts of resources, nor do all instances of problems in exponential complexity classes require exponential resources. Even if the exact solutions require exponential resources, good approximations to the solution may not. But if indeed the problem requires exponential resources, then it is clearly not feasibly solvable already for moderate size instances of the problem.

/pdf/on-the-weight-of-computations-dc37fhl71s.pdf

On the Weight of Computations.

On Sparse Sets in NP-P

The quantitative aspects of one-tape Turing machine computations are considered. It is shown, for instance, that there exists a sharp time bound which must be reached for the recognition of nonregular sets of sequences. It is shown that the computation time can be used to characterize the complexity of recursive sets of sequences, and several results are obtained about this classification. These results are then applied to the recognition speed of context-free languages and it is shown, among other things, that it is recursively undecidable how much time is required to recognize a nonregular context-free language on a one-tape Turing machine. Several unsolved problems are discussed.

/pdf/computational-complexity-of-one-tape-turing-machine-3h3gfmafy2.pdf

Computational Complexity of One-Tape Turing Machine Computations

Communication mechanisms used in practice.- Terminology.- to store-and-forward routing.- The routing number.- Offline routing protocols.- Oblivious routing protocols.- Adaptive routing protocols.- Compact routing protocols.- to wormhole routing.- Oblivious routing protocols.- Protocols for all-optical networks.- Summary and future directions.

Juris Hartmanis

Papers

Turing Award lecture on computational complexity and the nature of computer science

On the Weight of Computations.

On Sparse Sets in NP-P

Computational Complexity of One-Tape Turing Machine Computations

Universal Routing Strategies for Interconnection Networks