Decision trees have proved to be valuable tools for the description, classification and generalization of data. Work on constructing decision trees from data exists in multiple disciplines such as statistics, pattern recognition, decision theory, signal processing, machine learning and artificial neural networks. Researchers in these disciplines, sometimes working on quite different problems, identified similar issues and heuristics for decision tree construction. This paper surveys existing work on decision tree construction, attempting to identify the important issues involved, directions the work has taken and the current state of the art.

Automatic Construction of Decision Trees from Data: A Multi-Disciplinary Survey

Encompassing all the major topics students will encounter in courses on the subject, the authors teach both the underlying mathematical foundations and how these ideas are implemented in practice. They illustrate all the concepts with both worked examples and plenty of exercises, and, in addition, provide software so that students can try out numerical methods and so hone their skills in interpreting the results. As a result, this will make an ideal textbook for all those coming to the subject for the first time. Authors' note: A problem recently found with the software is due to a bug in Formula One, the third party commercial software package that was used for the development of the interface. It occurs when the date, currency, etc. format is set to a non-United States version. Please try setting your computer date/currency option to the United States option . The new version of Formula One, when ready, will be posted on Www.

Linear Programming 1: Introduction

State-of-the-art decision tree methods apply heuristics recursively to create each split in isolation, which may not capture well the underlying characteristics of the dataset. The optimal decision tree problem attempts to resolve this by creating the entire decision tree at once to achieve global optimality. In the last 25?years, algorithmic advances in integer optimization coupled with hardware improvements have resulted in an astonishing 800 billion factor speedup in mixed-integer optimization (MIO). Motivated by this speedup, we present optimal classification trees, a novel formulation of the decision tree problem using modern MIO techniques that yields the optimal decision tree for axes-aligned splits. We also show the richness of this MIO formulation by adapting it to give optimal classification trees with hyperplanes that generates optimal decision trees with multivariate splits. Synthetic tests demonstrate that these methods recover the true decision tree more closely than heuristics, refuting the notion that optimal methods overfit the training data. We comprehensively benchmark these methods on a sample of 53 datasets from the UCI machine learning repository. We establish that these MIO methods are practically solvable on real-world datasets with sizes in the 1000s, and give average absolute improvements in out-of-sample accuracy over CART of 1---2 and 3---5% for the univariate and multivariate cases, respectively. Furthermore, we identify that optimal classification trees are likely to outperform CART by 1.2---1.3% in situations where the CART accuracy is high and we have sufficient training data, while the multivariate version outperforms CART by 4---7% when the CART accuracy or dimension of the dataset is low.

/pdf/optimal-classification-trees-usepxx1b8x.pdf

Optimal classification trees

Before there were computers, there was thinking about the mind as a computer-as a machine. And in this way, computer science and engineering trace their roots to using natural examples. Within these fields of endeavor, AI drew its initial inspiration from nature, and work on computer-simulated brains received the lion's share of the early attention. But even back then, nature's other metaphor of adaptation planted a different seed that is now blossoming around the globe. Specifically, Darwinian evolution has spawned a family of computational methods called genetic algorithms (GAs) or evolutionary algorithms (EAs)

Genetic and evolutionary algorithms come of age

Protection of simple series and parallel systems with components of different values

We consider the problem of minimizing the expected cost of computing the correct value of a discrete-valued function when it is costly to determine (“inspect”) the values of its arguments. This type of problem arises in distributed computing, in the design of interactive expert systems, in reliability analysis of coherent systems, in classification of pattern vectors, and in many other applications. In this paper, we first show that the general problem is NP-hard and then introduce several efficient heuristic sequential inspection procedures for solving it approximately. We obtain theoretical results showing that the heuristics are optimal in important special cases; moreover, their computational structures make them well suited for parallel implementation. Next, for the special case of linear threshold (or “discrete linear discriminant”) functions, which are widely used in statistical classification procedures, we use Monte Carlo simulation to analyze the performances of the heuristics and to compare the heuristic solutions with the exact (true minimum expected cost) solutions over a wide range of randomly generated test problems. All of the heuristics give average relative errors, compared to the exact optimal solutions, of less than 2%. The best heuristic for this class of functions gives an average relative error of less than 0.05% and runs two to four orders of magnitude faster than the exact solution procedure, for functions with ten arguments.

Heuristic least-cost computation of discrete classification functions with uncertain argument values

Interoffice fiber-optic networks are already a multi-million dollar business for U S WEST Communications, with rapid expansion anticipated for the rest of the decade. U S WEST's operations research modeling group developed a mixed-integer programming model to help plan the least-cost expansion of interoffice fiber-optic networks in major metropolitan areas throughout the 14 midwestern states in U S WEST's territory. The model augments previous telecommunications planning tools by developing explicit optimal expansion plans for multi-year planning horizons. It was tested successfully in 1991 on network and demographic data from Colorado Springs, Colorado. An outgrowth of the model is now being used to design and plan the growth of fiber-optical ring networks in Seattle, Denver, Minneapolis, and other major metropolitan areas throughout U S WEST's region.

Optimal Expansion of Fiber-Optic Telecommunications Networks in Metropolitan Areas

We report a new heuristic algorithm useful for developing least-cost expansion plans for leased telecommunications networks in which there is a hierarchy of possible transmission facilities. The problem combines network topology design and capacity expansion problems. Implemented on a PC, the algorithm has successfully developed topology growth plans for twenty periods on networks with up to 31 nodes.

Planning for optimal expansion of leased line communication networks

Suppose that items of equipment are to be added to a supply station (e.g., new switch modules are to be added to a telecommunications switch) over time to meet growing demand requirements. Both supply and demand have multiple components: an item of equipment supplies different amounts of several resources, and demand may be expressed in terms of the vector of resources required. There are several different types of equipment to choose among, each type supplying known amounts of each resource per unit of equipment. The supply station is organized into bays, shelves, or other capacitated “containers” so that when the cumulative amount of equipment added exceeds the holding capacity of the installed containers, new containers must be added, creating a relatively large jump in cumulative costs. Thus, it is desirable to sequentially “pack” items of equipment into the available containers, by choosing which types of equipment to install when, so as to minimize the total cost of covering demand in each period. We discuss an instance of this problem arising from wireless telephony and describe the performance of a conventional branch-and-bound optimization algorithm for solving it. The branch-and-bound approach works well on small instances of the problem, and has been used successfully in practical planning. However, it can take CPU-days to run, thus preventing development of a useful interactive planning tool. Therefore, we introduce a novel “seed, repair, and replace” genetic algorithm (SRR-GA) for solving dynamic packing problems of this type. We contrast its performance with the branch-and-bound algorithm‘s on both hand-generated and randomly-created dynamic packing problems, finding that the SRR-GA is two to three orders of magnitude faster and produces solutions of equal or better quality on practical problems. Variations of the dynamic packing problem and of the SRR-GA for solving it are mentioned, and the paper concludes by suggesting other potential applications of the SRR-GA to hard combinatorial optimization problems.

Warren Kuehner

Papers

Heuristic least-cost computation of discrete classification functions with uncertain argument values

Optimal Expansion of Fiber-Optic Telecommunications Networks in Metropolitan Areas

Planning for optimal expansion of leased line communication networks

Dynamic Hierarchical Packing of WirelessSwitches Using a Seed, Repair and Replace Genetic Algorithm