Many different machine learning algorithms exist; taking into account each algorithm's hyperparameters, there is a staggeringly large number of possible alternatives overall. We consider the problem of simultaneously selecting a learning algorithm and setting its hyperparameters, going beyond previous work that attacks these issues separately. We show that this problem can be addressed by a fully automated approach, leveraging recent innovations in Bayesian optimization. Specifically, we consider a wide range of feature selection techniques (combining 3 search and 8 evaluator methods) and all classification approaches implemented in WEKA's standard distribution, spanning 2 ensemble methods, 10 meta-methods, 27 base classifiers, and hyperparameter settings for each classifier. On each of 21 popular datasets from the UCI repository, the KDD Cup 09, variants of the MNIST dataset and CIFAR-10, we show classification performance often much better than using standard selection and hyperparameter optimization methods. We hope that our approach will help non-expert users to more effectively identify machine learning algorithms and hyperparameter settings appropriate to their applications, and hence to achieve improved performance.

/pdf/auto-weka-combined-selection-and-hyperparameter-optimization-2295gslzkw.pdf

Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms

Many different machine learning algorithms exist; taking into account each algorithm's hyperparameters, there is a staggeringly large number of possible alternatives overall. We consider the problem of simultaneously selecting a learning algorithm and setting its hyperparameters, going beyond previous work that addresses these issues in isolation. We show that this problem can be addressed by a fully automated approach, leveraging recent innovations in Bayesian optimization. Specifically, we consider a wide range of feature selection techniques (combining 3 search and 8 evaluator methods) and all classification approaches implemented in WEKA, spanning 2 ensemble methods, 10 meta-methods, 27 base classifiers, and hyperparameter settings for each classifier. On each of 21 popular datasets from the UCI repository, the KDD Cup 09, variants of the MNIST dataset and CIFAR-10, we show classification performance often much better than using standard selection/hyperparameter optimization methods. We hope that our approach will help non-expert users to more effectively identify machine learning algorithms and hyperparameter settings appropriate to their applications, and hence to achieve improved performance.

Auto-WEKA: Combined Selection and Hyperparameter Optimization of Classification Algorithms

Particle swarm optimization (PSO). A tutorial

WEKA is a widely used, open-source machine learning platform. Due to its intuitive interface, it is particularly popular with novice users. However, such users often find it hard to identify the best approach for their particular dataset among the many available. We describe the new version of Auto-WEKA, a system designed to help such users by automatically searching through the joint space of WEKA's learning algorithms and their respective hyperparameter settings to maximize performance, using a state-of-the-art Bayesian optimization method. Our new package is tightly integrated with WEKA, making it just as accessible to end users as any other learning algorithm.

/pdf/auto-weka-2-0-automatic-model-selection-and-hyperparameter-2q2ndjgqex.pdf

Auto-WEKA 2.0: automatic model selection and hyperparameter optimization in WEKA

Classification of EMG signals using PSO optimized SVM for diagnosis of neuromuscular disorders

Focused on a variation of the euclidean traveling salesman problem (TSP), namely, the generalized traveling salesman problem (GTSP), this paper extends the ant colony optimization method from TSP to this field. By considering the group influence, an improved method is further improved. To avoid locking into local minima, a mutation process and a local searching technique are also introduced into this method. Numerical results show that the proposed method can deal with the GTSP problems fairly well, and the developed mutation process and local search technique are effective.

An ant colony optimization method for generalized TSP problem

A novel LS-SVMs hyper-parameter selection based on particle swarm optimization

Generalized chromosome genetic algorithm (GCGA) was proposed for solving generalized traveling salesman problems (GTSP) as reported in the authors' earlier work. Theoretically, the GCGA could also be used to solve the classical traveling salesman problem (CTSP), which has not been reported by others. In this paper, the generalized chromosome characteristics are analyzed and the feasibility for consistently solving the GTSP and CTSP is verified. Numerical experiments show the advantages of the GCGA for solving a largescale CTSP. (c) 2008 National Natural Science Foundation of China and Chinese Academy of Sciences. Published by Elsevier Limited and Science in China Press. All rights reserved.

Solving traveling salesman problems using generalized chromosome genetic algorithm

Full-text indices are data structures that can be used to find any substring of a given string. Many full-text indices require space larger than the original string. In this paper, we introduce the canonical Huffman code to the wavelet tree of a string T [1… n ]. Compared with Huffman code based wavelet tree, the memory space used to represent the shape of wavelet tree is not needed. In case of large alphabet, this part of memory is not negligible. The operations of wavelet tree are also simpler and more efficient due to the canonical Huffman code. Based on the resulting structure, the multi-key rank and select functions can be performed using at most nH 0  + ∣ Σ ∣(lg lg  n  + lg  n  − lg ∣ Σ ∣)+ O ( nH 0 ) bits and in O ( H 0 ) time for average cases, where H 0 is the zeroth order empirical entropy of T . In the end, we present an efficient construction algorithm for this index, which is on-line and linear.

Jinhui Yang

Papers

An ant colony optimization method for generalized TSP problem

A novel LS-SVMs hyper-parameter selection based on particle swarm optimization

Solving traveling salesman problems using generalized chromosome genetic algorithm

Canonical Huffman code based full-text index