Machine Learning is the study of methods for programming computers to learn. Computers are applied to a wide range of tasks, and for most of these it is relatively easy for programmers to design and implement the necessary software. However, there are many tasks for which this is difficult or impossible. These can be divided into four general categories. First, there are problems for which there exist no human experts. For example, in modern automated manufacturing facilities, there is a need to predict machine failures before they occur by analyzing sensor readings. Because the machines are new, there are no human experts who can be interviewed by a programmer to provide the knowledge necessary to build a computer system. A machine learning system can study recorded data and subsequent machine failures and learn prediction rules. Second, there are problems where human experts exist, but where they are unable to explain their expertise. This is the case in many perceptual tasks, such as speech recognition, hand-writing recognition, and natural language understanding. Virtually all humans exhibit expert-level abilities on these tasks, but none of them can describe the detailed steps that they follow as they perform them. Fortunately, humans can provide machines with examples of the inputs and correct outputs for these tasks, so machine learning algorithms can learn to map the inputs to the outputs. Third, there are problems where phenomena are changing rapidly. In finance, for example, people would like to predict the future behavior of the stock market, of consumer purchases, or of exchange rates. These behaviors change frequently, so that even if a programmer could construct a good predictive computer program, it would need to be rewritten frequently. A learning program can relieve the programmer of this burden by constantly modifying and tuning a set of learned prediction rules. Fourth, there are applications that need to be customized for each computer user separately. Consider, for example, a program to filter unwanted electronic mail messages. Different users will need different filters. It is unreasonable to expect each user to program his or her own rules, and it is infeasible to provide every user with a software engineer to keep the rules up-to-date. A machine learning system can learn which mail messages the user rejects and maintain the filtering rules automatically. Machine learning addresses many of the same research questions as the fields of statistics, data mining, and psychology, but with differences of emphasis. Statistics focuses on understanding the phenomena that have generated the data, often with the goal of testing different hypotheses about those phenomena. Data mining seeks to find patterns in the data that are understandable by people. Psychological studies of human learning aspire to understand the mechanisms underlying the various learning behaviors exhibited by people (concept learning, skill acquisition, strategy change, etc.).

Machine learning

Probability Distributions.- Linear Models for Regression.- Linear Models for Classification.- Neural Networks.- Kernel Methods.- Sparse Kernel Machines.- Graphical Models.- Mixture Models and EM.- Approximate Inference.- Sampling Methods.- Continuous Latent Variables.- Sequential Data.- Combining Models.

Pattern Recognition and Machine Learning

A survey on deep learning in medical image analysis

Differential evolution (DE) is arguably one of the most powerful stochastic real-parameter optimization algorithms in current use. DE operates through similar computational steps as employed by a standard evolutionary algorithm (EA). However, unlike traditional EAs, the DE-variants perturb the current-generation population members with the scaled differences of randomly selected and distinct population members. Therefore, no separate probability distribution has to be used for generating the offspring. Since its inception in 1995, DE has drawn the attention of many researchers all over the world resulting in a lot of variants of the basic algorithm with improved performance. This paper presents a detailed review of the basic concepts of DE and a survey of its major variants, its application to multiobjective, constrained, large scale, and uncertain optimization problems, and the theoretical studies conducted on DE so far. Also, it provides an overview of the significant engineering applications that have benefited from the powerful nature of DE.

/pdf/differential-evolution-a-survey-of-the-state-of-the-art-3lz840pz2t.pdf

Differential Evolution: A Survey of the State-of-the-Art

Differential evolution (DE) is an efficient and powerful population-based stochastic search technique for solving optimization problems over continuous space, which has been widely applied in many scientific and engineering fields. However, the success of DE in solving a specific problem crucially depends on appropriately choosing trial vector generation strategies and their associated control parameter values. Employing a trial-and-error scheme to search for the most suitable strategy and its associated parameter settings requires high computational costs. Moreover, at different stages of evolution, different strategies coupled with different parameter settings may be required in order to achieve the best performance. In this paper, we propose a self-adaptive DE (SaDE) algorithm, in which both trial vector generation strategies and their associated control parameter values are gradually self-adapted by learning from their previous experiences in generating promising solutions. Consequently, a more suitable generation strategy along with its parameter settings can be determined adaptively to match different phases of the search process/evolution. The performance of the SaDE algorithm is extensively evaluated (using codes available from P. N. Suganthan) on a suite of 26 bound-constrained numerical optimization problems and compares favorably with the conventional DE and several state-of-the-art parameter adaptive DE variants.

Differential Evolution Algorithm With Strategy Adaptation for Global Numerical Optimization

In this paper, we present a new performance improvement technique, window memoization, for software implementations of local image processing algorithms. Window memoization combines the memoization techniques proposed in software and hardware with data redundancy in image processing to improve the performance of local image processing algorithms. It minimizes the number of redundant computations performed on an image by identifying similar neighborhoods of pixels in the image and skipping the computations that are not necessary. This leads to performance improvement in software. We have developed an optimized architecture for window memoization in software and applied it to six image processing algorithms. We have also developed a performance model to predict the speedups obtained by window memoization in software. The typical (average) speedups range from 1.2x to 7.9x while the total average speedup for different algorithms with different input images across different processors is 3.95x.

Window memoization: toward high-performance image processing software

Template matching algorithms determine the best matching position of a reference image (template) on a larger image (scene) in either complete or incomplete information environment. In this work, our main objective is to devise a fuzzy integral-based aggregation scheme in an attempt to get more accurate and robust matching, by combining the matching decisions of a finite number of image template matching algorithms, Particularly, Choquet integrals associated with fuzzy measures can be used for handling fuzziness due to incomplete image information. In the present work, a fuzzy integral-based aggregated template matching system is developed on the basis of Choquet integral using belief, plausibility, and probability measure, while being interpreted as an optimistic, a pessimistic, and a noninteracting aggregation, respectively. Finally, to show a validation of Choquet integral-based template matching methods, three individual template matching methods (i,e., MOAD-matcher, SOAD-matcher, and SOSD-matcher) are combined using Choquet integral with respect to different fuzzy measures. Then, performance of these aggregated matchers is compared to individual matchers' performance. It is found that in a complementary sense a Choquet integral-based aggregation of template matching methods gives a better performance compared to the performance of the individual methods.

Choquet integral-based aggregation of image template matching algorithms

A Content-Based Image Retrieval (CBIR) system which identifies similar medical images based on a query image can assist clinicians for more accurate diagnosis. The recent CBIR research trend favors the construction and use of binary codes to represent images. Deep architectures could learn the non-linear relationship among image pixels adaptively, allowing the automatic learning of high-level features from raw pixels. However, most of them require class labels, which are expensive to obtain, particularly for medical images. The methods which do not need class labels utilize a deep autoencoder for binary hashing, but the code construction involves a specific training algorithm and an ad-hoc regularization technique. In this study, we explored using a deep de-noising autoencoder (DDA), with a new unsupervised training scheme using only backpropagation and dropout, to hash images into binary codes. We conducted experiments on more than 14,000 x-ray images. By using class labels only for evaluating the retrieval results, we constructed a 16-bit DDA and a 512-bit DDA independently. Comparing to other unsupervised methods, we succeeded to obtain the lowest total error by using the 512-bit codes for retrieval via exhaustive search, and speed up 9.27 times with the use of the 16-bit codes while keeping a comparable total error. We found that our new training scheme could reduce the total retrieval error significantly by 21.9%. To further boost the image retrieval performance, we developed Radon Autoencoder Barcode (RABC) which are learned from the Radon projections of images using a de-noising autoencoder. Experimental results demonstrated its superior performance in retrieval when it was combined with DDA binary codes.

Binary codes for tagging x-ray images via deep de-noising autoencoders

Backpropagation is the most popular algorithm for training neural networks. However, this gradient-based training method is known to have a tendency towards very long training times and convergence to local optima. Various methods have been proposed to alleviate these issues including, but not limited to, different training algorithms, automatic architecture design and different transfer functions. In this chapter we continue the exploration into improving gradient-based learning algorithms through dynamic transfer function modification. We propose opposite transfer functions as a means to improve the numerical conditioning of neural networks and extrapolate two backpropagation-based learning algorithms. Our experimental results show an improvement in accuracy and generalization ability on common benchmark functions. The experiments involve examining the sensitivity of the approach to learning parameters, type of transfer function and number of neurons in the network.

https://link.springer.com/content/pdf/10.1007%2F978-3-540-70829-2_12.pdf

Two Frameworks for Improving Gradient-Based Learning Algorithms

A new robust image thresholding technique is introduced in this paper. Comprehensive experiments show that a single thresholding method can not be successful for all kind of images. The proposed approach uses fusion of some well-known thresholding methods by applying weighted voting at the decision level. The main objective is improving robustness of thresholding approach by participating several methods. Although, the proposed approach can not guaranty the best result for all kind of images but it shows higher performance and consistent/smoother behavior in overall. The performance of the new approach and nine well-established thresholding methods are compared by applying to an image set with high image diversity. The comparison results show that the proposed approach outperforms other nine well-established thresholding approaches. The proposed approach has been explained in details and experimental results are provided.

Hamid R. Tizhoosh

Papers

Window memoization: toward high-performance image processing software

Choquet integral-based aggregation of image template matching algorithms

Binary codes for tagging x-ray images via deep de-noising autoencoders

Two Frameworks for Improving Gradient-Based Learning Algorithms

Weighted Voting-Based Robust Image Thresholding