Many statistical learning problems in NLP call for local model search methods. But accuracy tends to suffer with current techniques, which often explore either too narrowly or too broadly: hill-climbers can get stuck in local optima, whereas samplers may be inefficient. We propose to arrange individual local optimizers into organized networks. Our building blocks are operators of two types: (i) transform, which suggests new places to search, via non-random restarts from already-found local optima; and (ii) join, which merges candidate solutions to find better optima. Experiments on grammar induction show that pursuing different transforms (e.g., discarding parts of a learned model or ignoring portions of training data) results in improvements. Groups of locally-optimal solutions can be further perturbed jointly, by constructing mixtures. Using these tools, we designed several modular dependency grammar induction networks of increasing complexity. Our complete system achieves 48.6% accuracy (directed dependency macro-average over all 19 languages in the 2006/7 CoNLL data) — more than 5% higher than the previous state-of-the-art.

/pdf/breaking-out-of-local-optima-with-count-transforms-and-model-1ig9l6eo16.pdf

Breaking Out of Local Optima with Count Transforms and Model Recombination: A Study in Grammar Induction

Clustering is the process of classifying objects into different groups by partitioning sets of data into a series of subsets called clusters. Clustering has taken its roots from algorithms like k-medoids and k-medoids. However conventional k-medoids clustering algorithm suffers from many limitations. Firstly, it needs to have prior knowledge about the number of cluster parameter k. Secondly, it also initially needs to make random selection of k representative objects and if these initial k medoids are not selected properly then natural cluster may not be obtained. Thirdly, it is also sensitive to the order of input dataset. Mining knowledge from large amounts of spatial data is known as spatial data mining. It becomes a highly demanding field because huge amounts of spatial data have been collected in various applications ranging from geo-spatial data to bio-medical knowledge. The database can be clustered in many ways depending on the clustering algorithm employed, parameter settings used, and other factors. Multiple clustering can be combined so that the final partitioning of data provides better clustering. In this paper, an efficient density based k-medoids clustering algorithm has been proposed to overcome the drawbacks of DBSCAN and kmedoids clustering algorithms. The result will be an improved version of kmedoids clustering algorithm. This algorithm will perform better than DBSCAN while handling clusters of circularly distributed data points and slightly overlapped clusters.

/pdf/an-efficient-density-based-improved-k-medoids-clustering-4mty18zh6c.pdf

An Efficient Density based Improved K- Medoids Clustering algorithm

This paper attempts to overcome the tendency of the expectation-maximization (EM) algorithm to locate a local rather than global maximum when applied to estimate the hidden Markov model (HMM) parameters in speech signal modeling. We propose a hybrid algorithm for estimation of the HMM in automatic speech recognition (ASR) using a constraint-based evolutionary algorithm (EA) and EM, the CEL-EM. The novelty of our hybrid algorithm (CEL-EM) is that it is applicable for estimation of the constraint-based models with many constraints and large numbers of parameters (which use EM) like HMM. Two constraint-based versions of the CEL-EM with different fusion strategies have been proposed using a constraint-based EA and the EM for better estimation of HMM in ASR. The first one uses a traditional constraint-handling mechanism of EA. The other version transforms a constrained optimization problem into an unconstrained problem using Lagrange multipliers. Fusion strategies for the CEL-EM use a staged-fusion approach where EM has been plugged with the EA periodically after the execution of EA for a specific period of time to maintain the global sampling capabilities of EA in the hybrid algorithm. A variable initialization approach (VIA) has been proposed using a variable segmentation to provide a better initialization for EA in the CEL-EM. Experimental results on the TIMIT speech corpus show that CEL-EM obtains higher recognition accuracies than the traditional EM algorithm as well as a top-standard EM (VIA-EM, constructed by applying the VIA to EM).

/pdf/a-constraint-based-evolutionary-learning-approach-to-the-1v1gnqy23c.pdf

A Constraint-Based Evolutionary Learning Approach to the Expectation Maximization for Optimal Estimation of the Hidden Markov Model for Speech Signal Modeling

In this chapter, we present the convergence analysis and applications of particle swarm optimization algorithm. Although it is difficult to analyze the convergence of this algorithm, we discuss its convergence based on its iterated function system and probabilistic theory. The dynamic trajectory of the particle is described based on single individual. We also attempt to theoretically prove that the swarm algorithm converges with a probability of 1 toward the global optimal. We apply the algorithms to solve the scheduling problem and peer-to-peer neighbor selection problem. This chapter is also concerned to employ the nature-inspired optimization methods in machine learning. We introduce the swarm algorithm to reoptimize hidden Markov models.

Particle Swarm Algorithm: Convergence and Applications

In this paper, a novel data-driven approach to monitoring of systems operating under variable operating conditions is described. The method is based on characterizing the degradation process via a set of operation-specific hidden Markov models (HMMs), whose hidden states represent the unobservable degradation states of the monitored system while its observable symbols represent the sensor readings. Using the HMM framework, modeling, identification and monitoring methods are detailed that allow one to identify a HMM of degradation for each operation from mixed-operation data and perform operation-specific monitoring of the system. Using a large data set provided by a major manufacturer, the new methods are applied to a semiconductor manufacturing process running multiple operations in a production environment.

/pdf/degradation-modeling-and-monitoring-of-machines-using-1iuzo894gu.pdf

Degradation modeling and monitoring of machines using operation-specific hidden Markov models

Optimization of HMM by the Tabu Search Algorithm.

In this paper, the concept of previous medoid index is introduced The utilization of memory for efficient medoid search is also presented We propose a hybrid search approach for the problem of nearest neighbor search The hybrid search approach combines the previous medoid index, the utilization of memory, the criterion of triangular inequality elimination and the partial distance search The proposed hybrid search approach is applied to the k-medoids-based algorithms Experimental results based on Gauss-Markov source, curve data set and elliptic clusters demonstrate that the proposed algorithm applied to the CLARANS algorithm may reduce the number of distance calculations from 884% to 952% with the same average distance per object compared with CLARANS The proposed hybrid search approach can also be applied to nearest neighbor searching and the other clustering algorithms

/pdf/efficient-search-approaches-for-k-medoids-based-algorithms-2mt9f5dwmy.pdf

Efficient search approaches for k-medoids-based algorithms

In this paper, a simple version of the tabu search algorithm is employed to train a Hidden Markov Model (HMM) to search out the optimal parameter structure of HMM for automatic speech recognition. The proposed TS-HMM training provides a mecha- nism that allows the search process to escape from a local optimum and obtain a near global optimum. Experimental results show that the TS-HMM training has a higher probability of finding the optimal model parameters than traditional algorithms do.

Tsong-Yi Chen

Papers

Optimization of HMM by the Tabu Search Algorithm.

Efficient search approaches for k-medoids-based algorithms

Optimiztion of HMM by the Tabu Search Algorithm