Statistical learning theory

About: Statistical learning theory is a research topic. Over the lifetime, 1618 publications have been published within this topic receiving 158033 citations.

Regularizing Portfolio Optimization

Abstract: The optimization of large portfolios displays an inherent instability to estimation error. This poses a fundamental problem, because solutions that are not stable under sample fluctuations may look optimal for a given sample, but are, in effect, very far from optimal with respect to the average risk. In this paper, we approach the problem from the point of view of statistical learning theory. The occurrence of the instability is intimately related to over-fitting which can be avoided using known regularization methods. We show how regularized portfolio optimization with the expected shortfall as a risk measure is related to support vector regression. The budget constraint dictates a modification. We present the resulting optimization problem and discuss the solution. The L2 norm of the weight vector is used as a regularizer, which corresponds to a diversification "pressure". This means that diversification, besides counteracting downward fluctuations in some assets by upward fluctuations in others, is also crucial because it improves the stability of the solution. The approach we provide here allows for the simultaneous treatment of optimization and diversification in one framework that enables the investor to trade-off between the two, depending on the size of the available data set.

Application of Polynomial Neural Networks to Classification of Acoustic Warfare Signals

Abstract: : For both estimation and classification problems, the benefits of using artificial neural networks include inductive learning, rapid computation, and the ability to handle high-order and/or nonlinear processing. Neural networks reduce the need for simplifying assumptions that use a priori statistical models (such as 'additive Gaussian noise') or that neglect nonlinear terms, cross-coupling effects, and high-order dynamics. This report demonstrates the usefulness for acoustic warfare applications of an interdisciplinary approach that applies the rigorous theory and algorithms of statistical learning theory to the field of artificial neural networks. In particular, this approach provides two important results; (1) a generalized way of viewing neural modeling in terms of statistical function estimation, and (2) a constrained minimum- logistic-loss polynomial neural network (PNN) classification algorithm. These classification neural networks train rapidly, provide improved discrimination, and use an information-theoretic approach to limit structural complexity and thus avoid over-fitting training data. The report documents the successful application of these algorithms for the purpose of discriminating among broadband acoustic warfare signals and makes recommendations concerning further improvement of the algorithms. Artificial neural networks, Acoustic warfare, Sonar signal, Estimation, Machine learning, Processing, Classification, Modeling.

An improved LSI dimension reduction algorithm based on vector space modal

Abstract: there is not a good method to find k value when using LSI to reduce dimension of Vector Space Modal. to solve the problem above, an improved LSI dimension reduction algorithm called Dimension Reduction Parameter Produced algorithm (DRPP algorithm), was proposed. it adopts machine learning to find k value,instead of traditional manual set parameter. the k value can be obtained by using fit methods of statistical learning theory to fit the data. Theory analysis and experimental results show that the DRPP algorithm can enhance efficiency of dimension reduction.

Failures of Model-dependent Generalization Bounds for Least-norm Interpolation

Abstract: We consider bounds on the generalization performance of the least-norm linear regressor, in the over-parameterized regime where it can interpolate the data. We describe a sense in which any generalization bound of a type that is commonly proved in statistical learning theory must sometimes be very loose when applied to analyze the least-norm interpolant. In particular, for a variety of natural joint distributions on training examples, any valid generalization bound that depends only on the output of the learning algorithm, the number of training examples, and the confidence parameter, and that satisfies a mild condition (substantially weaker than monotonicity in sample size), must sometimes be very loose -- it can be bounded below by a constant when the true excess risk goes to zero.

Generalized regularized learning

Abstract: A classical algorithm in classification is the support vector machine (SVM) algorithm. Based on Vapnik's statistical learning theory, it tries to find a linear boundary with maximum margin to separate the given data into different classes. In non-separable case, SVM uses a kernel trick to map the data onto a feature space and finds a linear boundary in the new space. Instead of understanding SVM's behavior from Vapnik's theory, our work follows regularized learning viewpoint. In regularized learning, people try to find a solution from a function space which has small empirical error in explaining the input-output relationship for training data, yet keeping the simplicity of the solution. To provide the simplicity, the complexity of the solution is penalized, which involves all features in the function space. An equal penalty, as in standard regularized learning, is reasonable without knowing the significance of individual features. But how about if we have prior knowledge that some features are more important than others? Instead of penalizing all features, we study a generalized regularized learning framework where part of the function space is not penalized, and derive its corresponding solution. Different algorithms are derived from the framework. When the empirical error is defined by a quadratic loss, we have generalized regularized least-squares learning algorithm. When the idea is applied to SVM, we obtain semi-parametric SVM algorithm. Besides, we derive the third algorithm which generalizes the kernel logistic regression algorithm. How to choose non-regularized features? We give some empirical studies. We use dimensionality reduction techniques in text categorization, extract some non-regularized intrinsic features for the high dimensional data, and report improved results. Two generalized algorithms need to solve positive definite linear systems to get the parameters. How to solve a large-scale linear system efficiently? Different from previous work in machine learning where people generally resort to conjugate gradient method, our work proposes to use a domain decomposition approach. New interpretations and improved results are reported accordingly.