This paper is concerned with learning from predictions. Predictions are obtained by ensemble of classifiers such as random forests (RF) or extra-trees. One assumes that estimators are semi independent so that they can be considered as prediction space. Hence we project our feature vector to the space of estimators obtaining responses from each of them. The responses for RFs are conditional class probabilities. The responses might be considered as projections onto some direction in quasi-orthogonal space which are decision trees of a RF. After that one creates the connected Riemannian manifold by computing a matrix of pairwise products of predictions for all trees in the RF. These matrices are symmetric and positive definite which is a necessary and sufficient condition to have a connected Riemannian manifold (R manifold). Because outputs of trees are conditional probabilities we have to create as many such matrices as there are classes. Stacking all these matrices together we obtain a tensor which is passed to Convolutional Neural Networks (CNN) for learning. We tested our algorithm on 11 datasets from UCI repository representing difficult classification problems. The results show very fast learning and convergence of loss and prediction accuracy. The proposed algorithm outperforms feature-based classical classifier ensembles (RFs and extra-trees) for every tested dataset from UCI repository.

Prediction-based classification using learning on Riemannian manifolds

This paper introduces a new topic and research of geometric classifier ensemble learning using two types of objects: classifier prediction pairwise matrix (CPPM) and decision profiles (DPs). Learni...

Ensemble Learning Using Matrices of Classifier Interactions and Decision Profiles on Riemannian and Grassmann Manifolds

This paper considers different deep learning architectures to learn patterns that are objects lying on the Riemannian and Grassmann manifolds. Among them, we considered cascades of classifier ensembles (CCEs), convolutional neural networks (CNNs), and deep neural forests (DNFs). All aforementioned architectures have linearized and nonlinearized versions. Patterns that are objects of Riemannian manifolds are classifier prediction pairwise matrices (CPPMs) while objects of the Grassmann manifolds are obtained using decision profiles (DPs). We also compared our architectures with CCEs that operate in the Euclidean geometry. As seen from the experimental results deep learning architectures based on CNNs provided the best results. 

Analysis of Different Deep Learning Architectures to Learn Generalised Classifier Stacking on Riemannian and Grassmann Manifolds

This paper considers different deep learning architectures to learn patterns that are objects lying on the Riemannian and Grassmann manifolds. Among them, we considered cascades of classifier ensembles (CCEs), convolutional neural networks (CNNs), and deep neural forests (DNFs). All aforementioned architectures have linearized and nonlinearized versions. Patterns that are objects of Riemannian manifolds are classifier prediction pairwise matrices (CPPMs) while objects of the Grassmann manifolds are obtained using decision profiles (DPs). We also compared our architectures with CCEs that operate in the Euclidean geometry. As seen from the experimental results deep learning architectures based on CNNs provided the best results.

This paper is devoted to data decomposition analysis. We decompose data into functional groups depending on their complexity in terms of classification. We use consensus of classifiers as an effective algorithm for data decomposition. Present research considers data decomposition into two subsets of “easy” and “difficult” or “ambiguous” data. The easiest part of data is classified during decomposition using consensus of classifiers. For other part of data one has to apply other classifiers or classifier combination. One can prove experimentally that afore mentioned data decomposition using optimal consensus of classifiers leads to better performance and generalization ability of the entire classification algorithm.

Classification Boosting by Data Decomposition Using Consensus-Based Combination of Classifiers

The objective of this paper is to consider some properties of decisions produced by classifiers that are in consensus. Consensus allows strong classifiers to obtain very reliable classification on the objects on which consensus has been reached. For those ones where consensus is not reached the reclassification procedure should be applied based on other classification algorithms. Properties of different consensuses are described using algebraic approach and performance evaluation routine.

Some Properties of Consensus-Based Classification

In this paper, we briefly present classifier ensembles making use of nonlinear manifolds. Riemannian manifolds have been created using classifier interactions which are presented as symmetric and positive-definite (SPD) matrices. Grassmann manifolds as some particular case of Riemannian manifolds are constructed using decision profiles. Experimental routine shows advantages of Riemannian geometry and nonlinear manifolds for classifier ensemble learning.

Vitaliy Tayanov

Papers

Prediction-based classification using learning on Riemannian manifolds

Classification Boosting by Data Decomposition Using Consensus-Based Combination of Classifiers

Some Properties of Consensus-Based Classification

Manifold-Based Classifier Ensembles