Our experience of the world is multimodal - we see objects, hear sounds, feel texture, smell odors, and taste flavors Modality refers to the way in which something happens or is experienced and a research problem is characterized as multimodal when it includes multiple such modalities In order for Artificial Intelligence to make progress in understanding the world around us, it needs to be able to interpret such multimodal signals together Multimodal machine learning aims to build models that can process and relate information from multiple modalities It is a vibrant multi-disciplinary field of increasing importance and with extraordinary potential Instead of focusing on specific multimodal applications, this paper surveys the recent advances in multimodal machine learning itself and presents them in a common taxonomy We go beyond the typical early and late fusion categorization and identify broader challenges that are faced by multimodal machine learning, namely: representation, translation, alignment, fusion, and co-learning This new taxonomy will enable researchers to better understand the state of the field and identify directions for future research

Multimodal Machine Learning: A Survey and Taxonomy

We introduce Deep Canonical Correlation Analysis (DCCA), a method to learn complex nonlinear transformations of two views of data such that the resulting representations are highly linearly correlated. Parameters of both transformations are jointly learned to maximize the (regularized) total correlation. It can be viewed as a nonlinear extension of the linear method canonical correlation analysis (CCA). It is an alternative to the nonparametric method kernel canonical correlation analysis (KCCA) for learning correlated nonlinear transformations. Unlike KCCA, DCCA does not require an inner product, and has the advantages of a parametric method: training time scales well with data size and the training data need not be referenced when computing the representations of unseen instances. In experiments on two real-world datasets, we find that DCCA learns representations with significantly higher correlation than those learned by CCA and KCCA. We also introduce a novel non-saturating sigmoid function based on the cube root that may be useful more generally in feedforward neural networks.

/pdf/deep-canonical-correlation-analysis-1222dun9hc.pdf

Deep Canonical Correlation Analysis

Exploring and identifying structure is even more important for multivariate data than univariate data, given the difficulties in graphically presenting multivariate data and the comparative lack of parametric models to represent it. Unfortunately, such exploration is also inherently more difficult.

Multivariate Density Estimation

Recent developments in techniques for modeling, digitizing and visualizing 3D shapes has led to an explosion in the number of available 3D models on the Internet and in domain-specific databases. This has led to the development of 3D shape retrieval systems that, given a query object, retrieve similar 3D objects. For visualization, 3D shapes are often represented as a surface, in particular polygonal meshes, for example in VRML format. Often these models contain holes, intersecting polygons, are not manifold, and do not enclose a volume unambiguously. On the contrary, 3D volume models, such as solid models produced by CAD systems, or voxels models, enclose a volume properly. This paper surveys the literature on methods for content based 3D retrieval, taking into account the applicability to surface models as well as to volume models. The methods are evaluated with respect to several requirements of content based 3D shape retrieval, such as: (1) shape representation requirements, (2) properties of dissimilarity measures, (3) efficiency, (4) discrimination abilities, (5) ability to perform partial matching, (6) robustness, and (7) necessity of pose normalization. Finally, the advantages and limitations of the several approaches in content based 3D shape retrieval are discussed.

/pdf/a-survey-of-content-based-3d-shape-retrieval-methods-1p1e9hurpn.pdf

A survey of content based 3D shape retrieval methods

Learning Deconvolution Network for Semantic Segmentation

The problem of directional and multiscale edge detection is considered. Orthogonal and linear-phase M-band wavelet transform is used to decompose the image into channels which correspond to different directions and resolution levels. "Meaningful" combinations of the synthesis filter outputs at the selected channels yield edge maps with desired properties. In addition, with the application of energy operators at the analysis stage, it is possible to obtain a reduction in unwanted zero-crossings. Final edge maps of images are obtained through simple combinations of directional edge maps. >

Use of M-band wavelet transform for multidirectional and multiscale edge detection

We propose a new iterative isometric point correspondence method that relies on diffusion distance to handle challenges posed by commodity depth sensors, which usually provide incomplete and noisy surface data exhibiting holes and gaps. We formulate the correspondence problem as finding an optimal partial mapping between two given point sets, that minimizes deviation from isometry. Our algorithm starts with an initial rough correspondence between keypoints, obtained via a standard descriptor matching technique. This initial correspondence is then pruned and updated by iterating a perfect matching algorithm until convergence to find as many reliable correspondences as possible. For shapes with intrinsic symmetries such as human models, we additionally provide a symmetry aware extension to improve our formulation. The experiments show that our method provides state of the art performance over depth frames exhibiting occlusions, large deformations and topological noise.

/pdf/reliable-isometric-point-correspondence-from-depth-ipyaq8xrkl.pdf

Reliable Isometric Point Correspondence from Depth

View-based indexing schemes for 3D object retrieval are gaining popularity since they provide good retrieval results.
These schemes are coherent with the theory that humans recognize objects based on their 2D appearances. The viewbased
techniques also allow users to search with various queries such as binary images, range images and even 2D
sketches.
The previous view-based techniques use classical 2D shape descriptors such as Fourier invariants, Zernike moments,
Scale Invariant Feature Transform-based local features and 2D Digital Fourier Transform coefficients. These methods
describe each object independent of others. In this work, we explore data driven subspace models, such as Principal
Component Analysis, Independent Component Analysis and Nonnegative Matrix Factorization to describe the shape
information of the views. We treat the depth images obtained from various points of the view sphere as 2D intensity
images and train a subspace to extract the inherent structure of the views within a database. We also show the benefit of
categorizing shapes according to their eigenvalue spread. Both the shape categorization and data-driven feature set
conjectures are tested on the PSB database and compared with the competitor view-based 3D shape retrieval algorithms.

/pdf/view-subspaces-for-indexing-and-retrieval-of-3d-models-1ipfy25s54.pdf

View subspaces for indexing and retrieval of 3D models

An object-oriented codec for video transmission at very low bit rates is proposed. The object motion is modeled by quadratic transform for which the coefficients are estimated by the nonlinear quasi-Newton method. The segmentation problem is put forward as a constrained optimization problem which interacts with the motion estimation process in the course of region growing. A context-based shape coding method which takes into account the image synthesis error as well as the geometric distortion, is also proposed. Quantitative and subjective performance results of the codec on various test sequences are included.

Yücel Yemez

Papers

Use of M-band wavelet transform for multidirectional and multiscale edge detection

Reliable Isometric Point Correspondence from Depth

View subspaces for indexing and retrieval of 3D models

An object-oriented video codec based on region growing motion segmentation

Non-rigid 3D shape tracking from multiview video