Our experience of the world is multimodal - we see objects, hear sounds, feel texture, smell odors, and taste flavors Modality refers to the way in which something happens or is experienced and a research problem is characterized as multimodal when it includes multiple such modalities In order for Artificial Intelligence to make progress in understanding the world around us, it needs to be able to interpret such multimodal signals together Multimodal machine learning aims to build models that can process and relate information from multiple modalities It is a vibrant multi-disciplinary field of increasing importance and with extraordinary potential Instead of focusing on specific multimodal applications, this paper surveys the recent advances in multimodal machine learning itself and presents them in a common taxonomy We go beyond the typical early and late fusion categorization and identify broader challenges that are faced by multimodal machine learning, namely: representation, translation, alignment, fusion, and co-learning This new taxonomy will enable researchers to better understand the state of the field and identify directions for future research

Multimodal Machine Learning: A Survey and Taxonomy

We introduce Deep Canonical Correlation Analysis (DCCA), a method to learn complex nonlinear transformations of two views of data such that the resulting representations are highly linearly correlated. Parameters of both transformations are jointly learned to maximize the (regularized) total correlation. It can be viewed as a nonlinear extension of the linear method canonical correlation analysis (CCA). It is an alternative to the nonparametric method kernel canonical correlation analysis (KCCA) for learning correlated nonlinear transformations. Unlike KCCA, DCCA does not require an inner product, and has the advantages of a parametric method: training time scales well with data size and the training data need not be referenced when computing the representations of unseen instances. In experiments on two real-world datasets, we find that DCCA learns representations with significantly higher correlation than those learned by CCA and KCCA. We also introduce a novel non-saturating sigmoid function based on the cube root that may be useful more generally in feedforward neural networks.

/pdf/deep-canonical-correlation-analysis-1222dun9hc.pdf

Deep Canonical Correlation Analysis

Exploring and identifying structure is even more important for multivariate data than univariate data, given the difficulties in graphically presenting multivariate data and the comparative lack of parametric models to represent it. Unfortunately, such exploration is also inherently more difficult.

Multivariate Density Estimation

Recent developments in techniques for modeling, digitizing and visualizing 3D shapes has led to an explosion in the number of available 3D models on the Internet and in domain-specific databases. This has led to the development of 3D shape retrieval systems that, given a query object, retrieve similar 3D objects. For visualization, 3D shapes are often represented as a surface, in particular polygonal meshes, for example in VRML format. Often these models contain holes, intersecting polygons, are not manifold, and do not enclose a volume unambiguously. On the contrary, 3D volume models, such as solid models produced by CAD systems, or voxels models, enclose a volume properly. This paper surveys the literature on methods for content based 3D retrieval, taking into account the applicability to surface models as well as to volume models. The methods are evaluated with respect to several requirements of content based 3D shape retrieval, such as: (1) shape representation requirements, (2) properties of dissimilarity measures, (3) efficiency, (4) discrimination abilities, (5) ability to perform partial matching, (6) robustness, and (7) necessity of pose normalization. Finally, the advantages and limitations of the several approaches in content based 3D shape retrieval are discussed.

/pdf/a-survey-of-content-based-3d-shape-retrieval-methods-1p1e9hurpn.pdf

A survey of content based 3D shape retrieval methods

Learning Deconvolution Network for Semantic Segmentation

A key aspect of social human-robot interaction is natural non-verbal communication. In this work, we train an agent with batch reinforcement learning to generate nods and smiles as backchannels in order to increase the naturalness of the interaction and to engage humans. We introduce the Sequential Random Deep Q-Network (SRDQN) method to learn a policy for backchannel generation, that explicitly maximizes user engagement. The proposed SRDQN method outperforms the existing vanilla Q-learning methods when evaluated using off-policy policy evaluation techniques. Furthermore, to verify the effectiveness of SRDQN, a human-robot experiment has been designed and conducted with an expressive 3d robot head. The experiment is based on a story-shaping game designed to create an interactive social activity with the robot. The engagement of the participants during the interaction is computed from user's social signals like backchannels, mutual gaze and adjacency pair. The subjective feedback from participants and the engagement values strongly indicate that our framework is a step forward towards the autonomous learning of a socially acceptable backchanneling behavior.

Training Socially Engaging Robots: Modeling Backchannel Behaviors with Batch Reinforcement Learning

In this work, we introduce the score fusion problem for 3D object retrieval. Ongoing research in 3D object retrieval shows that no single descriptor is capable of providing fine grain discrimination required by prospective 3D search engines. We present a fusion algorithm that linearly combines similarity information originating from multiple shape descriptors. We learn the optimal set of weights in the linear combination by minimizing the emprical ranking risk. The algorithm is based on a recently introduced rigorous statistical ranking framework, for which consistency and fast rate of convergence of empirical ranking risk minimizers have been established. We report the results of relevance feedback search on a large 3D object database, the Princeton Shape Benchmark. Experiments show that, under query formulations with user intervention, the proposed score fusion scheme boosts the performance of the 3D retrieval machine significantly.

Statistical score fusion for 3D object retrieval

Next-generationvehicles willundoubtedly featurebiometric personrecognition as partof an effort toimprove the drivingexperience. Today’stechnology preventssuch systems fromoperatingsatisfactorily underadverse conditions.A proposedframework forachieving personrecognitionsuccessfullycombines differentbiometric modalities,borne out in twocase studies.

Multimodal Recognition for Human−Vehicle Interaction

We address the problem of correspondence between 3D isometric shapes. We present an automatic method that finds the optimal correspondence between two given (nearly) isometric shapes by minimizing the amount of deviation from isometry. We optimize the isometry error in two steps. In the first step, the 3D points uniformly sampled from the shape surfaces are transformed into spectral domain based on geodesic affinity, where the isometry errors are minimized in polynomial time by complete bipartite graph matching. The second step of optimization, which is well-initialized by the resulting correspondence of the first step, explicitly minimizes the isometry cost via an iterative greedy algorithm in the original 3D Euclidean space. Our method is put to test using (nearly) isometric pairs of shapes and its performance is measured via ground-truth correspondence information when available.

3D Isometric shape correspondence

We address content-based retrieval of complete 3D object models by a probabilistic generative description of local shape properties. The proposed shape description framework characterizes a 3D object with sampled multivariate probability density functions of its local surface features. This density-based descriptor can be efficiently computed via kernel density estimation (KDE) coupled with fast Gauss transform. The nonparametric KDE technique allows reliable characterization of a diverse set of shapes and yields descriptors which remain relatively insensitive to small shape perturbations and mesh resolution. Density-based characterization also induces a permutation property which can be used to guarantee invariance at the shape matching stage. As proven by extensive retrieval experiments on several 3D databases, our framework provides state-of-the-art discrimination over a broad and heterogeneous set of shape categories. Index Terms—Shape matching, retrieval, surface representations, nonparametric statistics, geometric transformations, invariance, feature evaluation and selection, performance evaluation.

Yücel Yemez

Papers

Training Socially Engaging Robots: Modeling Backchannel Behaviors with Batch Reinforcement Learning

Statistical score fusion for 3D object retrieval

Multimodal Recognition for Human−Vehicle Interaction

3D Isometric shape correspondence

3D Model Retrieval Using Probability