Our experience of the world is multimodal - we see objects, hear sounds, feel texture, smell odors, and taste flavors Modality refers to the way in which something happens or is experienced and a research problem is characterized as multimodal when it includes multiple such modalities In order for Artificial Intelligence to make progress in understanding the world around us, it needs to be able to interpret such multimodal signals together Multimodal machine learning aims to build models that can process and relate information from multiple modalities It is a vibrant multi-disciplinary field of increasing importance and with extraordinary potential Instead of focusing on specific multimodal applications, this paper surveys the recent advances in multimodal machine learning itself and presents them in a common taxonomy We go beyond the typical early and late fusion categorization and identify broader challenges that are faced by multimodal machine learning, namely: representation, translation, alignment, fusion, and co-learning This new taxonomy will enable researchers to better understand the state of the field and identify directions for future research

Multimodal Machine Learning: A Survey and Taxonomy

We introduce Deep Canonical Correlation Analysis (DCCA), a method to learn complex nonlinear transformations of two views of data such that the resulting representations are highly linearly correlated. Parameters of both transformations are jointly learned to maximize the (regularized) total correlation. It can be viewed as a nonlinear extension of the linear method canonical correlation analysis (CCA). It is an alternative to the nonparametric method kernel canonical correlation analysis (KCCA) for learning correlated nonlinear transformations. Unlike KCCA, DCCA does not require an inner product, and has the advantages of a parametric method: training time scales well with data size and the training data need not be referenced when computing the representations of unseen instances. In experiments on two real-world datasets, we find that DCCA learns representations with significantly higher correlation than those learned by CCA and KCCA. We also introduce a novel non-saturating sigmoid function based on the cube root that may be useful more generally in feedforward neural networks.

/pdf/deep-canonical-correlation-analysis-1222dun9hc.pdf

Deep Canonical Correlation Analysis

Exploring and identifying structure is even more important for multivariate data than univariate data, given the difficulties in graphically presenting multivariate data and the comparative lack of parametric models to represent it. Unfortunately, such exploration is also inherently more difficult.

Multivariate Density Estimation

Recent developments in techniques for modeling, digitizing and visualizing 3D shapes has led to an explosion in the number of available 3D models on the Internet and in domain-specific databases. This has led to the development of 3D shape retrieval systems that, given a query object, retrieve similar 3D objects. For visualization, 3D shapes are often represented as a surface, in particular polygonal meshes, for example in VRML format. Often these models contain holes, intersecting polygons, are not manifold, and do not enclose a volume unambiguously. On the contrary, 3D volume models, such as solid models produced by CAD systems, or voxels models, enclose a volume properly. This paper surveys the literature on methods for content based 3D retrieval, taking into account the applicability to surface models as well as to volume models. The methods are evaluated with respect to several requirements of content based 3D shape retrieval, such as: (1) shape representation requirements, (2) properties of dissimilarity measures, (3) efficiency, (4) discrimination abilities, (5) ability to perform partial matching, (6) robustness, and (7) necessity of pose normalization. Finally, the advantages and limitations of the several approaches in content based 3D shape retrieval are discussed.

/pdf/a-survey-of-content-based-3d-shape-retrieval-methods-1p1e9hurpn.pdf

A survey of content based 3D shape retrieval methods

Learning Deconvolution Network for Semantic Segmentation

We address the 3D object retrieval problem using density-based shape descriptors. We explore first and second order local surface features and their multivariate combinations in the density estimation framework. We also experiment with descriptor level information fusion. The results, obtained using two different databases, Princeton Shape Benchmark and Sculpteur, show that, boosted with both feature level and descriptor level information fusion, the density-based shape description framework enables effective and efficient 3D object retrieval.

Feature-Level and Descriptor-Level Information Fusion for Density-Based 3D Shape Descriptors

Abstract The kitchen is one of the busiest and messiest hubs of a home, where the hands are usually spoiled with food. In this setting, gestural interaction can offer several advantages: efficient, intuitive, and touch-free orchestration of interactive devices. Yet, research scarcely investigates the user's perspective on gesture-based systems in the kitchen and lacks designs developed through a user-centred process. With the contribution of 234 participants, we conducted formative studies that investigated users' expectations, perceived benefits, and concerns of gesture-based control in the kitchen. These studies guided our conceptual design ‘2HandTouch’, an information display controlled by Hand Specific On-Skin Gestures (HSoS). Then, we evaluated the user experience of this design through a summative user study. We introduced four perceived benefits of HSoS use in the kitchen that has not been visible in such detail. These are i) hygiene, (ii) time management, (iii) imminent control, and (iv) an uninterrupted cooking practice. We also provided broader implications of our work for designing gesture-controlled devices in the kitchen context.

Gestural interaction in the kitchen: Insights into designing an interactive display controlled by hand specific on-skin gestures

Gesture and speech co-exist in time with a tight synchrony, and they are planned and shaped by the emotional state and produced together. In our early studies we have developed joint gesture-speech models and proposed algorithms for speech driven gesture animation. These algorithms mainly based on the Viterbi decoders and can not run in realtime. In this paper we are presenting necessary modifications of these algorithms to run in realtime. In the experimental studies we present the conditions that satisfy realtime factor of 1. We as well demonstrate the realtime and non-realtime speech driven gesture animations for subjective evaluations.

Real-time speech driven gesture animation

We present a multi-camera system for audio-visual analysis of dance figures. The multi-view video of a dancing actor is acquired using 8 synchronized cameras. The motion capture technique of the proposed system is based on 3D tracking of the markers attached to the person's body in the scene. The resulting set of 3D points is then used to extract the body motion features as 3D displacement vectors whereas MFC coefficients serve as the audio features. In the multi-modal analysis phase, we perform Hidden Markov Model (HMM) based unsupervised temporal segmentation of the audio and body motion features such as legs and arms, separately, to determine the recurrent elementary audio and body motion patterns in the first stage. Then in the second stage, we investigate the correlation of body motion patterns with audio patterns that can be used towards estimation and synthesis of realistic audio-driven body animation.

Multicamera audio-visual analysis of dance figures using segmented body model

We present an end-to-end deep learning model for robot navigation from raw visual pixel input and natural text instructions. The proposed model is an LSTM-based sequence-to-sequence neural network architecture with attention, which is trained on instruction-perception data samples collected in a synthetic environment. We conduct experiments on the SAIL dataset which we reconstruct in 3D so as to generate the 2D images associated with the data. Our experiments show that the performance of our model is on a par with state-of-the-art, despite the fact that it learns navigational language with end-to-end training from raw visual data.

/pdf/visually-grounded-language-learning-for-robot-navigation-ijf0ip3tlx.pdf

Yücel Yemez

Papers

Feature-Level and Descriptor-Level Information Fusion for Density-Based 3D Shape Descriptors

Gestural interaction in the kitchen: Insights into designing an interactive display controlled by hand specific on-skin gestures

Real-time speech driven gesture animation

Multicamera audio-visual analysis of dance figures using segmented body model

Visually Grounded Language Learning for Robot Navigation