Our experience of the world is multimodal - we see objects, hear sounds, feel texture, smell odors, and taste flavors Modality refers to the way in which something happens or is experienced and a research problem is characterized as multimodal when it includes multiple such modalities In order for Artificial Intelligence to make progress in understanding the world around us, it needs to be able to interpret such multimodal signals together Multimodal machine learning aims to build models that can process and relate information from multiple modalities It is a vibrant multi-disciplinary field of increasing importance and with extraordinary potential Instead of focusing on specific multimodal applications, this paper surveys the recent advances in multimodal machine learning itself and presents them in a common taxonomy We go beyond the typical early and late fusion categorization and identify broader challenges that are faced by multimodal machine learning, namely: representation, translation, alignment, fusion, and co-learning This new taxonomy will enable researchers to better understand the state of the field and identify directions for future research

Multimodal Machine Learning: A Survey and Taxonomy

We introduce Deep Canonical Correlation Analysis (DCCA), a method to learn complex nonlinear transformations of two views of data such that the resulting representations are highly linearly correlated. Parameters of both transformations are jointly learned to maximize the (regularized) total correlation. It can be viewed as a nonlinear extension of the linear method canonical correlation analysis (CCA). It is an alternative to the nonparametric method kernel canonical correlation analysis (KCCA) for learning correlated nonlinear transformations. Unlike KCCA, DCCA does not require an inner product, and has the advantages of a parametric method: training time scales well with data size and the training data need not be referenced when computing the representations of unseen instances. In experiments on two real-world datasets, we find that DCCA learns representations with significantly higher correlation than those learned by CCA and KCCA. We also introduce a novel non-saturating sigmoid function based on the cube root that may be useful more generally in feedforward neural networks.

/pdf/deep-canonical-correlation-analysis-1222dun9hc.pdf

Deep Canonical Correlation Analysis

Exploring and identifying structure is even more important for multivariate data than univariate data, given the difficulties in graphically presenting multivariate data and the comparative lack of parametric models to represent it. Unfortunately, such exploration is also inherently more difficult.

Multivariate Density Estimation

Recent developments in techniques for modeling, digitizing and visualizing 3D shapes has led to an explosion in the number of available 3D models on the Internet and in domain-specific databases. This has led to the development of 3D shape retrieval systems that, given a query object, retrieve similar 3D objects. For visualization, 3D shapes are often represented as a surface, in particular polygonal meshes, for example in VRML format. Often these models contain holes, intersecting polygons, are not manifold, and do not enclose a volume unambiguously. On the contrary, 3D volume models, such as solid models produced by CAD systems, or voxels models, enclose a volume properly. This paper surveys the literature on methods for content based 3D retrieval, taking into account the applicability to surface models as well as to volume models. The methods are evaluated with respect to several requirements of content based 3D shape retrieval, such as: (1) shape representation requirements, (2) properties of dissimilarity measures, (3) efficiency, (4) discrimination abilities, (5) ability to perform partial matching, (6) robustness, and (7) necessity of pose normalization. Finally, the advantages and limitations of the several approaches in content based 3D shape retrieval are discussed.

/pdf/a-survey-of-content-based-3d-shape-retrieval-methods-1p1e9hurpn.pdf

A survey of content based 3D shape retrieval methods

Learning Deconvolution Network for Semantic Segmentation

We present a multimodal open-set speaker identification system that integrates information coming from audio, face and lip motion modalities. For fusion of multiple modalities, we propose a new adaptive cascade rule that favors reliable modality combinations through a cascade of classifiers. The order of the classifiers in the cascade is adaptively determined based on the reliability of each modality combination. A novel reliability measure, that genuinely fits to the open-set speaker identification problem, is also proposed to assess accept or reject decisions of a classifier. The proposed adaptive rule is more robust in the presence of unreliable modalities, and outperforms the hard-level max rule and soft-level weighted summation rule, provided that the employed reliability measure is effective in assessment of classifier decisions. Experimental results that support this assertion are provided.

Adaptive classifier cascade for multimodal speaker identification.

We target to learn correlation models between music and dance performances to synthesize music driven dance choreographies. The proposed framework learns statistical mappings from mu- sical measures to dance figures using musical measure models, exchangeable figures model, choreography model and dance figure models. Alternative dance choreographies are synthe- sized based on these statistical mappings. Objective and subjec- tive evaluation results demonstrate that the proposed framework successfully synthesize music-driven choreographies.

http://ieeexplore.ieee.org/iel5/5873567/5929571/05929603.pdf

I DANS KOREOGRAF ú I MODEL ú MULTIMODAL DANCE CHOREOGRAPHY MODEL

In this work, we have examined two transform-based methods, namely Discrete Fourier Transform (DFT) and Radial Cosine Transform (RCT), for indexing and retrieval of three dimensional objects. The coefficients of Radial Cosine Transform are rotation-invariant. We have derived rotationinvariant features from Discrete Fourier Transform, and have used these features as the signature of a 3D object. DFT and RCT have been applied to two different voxel representations: First representation is a binary function, where the voxel takes a value as 1 if it intersects with the object surface, and 0 otherwise. Second representation is a function of the distance transform of the object, which is defined on 3D space. 1. Giris

3B Nesnelerin Endekslenmesi ve Erişimi için Dönüşüm Tabanli Yöntemler Transform-based Methods for Indexing and Retrieval of 3D Objects

In the literature of 3D object matching, there exist methods that reduce the shape matching problem to the one of matching shape distributions. A shape distribution is defined as the probability distribution of a shape function computed over the surface of a 3D object. In this study, we extend the existing method in a way to make profit of shape information brought by more than one shape function. We estimate these 3D object descriptors so as we call multivariate shape distributions on a sample 3D object base, using nonparametric approaches, and we compare them using alternative metrics. Precision-recall curves we obtain demonstrate that the method of matching multivariate shape functions may lead to a new 3D object matching paradigm.

https://ieeexplore.ieee.org/iel5/10485/33241/01567716.pdf

3D object matching via multivariate shape distributions

This paper presents a framework for audio-driven human body motion analysis and synthesis. The video is analyzed to capture the time-varying posture of the dancer's body whereas the musical audio signal is processed to extract the beat in- formation. The human body posture is extracted from multi- view video information without any human intervention using a novel marker-based algorithm based on annealing particle fil- tering. Body movements of the dancer are characterized by a set of recurring semantic motion patterns, i.e., dance figures. Each dance figure is modeled in a supervised manner with a set of HMM (Hidden Markov Model) structures and the associated beat frequency. In synthesis, given an audio signal of a learned musical type, the motion parameters of the corresponding dance figures are synthesized via the trained HMM structures in syn- chrony with the input audio signal based on the estimated tempo information. Finally, the generated motion parameters are ani- mated along with the musical audio using a graphical animation tool. Experimental results demonstrate the effectiveness of the proposed framework. 1. Giris ¸

http://ieeexplore.ieee.org/iel5/4620123/4632536/04632725.pdf

Yücel Yemez

Papers

Adaptive classifier cascade for multimodal speaker identification.

I DANS KOREOGRAF ú I MODEL ú MULTIMODAL DANCE CHOREOGRAPHY MODEL

3B Nesnelerin Endekslenmesi ve Erişimi için Dönüşüm Tabanli Yöntemler Transform-based Methods for Indexing and Retrieval of 3D Objects

3D object matching via multivariate shape distributions

orsel Dans Verilerinin Analizi ve Sentezi Analysis and Synthesis of Multiview Audio-Visual Dance Figures