IEEE Transactions on Pattern Analysis and Machine Intelligence

Learning disentangled representations is considered a cornerstone problem in representation learning. Recently, Locatello et al. (2019) demonstrated that unsupervised disentanglement learning without inductive biases is theoretically impossible and that existing inductive biases and unsupervised methods do not allow to consistently learn disentangled representations. However, in many practical settings, one might have access to a limited amount of supervision, for example through manual labeling of (some) factors of variation in a few training examples. In this paper, we investigate the impact of such supervision on state-of-the-art disentanglement methods and perform a large scale study, training over 52000 models under well-defined and reproducible experimental conditions. We observe that a small number of labeled examples (0.01--0.5% of the data set), with potentially imprecise and incomplete labels, is sufficient to perform model selection on state-of-the-art unsupervised models. Further, we investigate the benefit of incorporating supervision into the training process. Overall, we empirically validate that with little and imprecise supervision it is possible to reliably learn disentangled representations.

/pdf/disentangling-factors-of-variations-using-few-labels-4tic5ayd30.pdf

Disentangling Factors of Variations Using Few Labels

Recently there has been a significant interest in learning disentangled representations, as they promise increased interpretability, generalization to unseen scenarios and faster learning on downstream tasks. In this paper, we investigate the usefulness of different notions of disentanglement for improving the fairness of downstream prediction tasks based on representations. We consider the setting where the goal is to predict a target variable based on the learned representation of high-dimensional observations (such as images) that depend on both the target variable and an unobserved sensitive variable. We show that in this setting both the optimal and empirical predictions can be unfair, even if the target variable and the sensitive variable are independent. Analyzing the representations of more than 12600 trained state-of-the-art disentangled models, we observe that several disentanglement scores are consistently correlated with increased fairness, suggesting that disentanglement may be a useful property to encourage fairness when sensitive variables are not observed.

/pdf/on-the-fairness-of-disentangled-representations-404gffobj4.pdf

On the Fairness of Disentangled Representations

A disentangled representation encodes information about the salient factors of variation in the data independently. Although it is often argued that this representational format is useful in learning to solve many real-world down-stream tasks, there is little empirical evidence that supports this claim. In this paper, we conduct a large-scale study that investigates whether disentangled representations are more suitable for abstract reasoning tasks. Using two new tasks similar to Raven's Progressive Matrices, we evaluate the usefulness of the representations learned by 360 state-of-the-art unsupervised disentanglement models. Based on these representations, we train 3600 abstract reasoning models and observe that disentangled representations do in fact lead to better down-stream performance. In particular, they enable quicker learning using fewer samples.

/pdf/are-disentangled-representations-helpful-for-abstract-visual-5a7mw6kok0.pdf

Are Disentangled Representations Helpful for Abstract Visual Reasoning

Learning disentangled representations from visual data, where different high-level generative factors are independently encoded, is of importance for many computer vision tasks. Solving this problem, however, typically requires to explicitly label all the factors of interest in training images. To alleviate the annotation cost, we introduce a learning setting which we refer to as "reference-based disentangling". Given a pool of unlabeled images, the goal is to learn a representation where a set of target factors are disentangled from others. The only supervision comes from an auxiliary "reference set" containing images where the factors of interest are constant. In order to address this problem, we propose reference-based variational autoencoders, a novel deep generative model designed to exploit the weak-supervision provided by the reference set. By addressing tasks such as feature learning, conditional image generation or attribute transfer, we validate the ability of the proposed model to learn disentangled representations from this minimal form of supervision.

Learning Disentangled Representations with Reference-Based Variational Autoencoders

In recent years, affective computing and its applications have become a fast-growing research topic Furthermore, the rise of deep learning has introduced significant improvements in the emotion recognition system compared to classical methods In this work, we propose a multi-modal emotion recognition model based on deep learning techniques using the combination of peripheral physiological signals and facial expressions Moreover, we present an improvement to proposed models by introducing latent features extracted from our internal Bio Auto-Encoder (BAE) Both models are trained and evaluated on AMIGOS datasets reporting valence, arousal, and emotion state classification Finally, to demonstrate a possible medical application in affective computing using deep learning techniques, we applied the proposed method to the assessment of anxiety therapy To this purpose, a reduced multimodal database has been collected by recording facial expressions and peripheral signals such as electrocardiogram (ECG) and galvanic skin response (GSR) of each patient Valence and arousal estimates were extracted using our proposed model across the duration of the therapy, with successful evaluation to the different emotional changes in the temporal domain

/pdf/end-to-end-facial-and-physiological-model-for-affective-41vsda4rdv.pdf

End-to-end Facial and Physiological Model for Affective Computing and Applications

Human facial tracking is an important task in computer vision, which has recently lost pace compared to other facial analysis tasks. The majority of current available tracker possess two major limitations: their little use of temporal information and the widespread use of handcrafted features, without taking full advantage of the large annotated datasets that have recently become available. In this paper we present a fully end-to-end facial tracking model based on current state of the art deep model architectures that can be effectively trained from the available annotated facial landmark datasets. We build our model from the recently introduced general object tracker Re3, which allows modeling the short and long temporal dependency between frames by means of its internal Long Short Term Memory (LSTM) layers. Facial tracking experiments on the challenging 300-VW dataset show that our model can produce state of the art accuracy and far lower failure rates than competing approaches. We specifically compare the performance of our approach modified to work in tracking-by-detection mode and showed that, as such, it can produce results that are comparable to state of the art trackers. However, upon activation of our tracking mechanism, the results improve significantly, confirming the advantage of taking into account temporal dependencies.

/pdf/fully-end-to-end-composite-recurrent-convolution-network-for-4uucf3571a.pdf

Fully End-to-End Composite Recurrent Convolution Network for Deformable Facial Tracking In The Wild

Automatic kinship recognition using Computer Vision, which aims to infer the blood relationship between individuals by only comparing their facial features, has started to gain attention recently The introduction of large kinship datasets, such as Family In The Wild (FIW), has allowed large scale dataset modeling using state of the art deep learning models Among other kinship recognition tasks, family classification task is lacking any significant progress due to its increasing difficulty in relation to the family member size Furthermore, most current state of-the-art approaches do not perform any data pre-processing (which try to improve models accuracy) and are trained without a regularizer (which results in models susceptible to overfitting) In this paper, we present the Deep Family Classifier (DFC), a deep learning model for family classification in the wild We build our model by combining two sub-networks: internal Image Feature Enhancer which operates by removing the image noise and provides an additional facial heatmap layer and Family Class Estimator trained with strong regularizers and a compound loss We observe progressive improvement in accuracy during the validation phase, with a state of the art results of 1689% is obtained for the track 2 in the RFIW2019 challenge and 1708% of familly classification task on FIW dataset

/pdf/heatmap-guided-balanced-deep-convolution-networks-for-family-adbhfa4zjb.pdf

Heatmap-Guided Balanced Deep Convolution Networks for Family Classification in the Wild

In recent years, Affective Computing and its applications have become a fast-growing research topic. Furthermore, the rise of Deep Learning has introduced significant improvements in the emotion recognition system compared to classical methods. In this work, we propose a multi-modal emotion recognition model based on deep learning techniques using the combination of peripheral physiological signals and facial expressions. Moreover, we present an improvement to proposed models by introducing latent features extracted from our internal Bio Auto-Encoder (BAE). Both models are trained and evaluated on AMIGOS datasets reporting valence, arousal, and emotion state classification. Finally, to demonstrate a possible medical application in affective computing using deep learning techniques, we applied the proposed method to the assessment of anxiety therapy. To this purpose, a reduced multi-modal database has been collected by recording facial expressions and peripheral signals such as Electrocardiogram (ECG) and Galvanic Skin Response (GSR) of each patient. Valence and arousal estimation was extracted using the proposed model from the beginning until the end of the therapy, with successful evaluation to the different emotional changes in the temporal domain.

Xavier Binefa

Papers

Learning Disentangled Representations with Reference-Based Variational Autoencoders

End-to-end Facial and Physiological Model for Affective Computing and Applications

Fully End-to-End Composite Recurrent Convolution Network for Deformable Facial Tracking In The Wild

Heatmap-Guided Balanced Deep Convolution Networks for Family Classification in the Wild

End-to-end facial and physiological model for Affective Computing and applications