Exploring and identifying structure is even more important for multivariate data than univariate data, given the difficulties in graphically presenting multivariate data and the comparative lack of parametric models to represent it. Unfortunately, such exploration is also inherently more difficult.

Multivariate Density Estimation

Dynamic intersubject neural synchronization reflects affective responses to sad music.

Media is created by humans for humans to tell stories. There exists a natural and imminent need for creating human-centered media analytics to illuminate the stories being told and to understand their impact on individuals and society at large. An objective understanding of media content has numerous applications for different stakeholders, from creators to decision-/policy-makers to consumers. Advances in multimodal signal processing and machine learning (ML) can enable detailed and nuanced characterization of media content (of who, what, how, where, and why) at scale. They can also aid our understanding of the impact of media on a range of issues, including individual experiences, behavioral, cultural, and societal trends, and commercial outcomes. Modern deep learning models combined with audiovisual signal processing can analyze entertainment media, such as Film & TV content to quantify gender, age, and race representations. This creates awareness in an objective way that was hitherto impossible. On the other hand, text mining and natural language processing allow nuanced understanding of language use and spoken interactions in media, such as News to track patterns and trends across different contexts. Moreover, advances in human sensing have enabled us to directly measure the influence of media on an individual’s physiology (and brain), while social media analysis enables tracking the societal impact of media content on different cross sections of the society. This article reviews representative methodologies and algorithms, tools, and systems advancing human-centered media understanding through ML in the pursuit of developing computational media intelligence.

Computational Media Intelligence: Human-Centered Machine Analysis of Media

Time series modeling is a challenging and demanding problem. In the recent year, deep learning (DL) has attracted huge attention in many fields of research, including time series analysis and forecasting. While the methods of DL are very broad and wide, we aim to review the most recent and impactful deep learning papers in order to provide insights from the notable DL models and evaluation methods on time series problems. Our main objective is to review and analyse the advantages and disadvantages of different models, evaluation methods, future trends and techniques of solving time series problem with DL.

Modeling Time Series Data with Deep Learning: A Review, Analysis, Evaluation and Future Trend

Music has a powerful influence on human experience. In this paper, we investigate how music affects brain activity, physiological response, and human-reported behavior. Using auditory features related to dynamics, timbre, harmony, rhythm, and register, we predicted brain activity in the form of phase synchronizations in bilateral Heschl's gyri and superior temporal gyri; physiological response in the form of galvanic skin response and heart activity; and emotional experience in the form of continuous, subjective descriptions reported by music listeners. We found that using multivariate time series models with attention mechanisms are effective in predicting emotional ratings, while vector-autoregressive models are effective in predicting involuntary human responses. Musical features related to dynamics, register, rhythm, and harmony were found to be particularly helpful in predicting these human reactions. This work adds to our understanding of how music affects multimodal human experience and has applications in affective computing, music emotion recognition, neuroscience, and music information retrieval.

A Multimodal View into Music's Effect on Human Neural, Physiological, and Emotional Experience

Film music varies tremendously across genre in order to bring about different responses in an audience. For instance, composers may evoke passion in a romantic scene with lush string passages or inspire fear throughout horror films with inharmonious drones. This study investigates such phenomena through a quantitative evaluation of music that is associated with different film genres. We construct supervised neural network models with various pooling mechanisms to predict a film's genre from its soundtrack. We use these models to compare handcrafted music information retrieval (MIR) features against VGGish audio embedding features, finding similar performance with the top-performing architectures. We examine the best-performing MIR feature model through permutation feature importance (PFI), determining that mel-frequency cepstral coefficient (MFCC) and tonal features are most indicative of musical differences between genres. We investigate the interaction between musical and visual features with a cross-modal analysis, and do not find compelling evidence that music characteristic of a certain genre implies low-level visual features associated with that genre. Furthermore, we provide software code to replicate this study at https://github.com/usc-sail/mica-music-in-media. This work adds to our understanding of music's use in multi-modal contexts and offers the potential for future inquiry into human affective experiences.

/pdf/a-computational-lens-into-how-music-characterizes-genre-in-4vk82u51xg.pdf

A computational lens into how music characterizes genre in film.

Music has a powerful influence on a listener’s emotions. In this paper, we represent lyrics and chords in a shared vector space using a phrase-aligned chord-and-lyrics corpus. We show that models that use these shared representations predict a listener’s emotion while hearing musical passages better than models that do not use these representations. Additionally, we conduct a visual analysis of these learnt shared vector representations and explain how they support existing theories in music. This work adds to our understanding of how lyrics and chords interact with one another in music and bears applications in music emotion recognition tasks and music information retrieval.

Learning Shared Vector Representations of Lyrics and Chords in Music

Whether in a happy mood or a sad mood, humans enjoy listening to music. In this paper, we introduce a novel method to identify auditory features that best predict listener-reported enjoyment ratings by splitting the features into qualitative feature groups, then training predictive models on these feature groups and comparing prediction performance. Using audio features that relate to dynamics, timbre, harmony, and rhythm, we predicted continuous enjoyment ratings for a set of happy and sad songs. We found that a distributed lag model with Ll regularization best predicted these responses and that timbre-related features were most relevant for predicting enjoyment ratings in happy music, while harmony-related features were most relevant to predicting enjoyment ratings in sad music. This work adds to our understanding of how music influences affective human experience.

Predicting Human-Reported Enjoyment Responses in Happy and Sad Music

Given the ever-increasing volume of music created and released every day, it has never been more important to study automatic music tagging. In this paper, we present an ensemble-based convolutional neural network (CNN) model trained using various loss functions for tagging musical genres from audio. We investigate the effect of different loss functions and resampling strategies on prediction performance, finding that using focal loss improves overall performance on the the MTG-Jamendo dataset: an imbalanced, multi-label dataset with over 18,000 songs in the public domain, containing 57 labels. Additionally, we report results from varying the receptive field on our base classifier—a CNN-based architecture trained using Mel spectrograms—which also results in a model performance boost and state-of-the-art performance on the Jamendo dataset. We conclude that the choice of the loss function is paramount for improving on existing methods in music tagging, particularly in the presence of class imbalance.

Benjamin Ma

Papers

A Multimodal View into Music's Effect on Human Neural, Physiological, and Emotional Experience

A computational lens into how music characterizes genre in film.

Learning Shared Vector Representations of Lyrics and Chords in Music

Predicting Human-Reported Enjoyment Responses in Happy and Sad Music

Loss Function Approaches for Multi-label Music Tagging