From the Publisher: 
The accessible presentation of this book gives both a general view of the entire computer vision enterprise and also offers sufficient detail to be able to build useful applications. Users learn techniques that have proven to be useful by first-hand experience and a wide range of mathematical methods. A CD-ROM with every copy of the text contains source code for programming practice, color images, and illustrative movies. Comprehensive and up-to-date, this book includes essential topics that either reflect practical significance or are of theoretical importance. Topics are discussed in substantial and increasing depth. Application surveys describe numerous important application areas such as image based rendering and digital libraries. Many important algorithms broken down and illustrated in pseudo code. Appropriate for use by engineers as a comprehensive reference to the computer vision enterprise.

Computer vision : a modern approach = 计算机视觉 : 一种现代的方法

Synchronization:a universal concept in nonlinear science

O VER THE PAST 25 YEARS OUR UNDERSTANDING of how music interacts with the human mind and brain has grown rapidly, and multimedia technologies have augmented the ways in which we engage with music. In Embodied Music Cognition and Mediation Technology, Marc Leman examines how these developments might be unified into something that is simultaneously a theory of music cognition and a blueprint for the music mediation technology of the future. Mediation refers to the mappings between the intentions and desires on the part of active musical participants and the technology that renders the music. The main mediating principle elaborated on in the monograph, which is more intellectual discourse than textbook, is rooted in the belief that musical interactions are socially charged, embodied affairs. Thus, individuals understand music in the same way that they understand others’ intentions during social interaction, and expressive intentions are attributed to music because patterns of sonic energy evoke bodily gestures that are meaningful to an individual due to his or her personal history as an active participant within a cultural environment. The first three chapters contextualize the embodied music cognition approach. In Chapter 1, Leman sets the scene by making clear the challenges that face those who are concerned with how subjective musical experiences are linked to physical sound patterns. Chapter 2 then deals with the diversity of paradigms that are relevant to the business of interdisciplinary music research. Here Leman adeptly identifies relationships between trends in music research, such as the emergence of systematic musicology, and landmark developments in the discipline of psychology, such as the advent of the Gestalt school and cognitivism. He also charts the progress made in the fields of technology, information theory, and computational modeling, with well selected philosophical matter visited along the way. This serves as a historical prelude to the birth of the modern embodied cognition paradigm, which asserts that, “knowledge does not emerge from passive perception, but from the need to act in an environment” (p. 43). In Chapter 3, Leman expands upon this ecological theme with a view toward music mediation technology, developing the premise that mediating technology should exploit the way in which individuals naturally engage themselves with music. The ensuing chapters delve into the details of what embodied music cognition means. Chapters 4 and 5 build a case for why it makes sense to think about engagement with music in terms of corporeal articulations and action-based ontologies, and how these lead to pleasurable experiences with music, whereas Chapters 6 and 7 describe how this type of framework might play itself out in musical instrument and music retrieval technologies. Throughout these chapters, Leman articulates a framework in which performer/music/listener interactions can be structured/mediated. The framework contends with the formidable challenges inherent in mapping between the intentions, actions, and percepts of individuals and very specific musical signals. Ultimately, the problem is one of identifying relationships between semantics and musical structure, and then specifying the technological requirements for accomplishing the translation from one to the other. Leman breaks the problem down into three interacting conceptual levels, which he talks about as first-person, second-person, and third-person descriptions. Thirdperson descriptions are objective representations of the structural features of the music, whereas first-person descriptions are subjectively assigned semantic labels that refer to expressive intentions. According to Leman, previous approaches to understanding music (e.g., traditional musicology) have fixated upon these two levels of description without giving adequate treatment to the “rules” that govern the mapping between objective representations and subjective interpretations. Such rules are needed to achieve his scientific goal of developing a complete theory of music, as well as his practical goal of developing a successful mediation technology. The key to Leman’s solution is the proposal that an understanding of musical intentions requires third-person and first-person descriptions to be linked via second-person descriptions, which are corporeal in nature. At this intermediate level, expressive bodily gestures from an individual’s repertoire of actions are used to describe moving sonic forms in a manner that the individual can interpret based on his or

https://www.jstor.org/stable/info/10.1525/mp.2009.26.3.289

Embodied music cognition and mediation technology

The performance of the Mel-Frequency Cepstrum Coefficients (MFCC) may be affected by (1) the number of filters, (2) the shape of filters, (3) the way in which filters are spaced, and (4) the way in which the power spectrum is warped. In this paper, several comparison experiments are done to find a best implementation. The traditional MFCC calculation excludes the 0th coefficient for the reason that it is regarded as somewhat unreliable. According to the analysis and experiments, the authors find that it can be regarded as the generalized frequency band energy (FBE) and is hence useful, which results in the FBE-MFCC. The authors also propose a better analysis, namely the auto-regressive analysis, on the frame energy,which outperforms its 1st and/or 2nd order differential derivatives. Experiments with the “863”Speech Database show that, compared with the traditional MFCC with its corresponding autoregressive analysis coefficients, the FBE-MFCC and the frame energy with their corresponding auto-regressive analysis coefficients form the best combination, reducing the Chinese syllable error rate (CSER) by about 10%, while the FBE-MFCC with the corresponding auto-regressive analysis coefficients reduces CSER by 2.5%. Comparison experiments are also done with a quite casual Chinese speech database, named Chinese Annotated Spontaneous Speech (CASS)corpus. The FBE-MFCC can reduce the error rate by about 2.9% on an average.

Comparison of Different Implementations of MFCC

This book can be read with interest by physicians, psychologists, and educators. It is not a comprehensive treatise on brain injuries occurring in children nor is it intended to be. It deals rather with \"the particular mental make-up of braininjured children and the resulting need for special educational techniques.\" The work is divided into two sections. The first deals with the psychopathology of the brain-injured child. It includes a brief historical review and preliminary remarks on the anatomy and function of certain aspects of the central nervous system germane to the problem. It discusses in detail the techniques employed and results obtained from studies designed to depict the handicaps of the brain-injured child, and to differentiate these handicaps from those seen in other types of mental retardation. The second section deals with methods of teaching employed to overcome and compensate for these handicaps. All will not agree with the authors' concepts of brain injury. At times too great an emphasis appears to be placed upon the history of a difficult delivery. However, all should agree that this is a stimulating and thought-provoking piece of work in a field which has often been considered hopeless. -D. L. DUNPHY

Psychopathology and Education of the Brain-Injured Child.

Interpersonal musical entrainment—temporal synchronization and coordination between individuals in musical contexts—is a ubiquitous phenomenon related to music’s social functions of promoting group bonding and cohesion. Mechanisms other than sensorimotor synchronization are rarely discussed, while little is known about cultural variability or about how and why entrainment has social effects. In order to close these gaps, we propose a new model that distinguishes between different components of interpersonal entrainment: sensorimotor synchronization—a largely automatic process manifested especially with rhythms based on periodicities in the 100–2000 ms timescale—and coordination, extending over longer timescales and more accessible to conscious control. We review the state of the art in measuring these processes, mostly from the perspective of action production, and in so doing present the first cross-cultural comparisons between interpersonal entrainment in natural musical performances, with an exploratory analysis that identifies factors that may influence interpersonal synchronization in music. Building on this analysis we advance hypotheses regarding the relationship of these features to neurophysiological, social, and cultural processes. We propose a model encompassing both synchronization and coordination processes and the relationship between them, the role of culturally shared knowledge, and of connections between entrainment and social processes.

/pdf/interpersonal-entrainment-in-music-performance-theory-method-3cs3xensmk.pdf

Interpersonal entrainment in music performance : theory, method, and model

In this work we present a framework and an experimental approach to investigate human body movement qualities (i.e., the expressive components of non-verbal communication) in HCI. We first define a candidate movement quality conceptually, with the involvement of experts in the field (e.g., dancers, choreographers). Next, we collect a dataset of performances and we evaluate the perception of the chosen quality. Finally, we propose a computational model to detect the presence of the quality in a movement segment and we compare the outcomes of the model with the evaluation results. In the proposed on-going work, we apply this approach to a specific quality of movement: Fluidity. The proposed methods and models may have several applications, e.g., in emotion detection from full-body movement, interactive training of motor skills, rehabilitation.

/pdf/movement-fluidity-analysis-based-on-performance-and-3fbalddzww.pdf

Movement Fluidity Analysis Based on Performance and Perception

The measurement and tracking of body movement within musical performances can provide valuable sources of data for studying interpersonal interaction and coordination between musicians. The continued development of tools to extract such data from video recordings will offer new opportunities to research musical movement across a diverse range of settings, including field research and other ecological contexts in which the implementation of complex motion capture systems is not feasible or affordable. Such work might also make use of the multitude of video recordings of musical performances that are already available to researchers. The present study made use of such existing data, specifically, three video datasets of ensemble performances from different genres, settings, and instrumentation (a pop piano duo, three jazz duos, and a string quartet). Three different computer vision techniques were applied to these video datasets—frame differencing, optical flow, and kernelized correlation filters (KCF)—with the aim of quantifying and tracking movements of the individual performers. All three computer vision techniques exhibited high correlations with motion capture data collected from the same musical performances, with median correlation (Pearson’s r) values of .75 to .94. The techniques that track movement in two dimensions (optical flow and KCF) provided more accurate measures of movement than a technique that provides a single estimate of overall movement change by frame for each performer (frame differencing). Measurements of performer’s movements were also more accurate when the computer vision techniques were applied to more narrowly-defined regions of interest (head) than when the same techniques were applied to larger regions (entire upper body, above the chest or waist). Some differences in movement tracking accuracy emerged between the three video datasets, which may have been due to instrument-specific motions that resulted in occlusions of the body part of interest (e.g. a violinist’s right hand occluding the head whilst tracking head movement). These results indicate that computer vision techniques can be effective in quantifying body movement from videos of musical performances, while also highlighting constraints that must be dealt with when applying such techniques in ensemble coordination research.

/pdf/extracting-coarse-body-movements-from-video-in-music-2gjgxemr93.pdf

Extracting coarse body movements from video in music performance : a comparison of automated computer vision techniques with motion capture data.

Full-body human movement is characterized by fine-grain expressive qualities that humans are easily capable of exhibiting and recognizing in others’ movement. In sports (e.g., martial arts) and performing arts (e.g., dance), the same sequence of movements can be performed in a wide range of ways characterized by different qualities, often in terms of subtle (spatial and temporal) perturbations of the movement. Even a non-expert observer can distinguish between a top-level and average performance by a dancer or martial artist. The difference is not in the performed movements--the same in both cases--but in the “quality” of their performance.In this article, we present a computational framework aimed at an automated approximate measure of movement quality in full-body physical activities. Starting from motion capture data, the framework computes low-level (e.g., a limb velocity) and high-level (e.g., synchronization between different limbs) movement features. Then, this vector of features is integrated to compute a value aimed at providing a quantitative assessment of movement quality approximating the evaluation that an external expert observer would give of the same sequence of movements.Next, a system representing a concrete implementation of the framework is proposed. Karate is adopted as a testbed. We selected two different katas (i.e., detailed choreographies of movements in karate) characterized by different overall attitudes and expressions (aggressiveness, meditation), and we asked seven athletes, having various levels of experience and age, to perform them. Motion capture data were collected from the performances and were analyzed with the system. The results of the automated analysis were compared with the scores given by 14 karate experts who rated the same performances. Results show that the movement-quality scores computed by the system and the ratings given by the human observers are highly correlated (Pearson’s correlations r = 0.84, p = 0.001 and r = 0.75, p = 0.005).

https://dl.acm.org/doi/pdf/10.1145/3132369

Analysis of Movement Quality in Full-Body Physical Activities

In this paper we present a low-intrusive approach to the detection of expressive full-body movement qualities. We focus on two qualities: Lightness and Fragility and we detect them using the data captured by four wearable devices, two Inertial Movement Units (IMU) and two electromyographs (EMG), placed on the forearms. The work we present in the paper stems from a strict collaboration with expressive movement experts (e.g., contemporary dance choreographers) for defining a vocabulary of basic movement qualities. We recorded 13 dancers performing movements expressing the qualities under investigation. The recordings were next segmented and the perceived level of each quality for each segment was ranked by 5 experts using a 5-points Likert scale. We obtained a dataset of 150 segments of movement expressing Fragility and/or Lightness. In the second part of the paper, we define a set of features on IMU and EMG data and we extract them on the recorded corpus. We finally applied a set of supervised machine learning techniques to classify the segments. The best results for the whole dataset were obtained with a Naive Bayes classifier for Lightness (F-score 0.77), and with a Support Vector Machine classifier for Fragility (F-score 0.77). Our approach can be used in ecological contexts e.g., during artistic performances.

Paolo Alborno

Papers

Interpersonal entrainment in music performance : theory, method, and model

Movement Fluidity Analysis Based on Performance and Perception

Extracting coarse body movements from video in music performance : a comparison of automated computer vision techniques with motion capture data.

Analysis of Movement Quality in Full-Body Physical Activities

Low-intrusive recognition of expressive movement qualities