Advances in artificial intelligence, sensors and big data management have far-reaching societal impacts. As these systems augment our everyday lives, it becomes increasing-ly important for people to understand them and remain in control. We investigate how HCI researchers can help to develop accountable systems by performing a literature analysis of 289 core papers on explanations and explaina-ble systems, as well as 12,412 citing papers. Using topic modeling, co-occurrence and network analysis, we mapped the research space from diverse domains, such as algorith-mic accountability, interpretable machine learning, context-awareness, cognitive psychology, and software learnability. We reveal fading and burgeoning trends in explainable systems, and identify domains that are closely connected or mostly isolated. The time is ripe for the HCI community to ensure that the powerful new autonomous systems have intelligible interfaces built-in. From our results, we propose several implications and directions for future research to-wards this goal.

https://dl.acm.org/doi/pdf/10.1145/3173574.3174156

Trends and Trajectories for Explainable, Accountable and Intelligible Systems: An HCI Research Agenda

In the last decade, cognitive science underwent a change of paradigm by bringing human movement into the focus of research. Concepts such as ‘embodiment’ and ‘enactive’ have been proposed as core concepts reflecting the role of the human body in complex processes such as action and perception, and the interaction of mind and physical environment (Varela et al., 1991; Noë, 2004). In music research, human movement has often been related with the notion of gesture. The reason is that many musical activities (performance, conducting, dancing) involve body movements that evoke meanings, and therefore these movements are called gestures. In Camurri et al. (2005), musical gestures are addressed from the viewpoint of their expressive character. However, there are many ways in which music-related body movements can be approached, measured, described and applied. Accordingly, there are many ways in which musical gestures are meaningful. Given the different contexts in which gestures appear, and their close relationship to movement and meaning, one may be tempted to say that the notion of gesture is too broad, ill-defined and perhaps too vague. Yet the use of this notion is very convenient in modern music research, because it allows making a bridge between movement and meaning. A closer look at the term gesture reveals its potential as a core notion that provides access to central issues in action/perception processes and in mind/environment interactions.

Musical Gestures: concepts and methods in research

Body movement is integral to both performance and perception of music, and this dissertation suggests that we also think about music as movement. Based on ideas of embodied music cognition, it is argued that ecological knowledge of action-sound couplings guide our experience of music, both in perception and performance. Then follows a taxonomy of music-related body movements, before various observation studies of perceiver’s music-movement correspondences are presented: air instrument performance, free dance to music, and sound-tracing. These studies showed that both novices and experts alike seem to associate various types of body movement with features in the musical sound. Knowledge from the observation studies was used in the exploration of artificial actionsound relationships through the development of various prototype music controllers, including the Cheapstick, music balls, and the Music Troll. This exploration showed that it is possible to create low-cost and human-friendly music controllers that may be both intuitive and creatively interesting. The last part of the dissertation presents tools and methods that have been developed throughout the project, including the Musical Gestures Toolbox for the graphical programming environment Max/MSP/Jitter; techniques for creating motion history images and motiongrams of video material; and development of the Gesture Description Interchange Format (GDIF) for streaming and storing musicrelated movement data. These tools may be seen as an answer to many of the research questions posed in the dissertation, and have facilitated the analysis of music-related movement and creation of artificial action-sound relationships in the project.

Action-sound : developing methods and tools to study music-related body movement

Proceedings of the International Conference on New Interfaces for Musical Expression

At the outset of a discussion of evaluating digital musical instruments (DMIs)—that is to say, instruments whose sound generators are digital and separable (though not necessarily separate) from their control interfaces (Malloch et al. 2006)—it is reasonable to ask what the term evaluation in this context really means. After all, there may be many perspectives from which to view the effectiveness of the instruments we build. For most performers, performance on an instrument becomes a means of evaluating how well it functions in the context of live music making, and their measure of success is the response of the audience to their performance. Audiences evaluate performances on the basis of how engaged they feel by what they have seen and heard. When questioned, they are likely to describe good performances as “exciting,” “skillful,” “musical.” Bad performances are “boring,” and those which are marred by technical malfunction are often dismissed out of hand. If performance is considered to be a valid means of evaluating a musical instrument, then it follows that, for the field of DMI design, a much broader definition of the term “evaluation” than that typically used in human–computer interaction (HCI) is required to reflect the fact that there are a number of stakeholders involved in the design and evaluation of DMIs. In addition to players and audiences, there are also composers, instrument builders, component manufacturers, and perhaps even customers. And each of these stakeholders may have a different concept of what is meant by “evaluation.” Composers, for example, may evaluate an instrument in terms of how reliable it is. If a composer writes a piece of instrumental music to be performed on a DMI, then they ought to be able to assume that (1) the instrumentalist is skilled on their instrument, and (2) the instrument has a known space of sound attributes that the composer can draw upon for musical effect.

A framework for the evaluation of digital musical instruments

While several researchers have grappled with the problem of comparing musical devices across performance, installation, and related contexts, no methodology yet exists for producing holistic, informative visualizations for these devices. Drawing on existing research in performance interaction, human-computer interaction, and design space analysis, the authors propose a dimension space representation that can be adapted for visually displaying musical devices. This paper illustrates one possible application of the dimension space to existing performance and interaction systems, revealing its usefulness both in exposing patterns across existing musical devices and aiding in the design of new ones.

Towards a dimension space for musical devices

This paper describes the adaptation of an existing model of human information processing for the categorization of digital musical instruments in terms of performance context and behavior. It further presents a visualization intended to aid the analysis of existing DMIs and the design of new devices. Three new interfaces constructed by the authors are examined within this framework to illustrate its utility.

Towards a new conceptual framework for digital musical instruments

This paper describes the T-Stick, a new family of digital musical instruments. It presents the motivation behind the project, hardware and software design, and presents insights gained through collaboration with performers who have collectively practised and performed with the T-Stick for hundreds of hours, and with composers who have written pieces for the instrument in the context of McGill University's Digital Orchestra project. Each of the T-Sticks is based on the same general structure and sensing platform, but each also differs from its siblings in size, weight, timbre and range.

The T-Stick: from musical interface to musical instrument

This paper describes the design and implementation of a framework designed to aid collaborative development of a digital musical instrument mapping layer. The goal was to create a system that allows mapping between controller and sound parameters without requiring a high level of technical knowledge, and which needs minimal manual intervention for tasks such as configuring the network and assigning identifiers to devices. Ease of implementation was also considered, to encourage future developers of devices to adopt a compatible protocol.

System development included the design of a decentralized network for the management of peer-to-peer data connections using OpenSound Control. Example implementations were constructed using several different programming languages and environments. A graphical user interface for dynamically creating, modifying, and destroying mappings between control data streams and synthesis parameters is also presented.

A Network-Based Framework for Collaborative Development and Performance of Digital Musical Instruments

This paper presents the development of methods for gesture control of sound spatialization. It provides a comparison of seven popular software spatialization systems from a control point of view, and examines human-factors issues relevant to gesture control. An effort is made to reconcile these two design- and parameter-spaces, and draw useful conclusions regarding likely successful mapping strategies. Lastly, examples are given using several different gesture-tracking and motion capture systems controlling various parameters of the spatialization system.

Joseph Malloch

Papers

Towards a dimension space for musical devices

Towards a new conceptual framework for digital musical instruments

The T-Stick: from musical interface to musical instrument

A Network-Based Framework for Collaborative Development and Performance of Digital Musical Instruments

Gesture Control of Sound Spatialization for Live Musical Performance