Creativity: Flow and the Psychology of Discovery and Invention

A wide range of countries decided to go into lockdown to contain the coronavirus disease (COVID-19) pandemic of 2020, a setting separating people and restricting their movements. We investigated how musicians dealt with this sudden restriction in mobility. Responses of 234 people were collected. The majority of respondents (95%) resided in Belgium or the Netherlands. Results indicated a decrease of 79% of live music making in social settings during lockdown compared with before lockdown. In contrast, an increase of 264% was demonstrated for online joint music making. However, results showed that most respondents were largely or even completely unaccustomed with specialized platforms for online joint music making (e.g., JamKazam, Jamulus). Respondents reported to mostly use well-known video-conferencing platforms such as Zoom and Skype when playing together virtually. However, when such video-conferencing platforms were used, they were often not employed for synchronized playing and were generally reported to insufficiently deal with latency issues. Furthermore, respondents depending on music making as their main source of income explored online real-time methods significantly more than those relying on other income sources. Results also demonstrated an increase of 93% in the use of alternative remote joint music-making methods (e.g., recording parts separately and subsequently circulating these digital recordings). All in all, results of this study provide a more in-depth view on joint music making during the first weeks of lockdown induced by the COVID-19 pandemic of 2020, and demonstrate users' perceptions of performance and usability of online real-time platforms as well as alternative methods for musical interaction.

https://biblio.ugent.be/publication/8701684/file/8709754.pdf

Impact of Lockdown Measures on Joint Music Making: Playing Online and Physically Together

Human-centered approaches to machine learning have established theoretical foundations, design principles and interaction techniques to facilitate end-user interaction with machine learning systems. Yet, general-purpose toolkits supporting the design of interactive machine learning systems are still missing, despite their potential to foster reuse, appropriation and collaboration between different stakeholders including developers, machine learning experts, designers and end users. In this paper, we present an architectural model for toolkits dedicated to the design of human interactions with machine learning. The architecture is built upon a modular collection of interactive components that can be composed to build interactive machine learning workflows, using reactive pipelines and composable user interfaces. We introduce Marcelle, a toolkit for the design of human interactions with machine learning that implements this model. We illustrate Marcelle with two implemented case studies: (1) a HCI researcher conducts user studies to understand novice interaction with machine learning, and (2) a machine learning expert and a clinician collaborate to develop a skin cancer diagnosis system. Finally, we discuss our experience with the toolkit, along with its limitation and perspectives.

https://hal.archives-ouvertes.fr/hal-03335115/document

Marcelle: Composing Interactive Machine Learning Workflows and Interfaces

In many contemporary societies, the pervasiveness of technology is constantly expanding. From communication to social networks, digital health and welfare services, every aspect of social life in industrialised societies is being captured by technology with the objective of human enhancement, optimised services or automated management. Among these technologies, Machine Learning (ML) and the broader field of Artificial Intelligence (AI) received considerable attention in the past decades.

https://hal.archives-ouvertes.fr/hal-03046229/document

Artificial Intelligence in Music and Performance: A Subjective Art-Research Inquiry

In this paper, we outline a diffractive practice of machine learning (ML) in the frame of material-centered interaction design. To this aim, we review related work in ML, HCI, design, new interfaces for musical expression, and computational art, and introduce two practice-based studies of music performance and robotic art based on interactive machine learning tools, with the hope of revealing the computational materiality of ML, and the potential of embodiment to craft prototypes of ML that reconfigure conceptual or technical approaches to ML. We derive five interference conditions for such art-based ML prototypes—situational whole, small data, shallow model, learnable algorithm, and somaesthetic behaviour—and describe their widening of design and engineering practices of ML prototyping. Finally, we sketch how a process of intra-active machine learning could complement that of interactive machine learning to take materiality as an entry point for ML design within HCI.

https://hal.archives-ouvertes.fr/hal-03267282/document

Prototyping Machine Learning Through Diffractive Art Practice

Software tools for generating digital sound often present users with high-dimensional, parametric interfaces, that may not facilitate exploration of diverse sound designs. In this article, we propose to investigate artificial agents using deep reinforcement learning to explore parameter spaces in partnership with users for sound design. We describe a series of user-centred studies to probe the creative benefits of these agents and adapting their design to exploration. Preliminary studies observing users’ exploration strategies with parametric interfaces and testing different agent exploration behaviours led to the design of a fully-functioning prototype, called Co-Explorer, that we evaluated in a workshop with professional sound designers. We found that the Co-Explorer enables a novel creative workflow centred on human–machine partnership, which has been positively received by practitioners. We also highlight varied user exploration behaviours throughout partnering with our system. Finally, we frame design guidelines for enabling such co-exploration workflow in creative digital applications.

/pdf/designing-deep-reinforcement-learning-for-human-parameter-1x3j63sdq3.pdf

Designing Deep Reinforcement Learning for Human Parameter Exploration

Musical life became disrupted in 2020 due to the COVID-19 pandemic. Many musicians and venues turned to online alternatives, such as livestreaming. In this study, three livestreamed concerts were organized to examine separate, yet interconnected concepts-agency, presence, and social context-to ascertain which components of livestreamed concerts facilitate social connectedness. Hierarchical Bayesian modeling was conducted on 83 complete responses to examine the effects of the manipulations on feelings of social connectedness with the artist and the audience. Results showed that in concert 1, where half of the participants were allowed to vote for the final song to be played, this option did not result in the experience of more agency. Instead, if their preferred song was played (regardless of voting ability) participants experienced greater connectedness to the artist. In concert 2, participants who attended the concert with virtual reality headsets experienced greater feelings of physical presence, as well as greater feelings of connectedness with the artist, than those that viewed a normal YouTube livestream. In concert 3, attendance through Zoom led to greater experience of social presence, but predicted less connectedness with the artist, compared to a normal YouTube livestream. Crucially, a greater negative impact of COVID-19 (e.g., loneliness) predicted feelings of connectedness with the artist, possibly because participants fulfilled their social needs with this parasocial interaction. Examining data from all concerts suggested that physical presence was a predictor of connectedness with both the artist and the audience, while social presence only predicted connectedness with the audience. Correlational analyses revealed that reductions in loneliness and isolation were associated with feelings of shared agency, physical and social presence, and connectedness to the audience. Overall, the findings suggest that in order to reduce feelings of loneliness and increase connectedness, concert organizers and musicians could tune elements of their livestreams to facilitate feelings of physical and social presence.

/pdf/livestream-experiments-the-role-of-covid-19-agency-presence-2p3h1hmql6.pdf

Livestream Experiments: The Role of COVID-19, Agency, Presence, and Social Context in Facilitating Social Connectedness

Software tools for generating digital sound often present users with high-dimensional, parametric interfaces, that may not facilitate exploration of diverse sound designs. In this paper, we propose to investigate artificial agents using deep reinforcement learning to explore parameter spaces in partnership with users for sound design. We describe a series of user-centred studies to probe the creative benefits of these agents and adapting their design to exploration. Preliminary studies observing users' exploration strategies with parametric interfaces and testing different agent exploration behaviours led to the design of a fully-functioning prototype, called Co-Explorer, that we evaluated in a workshop with professional sound designers. We found that the Co-Explorer enables a novel creative workflow centred on human-machine partnership, which has been positively received by practitioners. We also highlight varied user exploration behaviors throughout partnering with our system. Finally, we frame design guidelines for enabling such co-exploration workflow in creative digital applications.

Virtual reality (VR) brings radical new possibilities to the empirical study of social music cognition and interaction. In the present article, we consider the role of VR as a research tool, based on its potential to create a sense of "social presence": the illusory feeling of being, and socially interacting, inside a virtual environment. This makes VR promising for bridging ecological validity ("research in the wild") and experimental control ("research in the lab") in empirical music research. A critical assumption however is the actual ability of VR to simulate real-life social interactions, either via human-embodied avatars or computer-controlled agents. The mediation of social musical interactions via VR is particularly challenging due to their embodied, complex, and emotionally delicate nature. In this article, we introduce a methodological framework to operationalize social presence by a combination of factors across interrelated layers, relating to the performance output, embodied co-regulation, and subjective experiences. This framework provides the basis for the proposal of a pragmatic approach to determine the level of social presence in virtual musical interactions, by comparing the outcomes across the multiple layers with the outcomes of corresponding real-life musical interactions. We applied and tested this pragmatic approach via a case-study of piano duet performances of the piece Piano Phase composed by Steve Reich. This case-study indicated that a piano duet performed in VR, in which the real-time interaction between pianists is mediated by embodied avatars, might lead to a strong feeling of social presence, as reflected in the measures of performance output, embodied co-regulation, and subjective experience. In contrast, although a piano duet in VR between an actual pianist and a computer-controlled agent led to a relatively successful performance output, it was inadequate in terms of both embodied co-regulation and subjective experience.

https://biblio.ugent.be/publication/8715656/file/8715657.pdf

A Methodological Framework for Assessing Social Presence in Music Interactions in Virtual Reality.

Since sound and music are powerful forces and drivers of human behavior and physiology, we propose the use of sonification to activate healthy breathing patterns in participants to induce relaxation. Sonification is often used in the context of biofeedback as it can represent an informational, non-invasive and real-time stimulus to monitor, motivate or modify human behavior. The first goal of this study is the proposal and evaluation of a distance-based biofeedback system using a tempo- and phase-aligned sonification strategy to adapt breathing patterns and induce states of relaxation. A second goal is the evaluation of several sonification stimuli on 18 participants that were recruited online and of which we analyzed psychometric and behavioral data using, respectively questionnaires and respiration rate and ratio. Sonification stimuli consisted of filtered noise mimicking a breathing sound, nature environmental sounds and a musical phrase. Preliminary results indicated the nature stimulus as most pleasant and as leading to the most prominent decrease of respiration rate. The noise sonification had the most beneficial effect on respiration ratio. While further research is needed to generalize these findings, this study and its methodological underpinnings suggest the potential of the proposed biofeedback system to perform ecologically valid experiments at participants' homes during the COVID-19 pandemic.

https://biblio.ugent.be/publication/8711733/file/8711737.pdf

Bavo Van Kerrebroeck

Papers

Designing Deep Reinforcement Learning for Human Parameter Exploration

Livestream Experiments: The Role of COVID-19, Agency, Presence, and Social Context in Facilitating Social Connectedness

Designing Deep Reinforcement Learning for Human Parameter Exploration

A Methodological Framework for Assessing Social Presence in Music Interactions in Virtual Reality.

A Breathing Sonification System to Reduce Stress During the COVID-19 Pandemic