We present the results of two controlled studies of free-improvised ensemble music-making on touch-screens. In our system, updates to an interface of harmonically-selected pitches are broadcast to every touch-screen in response to either a performer pressing a GUI button, or to interventions from an intelligent agent. In our first study, analysis of survey results and performance data indicated significant effects of the button on performer preference, but of the agent on performance length. In the second follow-up study, a mixed-initiative interface, where the presence of the button was interlaced with agent interventions, was developed to leverage both approaches. Comparison of this mixed-initiative interface with the always-on button-plus-agent condition of the first study demonstrated significant preferences for the former. The different approaches were found to shape the creative interactions that take place. Overall, this research offers evidence that an intelligent agent and a networked GUI both improve aspects of improvised ensemble music-making.

Intelligent Agents and Networked Buttons Improve Free-Improvised Ensemble Music-Making on Touch-Screens

A potential path for enabling greater creativity and collaboration is through increased arts and science, technology, engineering, and mathematics (STEM) integration in education and research. This approach has been a growing discussion in US national forums and is the foundation of the science, technology, engineering, and mathematics plus arts and design (STEAM) education movement. Developing authentic artistic integrations with STEM fields (or vice versa) is challenging, particularly in higher education, where traditional disciplinary structures and incentives can impede the creation of integrated programs. Measuring and assessing the outcomes of such integration efforts can be even more challenging, since traditional metrics do not necessarily capture new opportunities created for students and faculty, and the greatest impact may occur over a long period (a career). At Drexel University, we created the Expressive & Creative Interaction Technologies (ExCITe) Center as a standalone institute to pursue and enable such transdisciplinary arts-STEM collaborations, particularly with external arts and education partners. In this perspectives paper, we highlight a range of projects and outcomes resulting from such external collaborations, including graduate research with professional artists, undergraduate student work experiences, and STEAM-based education programs for kindergarten through 12th-grade (K-12) students. While each project has its own specific objectives and outcomes, we believe that they collectively demonstrate this integrated transdisciplinary approach to be impactful and potentially transformative for all levels of learning.

https://www.pnas.org/content/pnas/116/6/1878.full.pdf

Enabling creative collaboration for all levels of learning.

A critical decision when designing glanceable information displays is where to place the content. Since blocking the center of the field of view with virtual information is not desirable, designers often opt for placement in the visual periphery. Another option is to only show virtual content when needed. However, no study has been made to systematically evaluate world-locked content position, considering both cognitive and physiological constraints. With this goal in mind, we designed a scenario that mimics context switching between a real world-task and an information display. We then conducted a within-subjects study to evaluate the effect of position, parameterized by horizontal angle, vertical angle, and distance from the user. Our results show that context switching time increases as the information is displayed far from the task position. The same happens with discomfort: content placed at eye level, or below, was faster and more comfortable than in other positions. We also found participants preferred content at medium distances, although they were also faster with content at far distances.

Where to display? How Interface Position Affects Comfort and Task Switching Time on Glanceable Interfaces

Two major form factors for virtual reality are head-mounted displays and large display environments such as CAVE®and the LCD-based successor CAVE2®. Each of these has distinct advantages and limitations based on how they’re used. This work explores preserving the high resolution and sense of presence of CAVE2 environments in full stereoscopic mode by using a see-though augmented reality HMD to expand the user’s field of regard beyond the physical display walls. In our explorative study, we found that in a visual search task in a stereoscopic CAVE2, the addition of the HoloLens to expand the field of regard did not hinder the performance or accuracy of the participant, but promoted more physical navigation which in post-study interviews participants felt aided in their spatial awareness of the virtual environment.

https://dl.acm.org/doi/pdf/10.1145/3357251.3357579

Extending Virtual Reality Display Wall Environments Using Augmented Reality

This thesis concerns the making and performing of music with new digital musical instruments (DMIs) designed for ensemble performance. While computer music has advanced to the point where a huge variety of digital instruments are common in educational, recreational, and professional music-making, these instruments rarely seek to enhance the ensemble context in which they are used. Interaction models that map individual gestures to sound have been previously studied, but the interactions of ensembles within these models are not well understood. In this research, new ensemble-focussed instruments have been designed and deployed in an ongoing artistic practice. These instruments have also been evaluated to find out whether, and if so how, they affect the ensembles and music that is made with them. Throughout this thesis, six ensemble-focussed DMIs are introduced for mobile touch-screen computers. A series of improvised rehearsals and performances leads to the identification of a vocabulary of continuous performative touch-gestures and a system for tracking these collaborative performances in real time using tools from machine learning. The tracking system is posed as an intelligent agent that can continually analyse the gestural states of performers, and trigger a response in the performers’ user interfaces at appropriate moments. The hypothesis is that the agent interaction and UI response can enhance improvised performances, allowing performers to better explore creative interactions with each other, produce

Apps, Agents, and Improvisation: Ensemble Interaction with Touch-Screen Digital Musical Instruments

This paper presents the design and implementation of a system for simulating mixed reality in setups combining mobile devices and large backdrop displays. With a mixed reality simulator, one can perform usability studies and evaluate mixed reality systems while minimizing confounding variables. This paper describes how mobile device AR design factors can be flexibly and systematically explored without sacrificing the touch and direct unobstructed manipulation of a physical personal MR display. First, we describe general principles to consider when implementing a mixed reality simulator, enumerating design factors. Then, we present our implementation which utilizes personal mobile display devices in conjunction with a large surround-view display environment. Standing in the center of the display, a user may direct a mobile device, such as a tablet or head-mounted display, to a portion of the scene, which affords them a potentially annotated view of the area of interest. The user may employ gesture or touch screen interaction on a simulated augmented camera feed, as they typically would in video-see-through mixed reality applications. We present calibration and system performance results and illustrate our system's flexibility by presenting the design of three usability evaluation scenarios.

/pdf/mixed-reality-simulation-with-physical-mobile-display-rfm2vzylwt.pdf

Mixed reality simulation with physical mobile display devices

We document results from exploring ensemble feedback in loosely-structured electroacoustic improvisations. A conceptual justification for the explorations is provided, in addition to discussion of tools and methodologies. Physical configurations of intra-ensemble feedback networks are documented, along with qualitative analysis of their effectiveness.

Ensemble Feedback Instruments

We discuss live coding audio-visual worlds for large-scale virtual reality environments. We describe Alive, an instrument allowing multiple users to develop sonic and visual behaviors of agents in a virtual world, through a browserbased collaborative code interface, accessible while being immersed through spatialized audio and stereoscopic display. The interface adds terse syntax for query-based precise or stochastic selections and declarative agent manipulations, lazily-evaluated expressions for synthesis and behavior, event handling, and flexible scheduling.

Collaborative Live-Coding with an Immersive Instrument

This research project is based on 32 years of Kuchera-Morin's research and practice in spatio-temporal music composition and media arts. The project is an immersive interactive visual, sonic computational instrument presented as an installation, which includes the development of an open-source computational language, and Kuchera-Morin's immersive interactive visual/sonic composition PROBABLY/POSSIBLY? Using the mathematics of quantum mechanics, the immersive instrument and computational language facilitates the creation of new, unique visual/sonic art forms. This project allows the artist to drive scientific and technological research for creative expression. This same technology is giving physicists insight into higher dimensional representation. The immersive visual/sonic instrument and language is based on the time-dependent Schrodinger equation splitting a hydrogen-like atom's electron in superposition in various orbitals. The immersive media composition, PROBABLY/POSSIBLY? can be interactively performed using our multimodal computational platform and open source language. The instrument/installation can also be used to compose and perform a number of art works based on the time-dependent Schrodinger equation

https://dl.acm.org/doi/pdf/10.1145/3123266.3129324

Tim Wood

Papers

Mixed reality simulation with physical mobile display devices

Ensemble Feedback Instruments

Collaborative Live-Coding with an Immersive Instrument

PROBABLY/POSSIBLY?: An Immersive Interactive Visual/Sonic Quantum Composition and Synthesizer