Physical activity is medicine for older adults

When designing movement sonifications, their effect on people’s movement must be considered. Recent work has shown how real-time sonification can be designed to alter the way people move. However, the mechanisms through which these sonifications alter people’s expectations of their movement is not well explained. This is especially important when considering musical sonifications, to which people bring their own associations and musical expectation, and which can, in turn, alter their perception of the sonification. This paper presents a Movement Expectation Sonification Model, based on theories of motor-feedback and expectation, to explore how musical sonification can impact the way people perceive their movement. Secondly, we present a study that validates the predictions of this model by exploring how harmonic stability within sonification interacts with contextual cues in the environment to impact movement behaviour and perceptions. We show how musical expectancy can be built to either reward or encourage movement, and how such an effect is mediated through the presence of additional cues. This model offers a way for sonification designers to create movement sonifications that not only inform movement but can be used to encourage progress and reward successes.

https://link.springer.com/content/pdf/10.1007/s12193-020-00322-2.pdf

Movement sonification expectancy model: leveraging musical expectancy theory to create movement-altering sonifications

While design theories in visual displays have been well developed and further refined, relatively little research has been conducted on design theories and models in auditory displays. The existing...

https://journals.sagepub.com/doi/pdf/10.1177/1071181319631340

Exploring Design Constructs In Sound Design With A Focus On Perceived Affordance

This paper describes Sonifyd, a sonification driven multimedia and audiovisual environment based on color-sound conversion for real-time manipulation. Sonifyd scans graphics horizontally or vertically from a scan line, generates sound and determines timbre according to its own additive synthesis based color-to-sound mapping. Color and sound relationships are fixed as default, but they can be organic for more tonal flexibility. Within this ecosystem, flexible timbre changes will be discovered by Sonifyd. The scan line is invisible, but Sonifyd provides another display that represents the scanning process in the form of dynamic imagery representation. The primary goal of this project is to be a functioning tool for a new kind of visual music, graphic sonification research and to further provide a synesthetic metaphor for audiences/users in the context of an art installation and audiovisual performance. The later section is a discussion about limitations that I have encountered: using an additive synthesis and frequency modulation technique with the line scanning method. In addition, it discusses potential possibilities for the future direction of development in relation to graphic expression and sound design context.

/pdf/sonifyd-a-graphical-approach-for-sound-synthesis-and-30hpz2wkbm.pdf

Sonifyd: A Graphical Approach for Sound Synthesis and Synesthetic Visual Expression

While engaging in physical activity is important for a healthy lifestyle, low self-efficacy, i.e. one's belief in one's own ability, can prevent engagement. Sound has been used in a variety of ways for physical activity: movement sonification to inform about movement, music to encourage and direct movement, and auditory illusions to adapt people's bodily representation and movement behaviour. However, no approach provides the whole picture when considering low self-efficacy. For example, sonification does not encourage movement past a person's expectation of their ability, music gives no information of one's capabilities, and auditory illusions do not direct changes in movement behaviour in a directed way. This thesis proposes a combined method that leverages the agency felt over sonification, our embodiment of music and movement altering feedback to design \textit{``musical expectancy sonifications''} which incorporate musical expectancy within sonification to alter movement perception and behaviour. This thesis proposes a Movement Sonification Expectation Model (MoSEM), which explores expectation within a movement sonification impact on people's perception of their abilities and the way they move. This MoSEM is then interrogated and developed in four initial control studies that investigate these sonifications for different types of movement as well as how they interact with one's expectation of a given movement. These findings led to an exploration of how the MoSEM can be applied to design sonification to support low-self efficacy in two case study populations: chronic pain rehabilitation, including one control study and one mixed methods study, and general well-being, including one interview study and two control studies. These studies show the impact of musical expectation on people's movement perception and behaviour. The findings from this thesis demonstrate not only how sonifications can be designed to use musical expectancy, but also shows a number of considerations that are needed when designing movement sonifications.

Musical expectancy within movement sonification to overcome low self-efficacy

Performing independent physical exercise is critical to maintain one's good health, but it is specifically hard for people with visual impairments. To address this problem, we have developed a Musical Exercise platform for people with visual impairments so that they can perform exercise in a good form consistently. We designed six different conditions, including blindfolded or visual without audio conditions, and blindfolded or visual with two different types of audio feedback (continuous vs. discrete) conditions. Eighteen sighted participants participated in the experiment, by doing two exercises - squat and wall sit with all six conditions. The results show that Musical Exercise is a usable exercise assistance system without any adverse effect on exercise completion time or perceived workload. Also, the results show that with a specific sound design (i.e., discrete), participants in the blindfolded condition can do exercise as consistently as participants in the non-blindfolded condition. This implies that not all sounds equally work and thus, care is required to refine auditory displays. Potentials and limitations of Musical Exercise and future works are discussed with the results.

/pdf/musical-exercise-for-people-with-visual-impairments-a-hnpdqfxgij.pdf

“Musical Exercise” for people with visual impairments: A preliminary study with the blindfolded

Memorable life events are important to form the present selfimage. Looking back on these memories provides an opportunity to ruminate meaning of life and envision future. Integrating the life-log concept and auditory graphs, we have implemented a mobile application, “LifeMusic”, which helps people reflect their memories by listening to their life event sonifcation that is synchronous to these memories. Reflecting the life events through LifeMusic can relieve users of the present and have them journey to the past moments and thus, they can keep balance of emotions in the present life. In the current paper, we describe the implementation and workflow of LifeMusic and briefly discuss focus group results, improvements, and future works.

/pdf/lifemusic-reflection-of-life-memories-by-data-sonification-4h3t78417f.pdf

Ridwan Ahmed Khan

Papers

“Musical Exercise” for people with visual impairments: A preliminary study with the blindfolded

LifeMusic: Reflection of life memories by data Sonification