Showing papers in "Multisensory Research in 2019"

Extrinsic Auditory Contributions to Food Perception & Consumer Behaviour: an Interdisciplinary Review.

Abstract: Food product-extrinsic sounds (i.e., those auditory stimuli that are not linked directly to a food or beverage product, or its packaging) have been shown to exert a significant influence over various aspects of food perception and consumer behaviour, often operating outside of conscious awareness. In this review, we summarise the latest evidence concerning the various ways in which what we hear can influence what we taste. According to one line of empirical research, background noise interferes with tasting, due to attentional distraction. A separate body of marketing-relevant research demonstrates that music can be used to bias consumers' food perception, judgments, and purchasing/consumption behaviour in various ways. Some of these effects appear to be driven by the arousal elicited by loud music as well as the entrainment of people's behaviour to the musical beat. However, semantic priming effects linked to the type and style of music are also relevant. Another route by which music influences food perception comes from the observation that our liking/preference for the music that we happen to be listening to carries over to influence our hedonic judgments of what we are tasting. A final route by which hearing influences tasting relates to the emerging field of 'sonic seasoning'. A developing body of research now demonstrates that people often rate tasting experiences differently when listening to soundtracks that have been designed to be (or are chosen because they are) congruent with specific flavour experiences (e.g., when compared to when listening to other soundtracks, or else when tasting in silence). Taken together, such results lead to the growing realization that the crossmodal influences of music and noise on food perception and consumer behaviour may have some important if, as yet, unrecognized implications for public health.

Not Just Another Pint! the Role of Emotion Induced by Music on the Consumer's Tasting Experience

Abstract: We introduce a novel methodology to assess the influence of the emotion induced by listening to music on the consumer's multisensory tasting experience. These crossmodal effects were analyzed when two contrasting music tracks (positive vs negative emotion) were presented to consumers while tasting beer. The results suggest that the emotional reactions triggered by the music influenced specific aspects of the multisensory tasting experience. Participants liked the beer more, and rated it as tasting sweeter, when listening to music associated with positive emotion. The same beer was rated as more bitter, with higher alcohol content, and as having more body, when the participants listened to music associated with negative emotion. Moreover, participants were willing to pay 7-8% more for the beer that was tasted while they listened to positive music. This novel methodology was subsequently replicated with two different styles of beer. These results are discussed along with practical implications concerning the way in which music can add significant value to how a consumer responds to a brand.

On the Relative Nature of (Pitch-Based) Crossmodal Correspondences

Abstract: This review deals with the question of the relative vs absolute nature of crossmodal correspondences, with a specific focus on those correspondences involving the auditory dimension of pitch. Crossmodal correspondences have been defined as the often-surprising crossmodal associations that people experience between features, attributes, or dimensions of experience in different sensory modalities, when either physically present, or else merely imagined. In the literature, crossmodal correspondences have often been contrasted with synaesthesia in that the former are frequently said to be relative phenomena (e.g., it is the higher-pitched of two sounds that is matched with the smaller of two visual stimuli, say, rather than there being a specific one-to-one crossmodal mapping between a particular pitch of sound and size of object). By contrast, in the case of synaesthesia, the idiosyncratic mapping between inducer and concurrent tends to be absolute (e.g., it is a particular sonic inducer that elicits a specific colour concurrent). However, a closer analysis of the literature soon reveals that the distinction between relative and absolute in the case of crossmodal correspondences may not be as clear-cut as some commentators would have us believe. Furthermore, it is important to note that the relative vs absolute question may receive different answers depending on the particular (class of) correspondence under empirical investigation.

A sweet voice: The influence of cross-modal correspondences between taste and vocal pitch on advertising effectiveness

Abstract: We have seen a rapid growth of interest in cross-modal correspondences between sound and taste over recent years. People consistently associate higher-pitched sounds with sweet/sour foods, while lower-pitched sounds tend to be associated with bitter foods. The human voice is key in broadcast advertising, and the role of voice in communication generally is partly characterized by acoustic parameters of pitch. However, it remains unknown whether voice pitch and taste interactively influence consumer behavior. Since consumers prefer congruent sensory information, it is plausible that voice pitch and taste interactively influence consumers' responses to advertising stimuli. Based on the cross-modal correspondence phenomenon, this study aimed to elucidate the role played by voice pitch-taste correspondences in advertising effectiveness. Participants listened to voiceover advertisements (at a higher or lower pitch than the original narrator's voice) for three food products with distinct tastes (sweet, sour, and bitter) and rated their buying intention (an indicator of advertising effectiveness). The results show that the participants were likely to exhibit greater buying intention toward both sweet and sour food when they listened to higher-pitched (vs lower-pitched) voiceover advertisements. The influence of a higher pitch on sweet and sour food preferences was observed in only two of the three studies: studies 1 and 2 for sour food, and studies 2 and 3 for sweet food. These findings emphasize the role that voice pitch-taste correspondence plays in preference formation, and advance the applicability of cross-modal correspondences to business.

Having a Drink with Tchaikovsky: The Crossmodal Influence of Background Music on the Taste of Beverages.

Abstract: Previous research has shown that auditory cues can influence the flavor of food and drink. For instance, wine tastes better when preferred music is played. We have investigated whether a music background can modify judgments of the specific flavor pattern of a beverage, as opposed to mere preference. This was indeed the case. We explored the nature of this crosstalk between auditory and gustatory perception, and hypothesized that the 'flavor' of the background music carries over to the perceived flavor (i.e., descriptive and evaluative aspects) of beverages. First, we collected ratings of the subjective flavor of different music pieces. Then we used a between-subjects design to cross the music backgrounds with taste evaluations of several beverages. Participants tasted four different samples of beverages under two contrasting audio conditions and rated their taste experiences. The emotional flavor of the music had the hypothesized effects on the flavor of the beverages. We also hypothesized that such an effect would be stronger for music novices than for music experts, and weaker for aqueous solutions than for wines. However, neither music expertise nor liquid type produced additional effects. We discuss implications of this audio-gustatory interaction.

Attending to the Chemical Senses.

Abstract: Theorizing around the topic of attention and its role in human information processing largely emerged out of research on the so-called spatial senses: vision, audition, and to a lesser extent, touch. Thus far, the chemical senses have received far less research interest (or should that be attention) from those experimental psychologists and cognitive neuroscientists interested in the topic. Nevertheless, this review highlights the key role that attentional selection also plays in chemosensory information processing and awareness. Indeed, many of the same theoretical approaches/experimental paradigms that were originally developed in the context of the spatial senses, can be (and in some cases already have been) extended to provide a useful framework for thinking about the perception of taste/flavour. Furthermore, a number of those creative individuals interested in modifying the perception of taste/flavour by manipulating product-extrinsic cues (such as, for example, music in the case of sonic seasoning) are increasingly looking to attentional accounts in order to help explain the empirical phenomena that they are starting to uncover. However, separate from its role in explaining sonic seasoning, gaining a better understanding of the role of attentional distraction in modulating our eating/drinking behaviours really ought to be a topic of growing societal concern. This is because distracted diners (e.g., those who eat while watching TV, fiddling with a mobile device or smartphone, or even while driving) consume significantly more than those who mindfully pay attention to the sensations associated with eating and drinking.

Analysing the Impact of Music on the Perception of Red Wine via Temporal Dominance of Sensations.

Abstract: Several studies have examined how music may affect the evaluation of food and drink, but the vast majority have not observed how this interaction unfolds in time. This seems to be quite relevant, since both music and the consumer experience of food/drink are time-varying in nature. In the present study we sought to fix this gap, using Temporal Dominance of Sensations (TDS), a method developed to record the dominant sensory attribute at any given moment in time, to examine the impact of music on the wine taster's perception. More specifically, we assessed how the same red wine might be experienced differently when tasters were exposed to various sonic environments (two pieces of music plus a silent control condition). The results revealed diverse patterns of dominant flavours for each sound condition, with significant differences in flavour dominance in each music condition as compared to the silent control condition. Moreover, musical correspondence analysis revealed that differences in perceived dominance of acidity and bitterness in the wine were correlated in the temporality of the experience, with changes in basic auditory attributes. Potential implications for the role of attention in auditory flavour modification and opportunities for future studies are discussed.

Variations in Food Acceptability with Respect to Pitch, Tempo, and Volume Levels of Background Music.

Abstract: This study aimed to determine whether pitch, tempo, and volume levels of music stimuli affect sensory perception and acceptance of foods. A traditional music piece was arranged into versions at two pitches, two tempos, and two volumes. For each session, chocolate and bell peppers were presented for consumption under three sound conditions: 1) upper or 2) lower level with respect to each of the three music elements, and 3) silence. Over three sessions, participants evaluated flavor intensity, pleasantness of flavor, texture impression, and overall impression of food samples, in addition to the pleasantness and stimulation evoked by the music stimuli. Results showed that lower-pitched and louder music stimuli increased hedonic impressions of foods compared to their respective counterparts and/or the silent condition. While the effects of music element levels on hedonic impressions differed with the type of food consumed, the participants liked the foods more when music stimuli were perceived as more pleasant and stimulating. Flavor was perceived as more intense when participants were more stimulated by the music samples. Although a specific element of music stimuli was manipulated, perceptions of other elements also varied, leading to large variations in the music-evoked pleasantness and stimulation. In conclusion, the findings provide empirical evidence that hedonic impressions of foods may be influenced by emotions evoked by music selections varying in music element levels, but it should be also noted that the influences were food-dependent and not pronounced.

Taste the Bass: Low Frequencies Increase the Perception of Body and Aromatic Intensity in Red Wine.

Abstract: Associations between heaviness and bass/low-pitched sounds reverberate throughout music, philosophy, literature, and language. Given that recent research into the field of cross-modal correspondences has revealed a number of robust relationships between sound and flavour, this exploratory study was designed to investigate the effects of lower frequency sound (10 Hz to 200 Hz) on the perception of the mouthfeel character of palate weight/body. This is supported by an overview of relevant cross-modal studies and cultural production. Wines were the tastants - a New Zealand Pinot Noir and a Spanish Garnacha - which were tasted in silence and with a 100 Hz (bass) and a higher 1000 Hz sine wave tone. Aromatic intensity was included as an additional character given suggestions that pitch may influence the perception of aromas, which might presumably affect the perception of wine body. Intensity of acidity and liking were also evaluated. The results revealed that the Pinot Noir wine was rated as significantly fuller-bodied when tasted with a bass frequency than in silence or with a higher frequency sound. The low frequency stimulus also resulted in the Garnacha wine being rated as significantly more aromatically intense than when tasted in the presence of the higher frequency auditory stimulus. Acidity was rated considerably higher with the higher frequency in both wines by those with high wine familiarity and the Pinot Noir significantly better liked than the Garnacha. Possible reasons as to why the tones used in this study affected perception of the two wines differently are discussed. Practical application of the findings are also proposed.

Multisensory, Multi-Tasking Performance of Older Adults With and Without Subjective Cognitive Decline

Abstract: As the population ages, it is increasingly important to detect non-normative cognitive declines as early as possible. Measures of combined sensory-motor-cognitive functioning may be early markers for identifying individuals who are at increased risk of developing dementia. Further, older adults experiencing subjective cognitive decline (SCD) may have elevated risk of dementia compared to those without SCD. Tasks involving complex, multisensory interactions reflective of everyday challenges may be particularly sensitive to subjectively perceived, pre-clinical declines. In the current study, older adults with and without SCD were asked to simultaneously perform a standing balance task and a listening task under increasingly challenging sensory/cognitive/motor conditions using a dual-task paradigm in a realistic, immersive virtual environment. It was hypothesized that, compared to older adults without SCD, those with SCD would exhibit greater decrements in postural control and listening response accuracy as sensory/motor/cognitive loads increased. However, counter to predictions, older adults with SCD demonstrated greater reductions in postural sway under more challenging dual-task conditions than those without SCD. Across both groups, poorer postural task performance was associated with poorer cognitive function and speech-in-noise thresholds measured with standard baseline tests. Poorer listening task performance was associated with poorer global cognitive function, poorer mobility, and poorer speech-in-noise detection. Overall, the results provide additional support for the growing evidence demonstrating associations between sensory, motor, and cognitive functioning and contribute to an evolving consideration of how best to categorize and characterize SCD in a way that guides strategies for screening, assessment, and intervention.

Temporal Metrics of Multisensory Processing Change in the Elderly.

Abstract: Older adults exhibit greater multisensory response time (RT) facilitation by violating the race model more than young adults; this is commonly interpreted as an enhancement in perception. Older adults typically exhibit wider temporal binding windows (TBWs) and points of subjective simultaneity (PSS) that typically lie farther from true simultaneity as compared to young adults when simultaneity judgment (SJ) and temporal-order judgment (TOJ) tasks are utilized; this is commonly interpreted as an impairment in perception. Here we explore the relation between the three tasks in order to better assess audiovisual multisensory temporal processing in both young and older adults. Our results confirm previous reports showing that audiovisual RT, TBWs and PSSs change with age; however, we show for the first time a significant positive relation between the magnitude of race model violation in young adults as a function of the PSS obtained from the audiovisual TOJ task (r: 0.49, p: 0.007), that is absent in older adults (r: 0.13, p: 0.58). Furthermore, we find no evidence for the relation between race model violation as a function of the PSS obtained from the audiovisual SJ task in both young (r: -0.01, p: 0.94) and older adults (r: 0.1, p: 0.66). Our results confirm previous reports that (i) audiovisual temporal processing changes with age; (ii) distinct processes are likely involved in simultaneity and temporal-order perception; and (iii) common processing between race model violation and temporal-order judgment is impaired in the elderly.

Event-Related Potentials Reveal Evidence for Late Integration of Emotional Prosody and Facial Expression in Dynamic Stimuli: An ERP Study

Abstract: The aim of the present study was to test whether multisensory interactions of emotional signals are modulated by intermodal attention and emotional valence. Faces, voices and bimodal emotionally congruent or incongruent face–voice pairs were randomly presented. The EEG was recorded while participants were instructed to detect sad emotional expressions in either faces or voices while ignoring all stimuli with another emotional expression and sad stimuli of the task irrelevant modality. Participants processed congruent sad face–voice pairs more efficiently than sad stimuli paired with an incongruent emotion and performance was higher in congruent bimodal compared to unimodal trials, irrespective of which modality was task-relevant. Event-related potentials (ERPs) to congruent emotional face–voice pairs started to differ from ERPs to incongruent emotional face–voice pairs at 180 ms after stimulus onset: Irrespectively of which modality was task-relevant, ERPs revealed a more pronounced positivity (180 ms post-stimulus) to emotionally congruent trials compared to emotionally incongruent trials if the angry emotion was presented in the attended modality. A larger negativity to incongruent compared to congruent trials was observed in the time range of 400–550 ms (N400) for all emotions (happy, neutral, angry), irrespectively of whether faces or voices were task relevant. These results suggest an automatic interaction of emotion related information.

Reciprocal Interactions Between Audition and Touch in Flutter Frequency Perception.

Abstract: In both audition and touch, sensory cues comprising repeating events are perceived either as a continuous signal or as a stream of temporally discrete events (flutter), depending on the events' repetition rate. At high repetition rates (>100 Hz), auditory and tactile cues interact reciprocally in pitch processing. The frequency of a cue experienced in one modality systematically biases the perceived frequency of a cue experienced in the other modality. Here, we tested whether audition and touch also interact in the processing of low-frequency stimulation. We also tested whether multisensory interactions occurred if the stimulation in one modality comprised click trains and the stimulation in the other modality comprised amplitude-modulated signals. We found that auditory cues bias touch and tactile cues bias audition on a flutter discrimination task. Even though participants were instructed to attend to a single sensory modality and ignore the other cue, the flutter rate in the attended modality is perceived to be similar to that of the distractor modality. Moreover, we observed similar interaction patterns regardless of stimulus type and whether the same stimulus types were experienced by both senses. Combined with earlier studies, our results suggest that the nervous system extracts and combines temporal rate information from multisensory environmental signals, regardless of stimulus type, in both the low- and high temporal frequency domains. This function likely reflects the importance of temporal frequency as a fundamental feature of our multisensory experience.

Aging Impairs Temporal Sensitivity, but not Perceptual Synchrony, Across Modalities.

Abstract: Encoding the temporal properties of external signals that comprise multimodal events is a major factor guiding everyday experience. However, during the natural aging process, impairments to sensory processing can profoundly affect multimodal temporal perception. Various mechanisms can contribute to temporal perception, and thus it is imperative to understand how each can be affected by age. In the current study, using three different temporal order judgement tasks (unisensory, multisensory, and sensorimotor), we investigated the effects of age on two separate temporal processes: synchronization and integration of multiple signals. These two processes rely on different aspects of temporal information, either the temporal alignment of processed signals or the integration/segregation of signals arising from different modalities, respectively. Results showed that the ability to integrate/segregate multiple signals decreased with age regardless of the task, and that the magnitude of such impairment correlated across tasks, suggesting a widespread mechanism affected by age. In contrast, perceptual synchrony remained stable with age, revealing a distinct intact mechanism. Overall, results from this study suggest that aging has differential effects on temporal processing, and general impairments with aging may impact global temporal sensitivity while context-dependent processes remain unaffected.

Measuring Multisensory Imagery of Wine: the Vividness of Wine Imagery Questionnaire

Abstract: When we imagine objects or events, we often engage in multisensory mental imagery. Yet, investigations of mental imagery have typically focused on only one sensory modality — vision. One reason for this is that the most common tool for the measurement of imagery, the questionnaire, has been restricted to unimodal ratings of the object. We present a new mental imagery questionnaire that measures multisensory imagery. Specifically, the newly developed Vividness of Wine Imagery Questionnaire (VWIQ) measures mental imagery of wine in the visual, olfactory, and gustatory modalities. Wine is an ideal domain to explore multisensory imagery because wine drinking is a multisensory experience, it involves the neglected chemical senses (smell and taste), and provides the opportunity to explore the effect of experience and expertise on imagery (from wine novices to experts). The VWIQ questionnaire showed high internal consistency and reliability, and correlated with other validated measures of imagery. Overall, the VWIQ may serve as a useful tool to explore mental imagery, for researchers, as well as individuals in the wine industry during sommelier training and evaluation of wine professionals.

Still More Than a Feeling: Commentary on Cash et al., "Expectancy Effects in the Autonomous Sensory Meridian Response" and Recommendations for Measurement in Future ASMR Research.

Abstract: Autonomous Sensory Meridian Response (ASMR) - the sensory phenomenon experienced by some people in response to visual and auditory stimuli such as whispering - has attracted substantial public attention but is not yet well understood or well established within the scientific community. Recent research published in PeerJ by Cash, Heisick and Papesh (2018) investigated whether ASMR could be a placebo effect (resulting from expectation) rather than a genuine experience triggered by ASMR-inducing stimuli. In this article, we provide a commentary on Cash et al.'s findings and argue that they provide evidence for (rather than against) the veracity of ASMR. We discuss issues regarding the measurement of ASMR and end by providing some recommendations on how to assess ASMR as both a state and a trait, in the hope of galvanising collaborative research efforts in the emerging field of ASMR.

The Effect of Irrelevant Environmental Noise on the Performance of Visual-to-Auditory Sensory Substitution Devices Used by Blind Adults

Abstract: Visual-to-auditory Sensory Substitution Devices (SSDs) are a family of non-invasive devices for visual rehabilitation aiming at conveying whole-scene visual information through the intact auditory modality. Although proven effective in lab environments, the use of SSDs has yet to be systematically tested in real-life situations. To start filling this gap, in the present work we tested the ability of expert SSD users to filter out irrelevant background noise while focusing on the relevant audio information. Specifically, nine blind expert users of the EyeMusic visual-to-auditory SSD performed a series of identification tasks via SSDs (i.e., shape, color, and conjunction of the two features). Their performance was compared in two separate conditions: silent baseline, and with irrelevant background sounds from real-life situations, using the same stimuli in a pseudo-random balanced design. Although the participants described the background noise as disturbing, no significant performance differences emerged between the two conditions (i.e., noisy; silent) for any of the tasks. In the conjunction task (shape and color) we found a non-significant trend for a disturbing effect of the background noise on performance. These findings suggest that visual-to-auditory SSDs can indeed be successfully used in noisy environments and that users can still focus on relevant auditory information while inhibiting irrelevant sounds. Our findings take a step towards the actual use of SSDs in real-life situations while potentially impacting rehabilitation of sensory deprived individuals.

Differences in Emotion Recognition From Body and Face Cues Between Deaf and Hearing Individuals.

Abstract: Deaf individuals may compensate for the lack of the auditory input by showing enhanced capacities in certain visual tasks. Here we assessed whether this also applies to recognition of emotions expressed by bodily and facial cues. In Experiment 1, we compared deaf participants and hearing controls in a task measuring recognition of the six basic emotions expressed by actors in a series of video-clips in which either the face, the body, or both the face and body were visible. In Experiment 2, we measured the weight of body and face cues in conveying emotional information when intense genuine emotions are expressed, a situation in which face expressions alone may have ambiguous valence. We found that deaf individuals were better at identifying disgust and fear from body cues (Experiment 1) and in integrating face and body cues in case of intense negative genuine emotions (Experiment 2). Our findings support the capacity of deaf individuals to compensate for the lack of the auditory input enhancing perceptual and attentional capacities in the spared modalities, showing that this capacity extends to the affective domain.

Ventriloquist Illusion Produced With Virtual Acoustic Spatial Cues and Asynchronous Audiovisual Stimuli in Both Young and Older Individuals.

Abstract: Ventriloquist illusion, the change in perceived location of an auditory stimulus when a synchronously presented but spatially discordant visual stimulus is added, has been previously shown in young healthy populations to be a robust paradigm that mainly relies on automatic processes. Here, we propose ventriloquist illusion as a potential simple test to assess audiovisual (AV) integration in young and older individuals. We used a modified version of the illusion paradigm that was adaptive, nearly bias-free, relied on binaural stimulus representation using generic head-related transfer functions (HRTFs) instead of multiple loudspeakers, and tested with synchronous and asynchronous presentation of AV stimuli (both tone and speech). The minimum audible angle (MAA), the smallest perceptible difference in angle between two sound sources, was compared with or without the visual stimuli in young and older adults with no or minimal sensory deficits. The illusion effect, measured by means of MAAs implemented with HRTFs, was observed with both synchronous and asynchronous visual stimulus, but only with tone and not speech stimulus. The patterns were similar between young and older individuals, indicating the versatility of the modified ventriloquist illusion paradigm.

Age-Related Effects on Cross-Modal Duration Perception.

Abstract: Reliable duration perception of external events is necessary to coordinate perception with action, precisely discriminate speech, and for other daily functions. Visual duration perception can be heavily influenced by concurrent auditory signals; however, age-related effects on this process have received minimal attention. In the present study, we examined the effect of aging on duration perception by quantifying (1) duration discrimination thresholds, (2) auditory temporal dominance, and (3) visual duration expansion/compression percepts induced by an accompanying auditory stimulus of longer/shorter duration. Duration discrimination thresholds were significantly greater for visual than auditory tasks in both age groups, however there was no effect of age. While the auditory modality retained dominance in duration perception with age, older adults still performed worse than young adults when comparing durations of two target stimuli (e.g., visual) in the presence of distractors from the other modality (e.g., auditory). Finally, both age groups perceived similar visual duration compression, whereas older adults exhibited visual duration expansion over a wider range of auditory durations compared to their younger counterparts. Results are discussed in terms of multisensory integration and possible decision strategies that change with age.

Cross-Sensory Stimuli Modulate Reactions to Aversive Sounds.

Abstract: We propose that cross-sensory stimuli presenting a positive attributable source of an aversive sound can modulate negative reactions to the sound. In Experiment 1, participants rated original video sources (OVS) of eight aversive sounds (e.g., nails scratching a chalkboard) as more aversive than eight positive attributable video sources (PAVS) of those same sounds (e.g., someone playing a flute) when these videos were presented silently. In Experiment 2, new participants were presented with those eight aversive sounds in three blocks. In Blocks 1 and 3, the sounds were presented alone; in Block 2, four of the sounds were randomly presented concurrently with their corresponding OVS videos, and the other four with their corresponding PAVS videos. Participants rated each sound, presented with or without video, on three scales: discomfort, unpleasantness, and bodily sensations. We found the concurrent presentation of videos robustly modulates participants' reactions to the sounds: compared to the sounds alone (Block 1), concurrent presentation of PAVS videos significantly reduced negative reactions to the sounds, and the concurrent presentation of OVS videos significantly increased negative reactions, across all three scales. These effects, however, did not linger into Block 3 when the sounds were presented alone again. Our results provide novel evidence that negative reactions to aversive sounds can be modulated through cross-sensory temporal syncing with a positive attributable video source. Although this research was conducted with a neurotypical population, we argue that our findings have implications for the treatment of misophonia.

Shared or Distinct Attentional Resources? Confounds in Dual Task Designs, Countermeasures, and Guidelines.

Abstract: Human information processing is limited by attentional resources. That is, via attentional mechanisms humans select information that is relevant for their goals, and discard other information. While limitations of attentional processing have been investigated extensively in each sensory modality, there is debate as to whether sensory modalities access shared resources, or if instead distinct resources are dedicated to individual sensory modalities. Research addressing this question has used dual task designs, with two tasks performed either in a single sensory modality or in two separate modalities. The rationale is that, if two tasks performed in separate sensory modalities interfere less or not at all compared to two tasks performed in the same sensory modality, then attentional resources are distinct across the sensory modalities. If task interference is equal regardless of whether tasks are performed in separate sensory modalities or the same sensory modality, then attentional resources are shared across the sensory modalities. Due to their complexity, dual task designs face many methodological difficulties. In the present review, we discuss potential confounds and countermeasures. In particular, we discuss 1) compound interference measures to circumvent problems with participants dividing attention unequally across tasks, 2) staircase procedures to match difficulty levels of tasks and counteracting problems with interpreting results, 3) choosing tasks that continuously engage participants to minimize issues arising from task switching, and 4) reducing motor demands to avoid sources of task interference, which are independent of the involved sensory modalities.

The Effect of Ageing on Optimal Integration of Conflicting and Non-conflicting Visual-haptic Stimuli

Abstract: Multisensory integration typically follows the predictions of a statistically optimal model whereby the contribution of each sensory modality is weighted according to its reliability. Previous research has shown that multisensory integration is affected by ageing, however it is less certain whether older adults follow this statistically optimal model. Additionally, previous studies often present multisensory cues which are conflicting in size, shape or location, yet naturally occurring multisensory cues are usually non-conflicting. Therefore, the mechanisms of integration in older adults might differ depending on whether the multisensory cues are consistent or conflicting. In the current experiment, young ( n = 21) and older ( n = 30) adults were asked to make judgements regarding the height of wooden blocks using visual, haptic or combined visual-haptic information. Dual modality visual-haptic blocks could be presented as equal or conflicting in size. Young and older adults' size discrimination thresholds (i.e., precision) were not significantly different for visual, haptic or visual-haptic cues. In addition, both young and older adults' discrimination thresholds and points of subjective equality did not follow model predictions of optimal integration, for both conflicting and non-conflicting cues. Instead, there was considerable between subject variability as to how visual and haptic cues were processed when presented simultaneously. This finding has implications for the development of multisensory therapeutic aids and interventions to assist older adults with everyday activities, where these should be tailored to the needs of each individual.

Explaining the Effect of Likelihood Manipulation and Prior Through a Neural Network of the Audiovisual Perception of Space

Abstract: Results in the recent literature suggest that multisensory integration in the brain follows the rules of Bayesian inference. However, how neural circuits can realize such inference and how it can be learned from experience is still the subject of active research. The aim of this work is to use a recent neurocomputational model to investigate how the likelihood and prior can be encoded in synapses, and how they affect audio-visual perception, in a variety of conditions characterized by different experience, different cue reliabilities and temporal asynchrony. The model considers two unisensory networks (auditory and visual) with plastic receptive fields and plastic crossmodal synapses, trained during a learning period. During training visual and auditory stimuli are more frequent and more tuned close to the fovea. Model simulations after training have been performed in crossmodal conditions to assess the auditory and visual perception bias: visual stimuli were positioned at different azimuth (±10° from the fovea) coupled with an auditory stimulus at various audio-visual distances (±20°). The cue reliability has been altered by using visual stimuli with two different contrast levels. Model predictions are compared with behavioral data. Results show that model predictions agree with behavioral data, in a variety of conditions characterized by a different role of prior and likelihood. Finally, the effect of a different unimodal or crossmodal prior, re-learning, temporal correlation among input stimuli, and visual damage (hemianopia) are tested, to reveal the possible use of the model in the clarification of important multisensory problems.

Temporal Cognition Can Affect Spatial Cognition More Than Vice Versa: The Effect of Task-Related Stimulus Saliency.

Abstract: Cognition of space and time affect each other; a line with longer length appears to be longer in exposure duration (space on time), and a line with longer exposure duration appears to be longer in length (time on space). This cognitive interaction is known to be asymmetric; the effect of space on time is larger than that of time on space. We conjectured that this asymmetry is not intrinsic but may depend on the saliency of relevant signals. Participants were asked to judge the visual exposure duration of lines that varied in length or the lengths of the lines with different exposure times. The ranges of task-relevant and -irrelevant stimulus values were the same in the spatial and temporal tasks. Task difficulty was also evaluated by subjective rating. We found that duration affected the judgment of length more than vice versa, when the spatial task was significantly more difficult than the temporal task. Together with our previous results that showed the opposite effect, our conjecture is supported that the saliency of stimuli should affect the balance of interactions.

On the Timing of Signals in Multisensory Integration and Crossmodal Interactions: a Scoping Review

Abstract: A scoping review was undertaken to explore research investigating early interactions and integration of auditory and visual stimuli in the human brain. The focus was on methods used to study low-level multisensory temporal processing using simple stimuli in humans, and how this research has informed our understanding of multisensory perception. The study of multisensory temporal processing probes how the relative timing between signals affects perception. Several tasks, illusions, computational models, and neuroimaging techniques were identified in the literature search. Research into early audiovisual temporal processing in special populations was also reviewed. Recent research has continued to provide support for early integration of crossmodal information. These early interactions can influence higher-level factors, and vice versa. Temporal relationships between auditory and visual stimuli influence multisensory perception, and likely play a substantial role in solving the 'correspondence problem' (how the brain determines which sensory signals belong together, and which should be segregated).

Advantage of Early Focus on Visual Information in Bi-Modal Training of Bimanual Coordination.

Abstract: Previous studies have shown that learning bimanual coordination is modality-specific, and both visual and kinesthetic information about relative phase can be used to facilitate learning. However, an extended training with focus on visual information leads to the neglect of kinesthesis and a complete reliance on vision to perform the coordination. The current study explored the bi-modal training of bimanual coordination, where the participants were guided to attend to both visual and kinesthetic information to learn 90° coordination. Thirty participants in their 20s were trained for 10 sessions (two sessions a day for five days), during which they were randomly divided into three groups of 10 participants each to practice the coordination. The V-K group was focused first on visual information, and then on kinesthetic information, to learn the 90° coordination. The K-V group was focused first on kinesthetic information, and then on visual information to learn the coordination. The Random group randomly shifted their focus between visual and kinesthetic information to learn the coordination. All participants were tested as they performed the 90° coordination with and without visual information before, halfway, and after the training. The results showed that the bi-modal training yielded more improved and transferred coordination than the uni-modal training. However, among the three types of bi-modal training, the V-K schedule exhibited the most modality-specific learning and transfer. Therefore, when using both visual demonstration and physical guidance to teach bimanual coordination skills, providing visual demonstration in the early stage should be recommended.

Multisensory Integration Is Modulated by Auditory Sound Frequency and Visual Spatial Frequency

Abstract: Accurate integration of auditory and visual information is essential for our ability to communicate with others. Previous studies have shown that the temporal discrepancies over which audiovisual speech stimuli will be integrated into a coherent percept are much wider than those typically observed for simple stimuli like beeps and flashes of light. However, our sensitivity to the low-level features of simple stimuli is not constant. We hypothesized that part of the enhanced integration of audiovisual speech may be due to it consisting predominantly of the sound frequencies and visual spatial frequencies that humans are most sensitive to. Here, we examined integration behaviors for pure tones across the sound frequency spectrum and visual gratings across the spatial frequency spectrum to examine how these low-level features modulate integration. The temporal window of integration was modulated by both sound frequency and visual spatial frequency, with the widest integration window occurring when both stimuli fell within their respective peak sensitivity ranges. These results suggest that part of the increased tolerance for temporal asynchrony typically observed for audiovisual speech may be due to the differential integration of low-level stimulus features that are dominant within complex audiovisual speech.

Head Motion Elicited by Viewing Affective Pictures as Measured by a New LED-Based Technique.

Abstract: The complex sensory input and motor reflexes that keep body posture and head position aligned are influenced by emotional reactions evoked by visual or auditory stimulation. Several theoretical approaches have emphasized the relevance of motor reactions in emotional response. Emotions are considered as a tendency or predisposition to act that depends on two motivational systems in the brain - the appetitive system, related to approach behaviours, and the defensive system, related to withdrawal or fight-or-flight behaviours. Few studies on emotion have been conducted employing kinematic methods, however. Motion analysis of the head may be a promising method for studying the impact of viewing affective pictures on emotional response. For this purpose, we presented unpleasant, neutral and pleasant affective pictures. Participants were instructed to view the pictures and to remain still. Two light-emitting diodes (LEDs) were attached to the foreheads of participants, and a Wii Remote controller, positioned 25 cm away, detected the position of the LEDs in the medial-lateral and anterior-posterior axes. We found more sway in response to unpleasant pictures. In addition, unpleasant pictures also provoked faster movements than both neutral and pleasant pictures. This response to unpleasant pictures, in contrast to pleasant ones, might reflect the readiness or predisposition to act. Our data also revealed that men moved faster than women, which is in accordance with previous findings related to gender differences.