University of Birmingham
The Curious Incident of Attention in Multisensory
Integration
MacAluso, Emiliano; Noppeney, Uta; Talsma, Durk; Vercillo, Tiziana; Hartcher-O'Brien, Jess;
Adam, Ruth
DOI:
10.1163/22134808-00002528
License:
None: All rights reserved
Document Version
Peer reviewed version
Citation for published version (Harvard):
MacAluso, E, Noppeney, U, Talsma, D, Vercillo, T, Hartcher-O'Brien, J & Adam, R 2016, 'The Curious Incident
of Attention in Multisensory Integration: Bottom-up vs. Top-down', Multisensory Research, vol. 29, no. 6-7, pp.
557-583. https://doi.org/10.1163/22134808-00002528
Link to publication on Research at Birmingham portal
Publisher Rights Statement:
Eligibility for repository: Checked on 27/9/2016
General rights
Unless a licence is specified above, all rights (including copyright and moral rights) in this document are retained by the authors and/or the
copyright holders. The express permission of the copyright holder must be obtained for any use of this material other than for purposes
permitted by law.
•Users may freely distribute the URL that is used to identify this publication.
•Users may download and/or print one copy of the publication from the University of Birmingham research portal for the purpose of private
study or non-commercial research.
•User may use extracts from the document in line with the concept of ‘fair dealing’ under the Copyright, Designs and Patents Act 1988 (?)
•Users may not further distribute the material nor use it for the purposes of commercial gain.
Where a licence is displayed above, please note the terms and conditions of the licence govern your use of this document.
When citing, please reference the published version.
Take down policy
While the University of Birmingham exercises care and attention in making items available there are rare occasions when an item has been
uploaded in error or has been deemed to be commercially or otherwise sensitive.
If you believe that this is the case for this document, please contact UBIRA@lists.bham.ac.uk providing details and we will remove access to
the work immediately and investigate.
Download date: 10. Aug. 2022
The Curious Incident of Attention in Multisensory Integration: Bottom-up vs. Top-
down
Emiliano Macaluso
1
*, Uta Noppeney
2
*, Durk Talsma
3
*, Tiziana Vercillo
4
*, Jessica Hartcher-
O'Brien
5$
, Ruth Adam
6$
1. Neuroimaging Laboratory, Fondazione Santa Lucia, Rome, Italy
2. Computational Neuroscience and Cognitive Robotics Centre, University of Birmingham, UK
3. Department of Experimental Psychology, Ghent University, Henri Dunantlaan 2, B- 9000 Ghent,
Belgium
4. Department of Psychology, University of Nevada, Reno, NV, USA
5. Sorbonne Universites, UPMC Univ Paris, 06, UMR 7222, ISIR, F-75005, Paris, France
6. Institute for Stroke and Dementia Research, Klinikum der Universität München, Ludwig-Maximilians-
Universität LMU, Munich, Germany
*equal contribution (ordered alphabetically)
$equal contribution
Short title: Attention and Multisensory Integration
Corresponding authors:
Jessica Hartcher-O'Brien
Institute of Intelligent Systems and Robotics
University Pierre et Marie Curie
4 Place Jussieu
75005 Paris France
Phone:+33 (0) 1 44 27 63 58
Email: hartcher@isir.upmc.fr
Ruth Adam
Institute for Stroke and Dementia Research
Klinikum der Universität München
Feodor-Lynen-Straße 17
D-81377 Munich
Germany
Phone: ++49 (0) 89 4400-46165
Fax : ++49 (0) 89 4400-46114
Email: Ruth.Adam@med.uni-muenchen.de
Summary
The role attention plays in our experience of a coherent, multisensory world is still
controversial. On the one hand, a subset of inputs may be selected for detailed processing
and multisensory integration in a top-down manner, i.e. guidance of multisensory integration
by attention. On the other hand, stimuli may be integrated in a bottom-up fashion outside the
focus of attention according to low-level properties such as spatial coincidence. Moreover,
bottom-up attentional processes may also facilitate MSI.
Moreover, attention itself is multifaceted and can be described via both top-down and
bottom-up mechanisms. Thus, the interaction between attention and multisensory integration
is complex and situation-dependent. The authors of this opinion paper are researchers who
have contributed to this discussion from behavioural, computational and neuronal angles.
We posed a series of questions, the goal of which was to illustrate the interplay between
bottom-up and top-down processes in various multisensory scenarios in order to clarify the
standpoint taken by each author and with the hope of reaching a consensus. Although
divergence of viewpoint emerges in the current responses, there is also considerable
overlap: In general, it can be concluded that the amount of influence that attention exerts on
MSI depends on the current task as well as prior knowledge and expectations of the
observer. Moreover stimulus properties such as the reliability and salience also determine
how open the processing is to influences of attention.
Keywords: Bayesian causal inference, Endogenous, Predictive coding, Salience, Stimulus-
driven
Introduction
The interplay between attention and multisensory integration (MSI) is a complex and
controversial topic. This may be due, in part, to the fact that attention and MSI interact at
multiple levels. Moreover, both attention and MSI are complex, multifaceted processes that
contribute to the control of sensory processing and, ultimately, to behaviour. In the current
context MSI is defined as the merging of information across two or more sensory modalities
in order to obtain a coherent, robust percept. MSI describes the interaction between sensory
signals: First, when sensory signals are redundant and second when there is sensory
combination with non-redundant cues. Redundant sensory signals arise from within the
same coordinate system (e.g. both visual and auditory information can be transformed into
craniotopic coordinates) and pertain to the same environmental property (e.g. Ernst &
Banks, 2004), whereas, sensory combination refers to multisensory interactions for sensory
signals that are not redundant, may be coded in different coordinate systems and have
potentially different units (e.g. Hecht & Reiner, 2009). Both processes are referred to in the
current discussion under the umbrella of MSI. Attention is primarily defined as a guiding
process in which relevant inputs are being selected for detailed processing and perceptual
awareness out of the inflow of all incoming information (Marois and Ivanoff, 2005, Talsma et
al., 2010, Adam et al., 2014). Top-down, endogenous attention can be voluntarily allocated
toward a stimulus, a sensory modality or a specific region of space in order to achieve task
goals (Li et al., 2004; Wolfe, Butcher et al., 2003). Attention can also be involuntarily
captured ‘bottom-up’ by external events, even though the attention capturing signals are
unrelated to the current goal-directed activity (Öhman et al., 2001; Zhang et al., 2012; Wolfe
et al., 2003).
The neural mechanisms that underlie endogenous and stimulus-driven processes have been
studied extensively in the visual modality. In the field of visuo-spatial attention control, a
relatively straightforward view concerns the distinction between endogenous (internal)
control in the dorsal fronto-parietal regions and stimulus-driven (external) control in the right
ventral fronto-parietal network (Corbetta and Shulman, 2002). These two attentional control
systems are thought to work together influencing the "responsiveness" of the occipital visual
cortex (sensory modulation), e.g. by boosting the processing of visual stimuli at the attended
location, and controlling the orienting of attention towards relevant and/or unexpected visual
stimuli (Corbetta et al., 2008). Several imaging studies indicated that these two control
systems also operate in situations involving non-visual stimuli. For example the dorsal
fronto-parietal network has been found to be activated when subjects focused endogenous
attention to discriminate either auditory or tactile targets (Hill and Miller, 2010; Yantis et al.,
2002; Macaluso et al., 2003; Krumbholz et al., 2008); while the ventral network was found to
be activated when participants re-oriented attention to discriminate these targets presented
at an unattended location (Macaluso et al., 2002b; Downar et al., 2000). The finding of
modality independent responses in the fronto-parietal attention networks is consistent with
supramodal mechanisms of attentional control (Farah et al., 1989; Macaluso and Driver,
2005), which provides us with a first link between attention and the processing of
multisensory stimuli.
The interaction between MSI and attention has previously been explained both in terms of
bottom-up and top-down mechanisms. According to the account of pre-attentive automatic
integration, stimuli are integrated spontaneously at the early stage of processing and this
integration itself may capture attention. The audio-visual ventriloquist illusion, in which a
spatially discrepant sound is perceived to arise from the vicinity of a synchronous visual
stimulus, exemplifies integration which is independent of both endogenous (Bertelson et al.,
2000) as well as exogenous unisensory attention (Vroomen et al., 2001b). This illusion
further enhances spatial attention to speech sounds (Driver, 1996; though see Jack, O'Shea,
Cottrell and Ritter 2013), again suggesting that multisensory binding has occurred
automatically and before auditory attentive selection. Similarly, Van der Burg and colleagues
(2008) have demonstrated that a sound decreases search times for a synchronized visual
object and that detection accuracy is related to an early ERP component (Van der Burg et
al., 2011), supporting the idea that the automatic integration of multisensory stimuli can
recruit attention. Furthermore, sounds can capture visual attention in cases of limited
resources as demonstrated with the attentional blink paradigm (Olivers and Van der Burg,
2008).
Alternatively, attention can limit or boost MSI, even at relatively early processing stages
(Karns and Knight, 2009; Senkowski et al., 2005). Attending to an object feature in one
modality can direct attention to another modality (Busse et al., 2005; Molholm et al, 2007)
and the attentional focus of subjects affects the unisensory weights and extent of integration
with e.g. multimodal attention as opposed to attending to a single modality, facilitating
integration (Oruc et al., 2008; Vercillo and Gori, 2015). Also, high level processes such as
task goals (Donohue et al., 2015) or prior knowledge (Adam and Noppeney, 2014) can
enhance integration. On the other hand, the McGurk effect, an illusory auditory perception
generated by incongruent audio-visual speech stimuli, is considerably reduced by a
secondary task suggesting that the high attentional load reduces multisensory processing
(Alsius et al., 2005; Alsius et al., 2007).
At first sight, these current findings are not consistent and even appear contradictory. We
have asked four researchers, Emiliano Macaluso, Uta Noppeney, Durk Talsma and Tiziana
Vercillo, who have contributed to research in this field and participated in the IMRF 2015
symposium “The Curious Incident of Attention in Multisensory Integration: Bottom-up and
Top-down” to cast their opinions on this issue. Specifically, we have restricted the discussion
to the role of attention on MSI and attention's modulatory elements in the non-chemical
senses.
The role of attention on MSI
Questions 1. What kind of role does attention play in MSI and how much of this role
can be accounted for by low-level perceptual processes and how much by top-down
influences?
