In many everyday situations, our senses are bombarded by many different unisensory signals at any given time. To gain the most veridical, and least variable, estimate of environmental stimuli/properties, we need to combine the individual noisy unisensory perceptual estimates that refer to the same object, while keeping those estimates belonging to different objects or events separate. How, though, does the brain “know” which stimuli to combine? Traditionally, researchers interested in the crossmodal binding problem have focused on the roles that spatial and temporal factors play in modulating multisensory integration. However, crossmodal correspondences between various unisensory features (such as between auditory pitch and visual size) may provide yet another important means of constraining the crossmodal binding problem. A large body of research now shows that people exhibit consistent crossmodal correspondences between many stimulus features in different sensory modalities. For example, people consistently match high-pitched sounds with small, bright objects that are located high up in space. The literature reviewed here supports the view that crossmodal correspondences need to be considered alongside semantic and spatiotemporal congruency, among the key constraints that help our brains solve the crossmodal binding problem.

/pdf/crossmodal-correspondences-a-tutorial-review-20vme8gbi3.pdf

Crossmodal correspondences: A tutorial review

https://www.staff.ncl.ac.uk/candy.rowe/Rowe99.pdf

Receiver psychology and the evolution of multicomponent signals.

Intersensory facilitation of reaction time: energy summation or preparation enhancement?

A series of four experiments explored how cross-modal similarities between sensory attributes in vision and hearing reveal themselves in speeded, two-stimulus discrimination. When subjects responded differentially to stimuli on one modality, speed and accuracy of response were greater on trials accompanied by informationally irrelevant "matching" versus "mismatching" stimuli from the other modality. Cross-modal interactions appeared in (a) responses to dim/bright lights and to dark/light colors accompanied by low-pitched/high-pitched tones; (b) responses to low-pitched/high-pitched tones accompanied by dim/bright lights or by dark/light colors; (c) responses to dim/bright lights, but not to dark/light colors, accompanied by soft/loud sounds; and (d) responses to rounded/sharp forms accompanied by low-pitched/high-pitched tones. These results concur with findings on cross-modal perception, synesthesia, and synesthetic metaphor, which reveal similarities between pitch and brightness, pitch and lightness, loudness and brightness, and pitch and form. The cross-modal interactions in response speed and accuracy may take place at a sensory/perceptual level of processing or after sensory stimuli are encoded semantically.

On cross-modal similarity: auditory-visual interactions in speeded discrimination.

A little over a decade ago, Martino and Marks (Current Directions in Psychological Science 10:61-65, 2001) put forward the influential claim that cases of intuitive matchings between stimuli in different sensory modalities should be considered as a weak form of synesthesia. Over the intervening years, many other researchers have agreed-at the very least, implicitly-with this position (e.g., Bien, ten Oever, Goebel, & Sack NeuroImage 59:663-672, 2012; Eagleman Cortex 45:1266-1277, 2009; Esterman, Verstynen, Ivry, & Robertson Journal of Cognitive Neuroscience 18:1570-1576, 2006; Ludwig, Adachi, & Matzuzawa Proceedings of the National Academy of Sciences of the United States of America 108:20661-20665, 2011; Mulvenna & Walsh Trends in Cognitive Sciences 10:350-352, 2006; Sagiv & Ward 2006; Zellner, McGarry, Mattern-McClory, & Abreu Chemical Senses 33:211-222:2008). Here, though, we defend the separatist view, arguing that these cases are likely to form distinct kinds of phenomena despite their superficial similarities. We believe that crossmodal correspondences should be studied in their own right and not assimilated, either in terms of the name used or in terms of the explanation given, to synesthesia. To conflate these two phenomena is both inappropriate and potentially misleading. Below, we critically evaluate the evidence concerning the descriptive and constitutive features of crossmodal correspondences and synesthesia and highlight how they differ. Ultimately, we wish to provide a general definition of crossmodal correspondences as acquired, malleable, relative, and transitive pairings between sensory dimensions and to provide a framework in which to integrate the nonsystematic cataloguing of new cases of crossmodal correspondences, a tendency that has increased in recent years.

http://www.daysyn.com/Deroy_and_Spence_2013b.pdf

Why we are not all synesthetes (not even weakly so).

Choice reaction times (RT) were obtained from four trained Ss to red vs blue targets in four conditions. In three conditions, a noninformative tone occurred on half the trials and could be (a) low frequency, (b) high frequency, or (c) either high or low frequency. Tone never occurred in the fourth condition. As previously observed, tone occurrence produced a nonselective reduction in RT. Contrary to some studies where tone selectivity facilitated detection of blue as opposed to red targets, no such selective effects were obtained. As prior studies (Bernstein & Edelstein, 1971) have found tone frequency to interact with RT to targets differing in location (above vs below fixation), it was concluded that the information-processing stage for color selection is different from the stage for location selection.

Hue-tone sensory interaction: a negative result

Four Ss participated in a visual discriminative reaction time (RT) task to explore the phenomenon of intersensory facilitation. Previous research has indicated that RT to a joint visual-auditory event is more rapid than RT to the visual component alone, even when the auditory component is noninformative. In the present study, a tone offset was used in place of the usual affirmative event (click). The results indicated that RT was more rapid when the tone offset in conjunction with the occurrence of the visual event as compared to when the tone remained on throughout the trial. The effect was comparable in magnitude to that obtained in prior studies using an affirmative auditory event.

/pdf/use-of-tone-offset-to-facilitate-reaction-time-to-light-4c0oyqqzeo.pdf

Thomas R. Eason

Papers

Hue-tone sensory interaction: a negative result

Use of tone offset to facilitate reaction time to light onset