Documente Academic
Documente Profesional
Documente Cultură
discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/6359158
CITATIONS READS
72 182
2 authors, including:
Terrence R Stanford
Wake Forest School of Medicine
87 PUBLICATIONS 3,859 CITATIONS
SEE PROFILE
All content following this page was uploaded by Terrence R Stanford on 16 January 2014.
The user has requested enhancement of the downloaded file. All in-text references underlined in blue are added to the original document
and are linked to publications on ResearchGate, letting you access and read them immediately.
SENSORYAND MOTOR SYSTEMS NEUROREPORT
Department of Neurobiology and Anatomy, Wake Forest University School of Medicine, Winston-Salem, North Carolina, USA
Correspondence to Terrence R. Stanford, Department of Neurobiology and Anatomy, Wake Forest University School of Medicine, Medical Center Blvd.,
Winston-Salem, NC 27157, USA
Tel: +1336 716 0359; fax: +1336 716 4534; e-mail: stanford@wfubmc.edu
Single-neuron studies have highlighted dramatic enhancements in reinforced by the increasing application of noninvasive techniques
neural activity consequent to multisensory integration. Most nota- such as functional imaging and event-related potential recording,
ble are superadditive enhancements in which the multisensory re- which depend on response nonlinearities to demonstrate underly-
sponse exceeds the sum of those evoked by the modality-specic ing multisensory processes. In promoting the idea that many multi-
stimulus components individually. Although all multisensory sensory behaviors may not rely on superadditivity, we consider
enhancements may have perceptual/behavioral consequences, more recent single-neuron studies that place its incidence in
superadditivity, which suggests a nonlinear combination of modal- context. NeuroReport 18:787^792 ! c 2007 Lippincott Williams &
ity-specic inuences, seems to have had a disproportionate inu- Wilkins.
ence within the multisensory literature. This inuence has been
0959- 4965 !
c Lippincott Williams & Wilkins Vol 18 No 8 28 May 2007 787
Copyright Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
NEUROREPORT STANFORD AND STEIN
A A
15
+1207%
100 ms
50
x Impulses/trial
10
V only A only VA
5
0
V A VA
Fig. 1 Superadditivity in multisensory integration: shown here in a gure reproducing Figure 2 of Meredith and Stein [17] is an example of highly robust
multisensory enhancement and one that is clearly superadditive. In this experiment, the activity of a cat superior colliculus neuron was recorded in
response to a visual stimulus, an auditory stimulus, and their cross-modal combination. Left, top-to-bottom: the stimulus trace (in this case visual), ras-
ters of the impulses evoked during each stimulus presentation, a peristimulus time histogram of the summed impulses across trials, and a single oscillo-
scope trace of the extracellular action potentials during a single trial.The second and third panels illustrate the same for the auditory (second panel) and
cross-modal (third panel) stimulus condition. Note that the vigor of the cross-modal response far exceeds that of either modality-specic response.These
dierences are summarized in the bar graph (fourth panel, far right), which compares the mean number of impulses per trial for each stimulus condition
and quanties the percent enhancement (1207) according to the index formula,VA"max (V, A)/max (V, A) # 100, whereVA is the response to the visual-
auditory crossmodal stimulus and max (V, A) is the response to the most eective modality-specic component stimulus (visual in this example). Note
that the multisensory response of approximately18 impulses per trial far exceeds the sum of the very weak responses to the individual visual and auditory
stimuli (V + Aapprox. two impulses/trial).
multisensory stimulus would approximate the sum of the integration and behavior. In this regard, it is especially
responses to its modality-specific components. For fMRI, important to highlight results from the most recent single-
superadditivity, which is not readily explained by the neuron studies that emphasize the wider spectrum of
separate pool hypothesis, has become the litmus test for multisensory interactions and, in doing so, place the
identifying multisensory enhancements that are due to incidence (and, by inference, the importance) of super-
integration by multisensory neurons (see Refs. [12,13] for additivity in a broader context. Such studies provide
reviews). For ERP studies, superadditivity is also diagnostic important constraints on models and conceptualizations of
of multisensory integration, although its attribution to how multisensory interactions within neural populations
enhancement requires certain additional assumptions are manifest in behavioral measures.
(see Refs. [1416] discussion of these issues).
As enticing as it may be as a phenomenon, an unbalanced
emphasis on superadditivity poses a risk, a risk that is The prevalence of superadditivity in the superior
certainly heightened by the increasing number of fMRI and colliculus
ERP studies, which, despite their considerable contribu- Multisensory enhancement is the most fundamental mani-
tions, are constrained by methodological limitations that festation of integration at the single neuron level; however,
render multisensory integration synonymous with response relatively early in their investigations of the superior
nonlinearity. A lack of due diligence on the part of author or colliculus, Meredith and Stein [17] noted that, in relative
reader has the potential to distort our view of multisensory terms, the greatest enhancements occurred for multi-
representations within the brain and, as a consequence, our sensory combinations of the weakest sensory stimuli (see
understanding of the relationships between multisensory Fig. 2). Appropriately, this association was termed inverse
(a) Optimal
A A
40
V V
30
x Impulses/trial
+110%
100 200 ms
20
10
0
V only A only VA V A VA
(b) Suboptimal
30
20
50
10
+258%
0
(c) Minimal
30
20
50 10
+483%
0
Fig. 2 Inverse eectiveness in multisensory integration: This gure, adapted from Figure 8 of Meredith and Stein [17], demonstrates the principle of
inverse eectiveness for a single neuron in the cat superior colliculus. Using the same format as Fig. 1, the gure illustrates that the magnitude of multi-
sensory enhancement depends on the ecacy of the modality-specic component stimuli, which were modulated by manipulating stimulus intensity.
According to the enhancement index formula of Fig. 1, which relates the multisensory response (VA) to that for the most eective modality-specic
stimulus (auditory in all cases here), multisensory enhancement grows from approximately 53% for the optimal (a) condition to 50% for the minimal
(c) condition (values estimated from bar graphs at far right), thus demonstrating the principle of inverse eectiveness as originally dened. It is
also readily apparent from visual inspection of the bar graphs that the optimal condition yields a multisensory response that is nominally subadditive
(VAoV + A), whereas the minimal condition yields a response that is clearly superadditive (VA44V + A).
effectiveness and, as noted below, it has a counterpart in strongly stimulus dependent, evidence for inverse effective-
certain behavioral measures. With the principle of inverse ness has not prevented a superadditive ideal of the
effectiveness, it was established early on that the magnitude multisensory interaction from seemingly gaining wide
and probability of multisensory enhancement is strongly acceptance. Ironically, awareness of inverse effectiveness
influenced by the intensities, or more precisely, the efficacies may have contributed to propagating this misconception by
of the constituent stimuli. Despite the clear inference that permitting experimenters to tailor stimulus parameters to
the likelihood of observing superadditivity must also then be reliably produce large and often superadditive multisensory
Vis
Aud 10.0 VA low
Percentage of reduction in RT
0.8 V +A
Normalized (re: visual) RT
Tlow A low
8.0 VA high
0.6
VTlow
6.0 Thigh A high
0.4 VThigh
4.0
0.2
2.0
0.0
21 dB 18 dB 12 dB 6 dB 0.0
Auditory signal to noise ratio V+A V+T T+A
Fig. 4 Saccadic reaction time and inverse eectiveness. This gure, Fig. 5 Manual reaction times and inverse eectiveness. Shown here are
which was adapted from Figure 10 of Corneil et al. [7], illustrates the the decrements in manual reaction time (RT) resulting from the addition
relationship between stimulus strength and the decrements in saccadic of a stimulus from a second modality (i.e. cross-modal versus modality-
reaction time (RT) resulting from combining visual and auditory stimuli. specic). They are inversely related to stimulus intensity. The data are
Human individuals were instructed to make saccades as quickly as possi- taken fromTable 4 of Diederich and Colonius [6]. In their experiment, hu-
ble to a visual stimulus, an auditory stimulus, or a spatially congruent man individuals were instructed to depress a response button with each
visual^auditory stimulus. Auditory stimuli were presented at dierent hand upon detecting any stimulus, and manual reaction times were mea-
signal strengths ("21, "18, "12, and "6 dB) with respect to background sured. Stimuli consisted of a visual ash, auditory pure tone, vibratory
noise. Plotted are mean RTs for saccades to the auditory stimulus alone tactile stimulus, or some combination of these three stimuli. Plotted are
(white bars) and the auditory^ visual stimulus pair (gray bars) as a function the % reductions in RT for the visual^auditory, tactile^ visual, and tactile^
of increasing auditory signal strength. All mean RTs are normalized rela- auditory stimulus combinations. For each pairing, the percent reduction
tive to that for saccades to the visual stimulus alone (black bars ^ value of in mean RT is plotted with reference to that for the modality-specic
1.0). As expected, mean RT for the auditory stimulus (white bars) declines component stimulus that yielded the shortest mean RT according to the
monotonically as a function of increasing auditory stimulus strength general formula: % reduction in RTmin(RT1, RT2)"(RT1 + 2)/min(RT1,
(increasing S/N ratio from left to right). Note, however, that S/N noise RT2) # 100, where RT1 and RT2 are mean RTs for responses to each of the
ratio also modulates the inuence of multisensory integration such stimulus components and RT1 + 2 is the mean RT for responses to the
that mean RT for the cross-modal pair (gray bars) is shorter than that stimulus combination. Note that for each cross-modal stimulus, the high-
for the auditory stimulus alone (white bars) for the weakest auditory er intensity combination (gray bars) yielded a proportionally lower reduc-
stimulus ("21dB), but not for the most intense auditory stimulus tion in RT than did the lower intensity combination (black bars), thus
("6 dB), a nding consistent with inverse eectiveness. It is also impor- illustrating that the concept of inverse eectiveness applies to manual
tant to note that the apparent RT benets associated with cross-modal RT. Note also that all stimuli were suprathreshold, illustrating that the
stimuli were not exclusive to the lowest S/N ratio, but appear to be behavioral benets for multisensory integration are not exclusive to near
present and of intermediate magnitude for the intermediate S/N ratios threshold stimulation and that behavioral benets, albeit attenuated, are
("12 and "18 dB). also observed for higher intensity combinations. Stimulus pairings VAlow,
VAhigh, VTlow, VThigh, TlowAlow, ThighAhigh correspond to VA70, VA90, T1V,
T3V, T1A70, T3A90, respectively.
many cases, linear combination) of information on uni- 9. Bolognini N, Rasi F, Coccia M, Ladavas E. Visual search improvement
sensory channels, it is not surprising that the function in hemianopic patients after audiovisual stimulation. Brain 2005; 128:
relating the behavioral products of multisensory integration 28302842.
10. Calvert GA, Spence C, Stein BE. The handbook of multisensory processes.
to stimulus intensity is qualitatively similar to that for Cambridge, MA: The MIT Press; 2004.
unisensory stimuli. For unisensory stimuli, the relationship 11. Foxe JJ, Schroeder CE. The case for feedforward multisensory
between stimulus intensity and stimulus detection or convergence during early cortical processing. NeuroReport 2005;
reaction time is also one of diminishing returns; psycho- 16:419423.
metric functions of stimulus detection probability versus 12. Calvert GA. Crossmodal processing in the human brain: insights from
stimulus intensity display the characteristic sigmoid shape, functional neuroimaging studies. Cereb Cortex 2001; 11:11101123.
13. Laurienti PJ, Perrault TJ, Stanford TR, Wallace MT, Stein BE. On the use of
positively accelerating near threshold and negatively accel- superadditivity as a metric for characterizing multisensory integration in
erating near maximal performance. Likewise, reaction time functional neuroimaging studies. Exp Brain Res 2005; 166:289297.
decrements with stimulus intensity are described by an 14. Foxe JJ, Morocz IA, Murray MM, Higgins BA, Javitt DC, Schroeder CE.
inverse power function (Pierons Law) that prescribes large Multisensory auditorysomatosensory interactions in early cortical
reaction time decrements for near-threshold intensity incre- processing revealed by high-density electrical mapping. Brain Res Cogn
ments and little to no effect for increases in the high- Brain Res 2000; 10:7783.
15. Molholm S, Ritter W, Murray MM, Javitt DC, Schroeder CE, Foxe JJ.
intensity range [30,31]. Multisensory auditoryvisual interactions during early sensory
The analogy drawn between increases in activity owing to processing in humans: a high-density electrical mapping study. Brain
increments in within-modal stimulus intensity and those Res Cogn Brain Res 2002; 14:115128.
due to the addition of stimuli from a second modality (i.e. 16. Fort A, Giard M. In: Calvert GA, Spence C, Stein BE, editors. The handbook
multisensory integration) is instructive; it reminds us that, of multisensory integration. Cambridge, MA: The MIT Press; 2004.
from the point of view of circuits that must read out 17. Meredith MA, Stein BE. Visual, auditory, and somatosensory
convergence on cells in superior colliculus results in multisensory
superior colliculus activity to produce behavior, the parti-
integration. J Neurophysiol 1986; 56:640662.
cular mechanism that gave rise to an increase in activity is 18. Wallace MT, McHaffie JG, Stein BE. Visual response properties and
irrelevant. With this in mind, we emphasize that super- visuotopic representation in the newborn monkey superior colliculus.
additivity is but one of several computations through which J Neurophysiol 1997; 78:27322741.
multisensory integration enhances the neural representation 19. Jiang W, Jiang H, Rowland BA, Stein BE. Multisensory orientation
of sensory signals and the behaviors that depend on them. behavior is disrupted by neonatal cortical ablation. J Neurophysiol 2006.
20. Wallace MT, Stein BE. Early experience determines how the senses will
interact. J Neurophysiol 2006.
21. Jiang W, Wallace MT, Jiang H, Vaughan JW, Stein BE. Two cortical areas
Acknowledgement mediate multisensory integration in superior colliculus neurons.
Grant support: NS36916 and NS22543. J Neurophysiol 2001; 85:506522.
22. Stanford TR, Quessy S, Stein BE. Evaluating the operations underlying
multisensory integration in the cat superior colliculus. J Neurosci 2005;
References 25:64996508.
1. Stein BE, Meredith MA. The merging of the senses. Cambridge, MA: The 23. Perrault TJ Jr, Vaughan JW, Stein BE, Wallace MT. Superior colliculus
MIT Press; 1993. neurons use distinct operational modes in the integration of multisenory
2. Stein BE, Meredith MA, Huneycutt WS, McDade L. Behavioral indices stimuli. J Neurophysiol 2005; 93:25752586.
of multisensory integration: orientation to visual cues is affected by 24. Populin LC, Yin TC. Bimodal interactions in the superior colliculus of the
auditory stimuli. J Cogn Neurosci 1989; 1:1224. behaving cat. J Neurosci 2002; 22:28262834.
3. Hughes HC, Reuter-Lorenz PA, Nozawa G, Fendrich R. Visualauditory 25. Wallace MT, Meredith MA, Stein BE. Multisensory integration in the
interactions in sensorimotor processing: saccades versus manual superior colliculus of the alert cat. J Neurophysiol 1998; 80:10061010.
responses. J Exp Psychol Hum Percept Perform 1994; 20:131153. 26. Barraclough NE, Xiao D, Baker CI, Oram MW, Perrett DI. Integration of
4. Nozawa G, Reuter-Lorenz PA, Hughes HC. Parallel and serial processes visual and auditory information by superior temporal sulcus neurons
in the human oculomotor system: bimodal integration and express responsive to the sight of actions. J Cogn Neurosci 2005; 17:377391.
saccades. Biol Cybern 1994; 72:1934. 27. Komura Y, Tamura R, Uwano T, Nishijo H, Ono T. Auditory thalamus
5. Colonius H, Arndt P. A two-stage model for visualauditory interaction integrates visual inputs into behavioral gains. Nat Neurosci 2005; 8:
in saccadic latencies. Percept Psychophys 2001; 63:126147. 12031209.
6. Diederich A, Colonius H. Bimodal and trimodal multisensory enhance- 28. Sparks DL. Translation of sensory signals into commands for control of
ment: effects of stimulus onset and intensity on reaction time. Percept saccadic eye movements: role of primate superior colliculus. Physiol Rev
Psychophys 2004; 66:13881404. 1986; 66:118171.
7. Corneil BD, Van Wanrooij M, Munoz DP, Van Opstal AJ. Auditoryvisual 29. Hall WC, Moschovakis A. The superior colliculus: new approaches for
interactions subserving goal-directed saccades in a complex scene. studying sensorimotor integration. Boca Raton, FL: CRC Press; 2004.
J Neurophysiol 2002; 88:438-454. 30. Pieron H. The sensations: their functions, processes and mechanisms. London:
8. Frens MA, Van Opstal AJ, Van der Willigen RF. Spatial and temporal Frederick Muller Ltd.; 1952.
factors determine auditoryvisual interactions in human saccadic eye 31. Luce RD. Response times: their role in inferring elementary mental
movements. Percept Psychophys 1995; 57:802816. organisation. New York: Clarendon Press; 1986.