seeing voices: high-density electrical mapping and source-analysis
TRANSCRIPT
A
pvo(uhalawtp©
K
1
pifAGcsi
0d
Neuropsychologia 45 (2007) 587–597
Seeing voices: High-density electrical mapping and source-analysis of themultisensory mismatch negativity evoked during the McGurk illusion
Dave Saint-Amour a, Pierfilippo De Sanctis a, Sophie Molholm a,b,Walter Ritter a,b, John J. Foxe a,b,∗
a Cognitive Neurophysiology Laboratory, Nathan S. Kline Institute for Psychiatric Research, Program in Cognitive Neuroscience andSchizophrenia, 140 Old Orangeburg Road, Orangeburg, NY 10962, USA
b Program in Cognitive Neuroscience, Department of Psychology, The City College of the City University of New York,138th Street & Convent Avenue, New York, NY 10031, USA
Received 26 October 2005; received in revised form 23 March 2006; accepted 31 March 2006Available online 6 June 2006
bstract
Seeing a speaker’s facial articulatory gestures powerfully affects speech perception, helping us overcome noisy acoustical environments. Onearticularly dramatic illustration of visual influences on speech perception is the “McGurk illusion”, where dubbing an auditory phoneme ontoideo of an incongruent articulatory movement can often lead to illusory auditory percepts. This illusion is so strong that even in the absencef any real change in auditory stimulation, it activates the automatic auditory change-detection system, as indexed by the mismatch negativityMMN) component of the auditory event-related potential (ERP). We investigated the putative left hemispheric dominance of McGurk-MMNsing high-density ERPs in an oddball paradigm. Topographic mapping of the initial McGurk-MMN response showed a highly lateralized leftemisphere distribution, beginning at 175 ms. Subsequently, scalp activity was also observed over bilateral fronto-central scalp with a maximalmplitude at ∼290 ms, suggesting later recruitment of right temporal cortices. Strong left hemisphere dominance was again observed during theast phase of the McGurk-MMN waveform (350–400 ms). Source analysis indicated bilateral sources in the temporal lobe just posterior to primary
uditory cortex. While a single source in the right superior temporal gyrus (STG) accounted for the right hemisphere activity, two separate sourcesere required, one in the left transverse gyrus and the other in STG, to account for left hemisphere activity. These findings support the notionhat visually driven multisensory illusory phonetic percepts produce an auditory-MMN cortical response and that left hemisphere temporal cortexlays a crucial role in this process.
2006 Published by Elsevier Ltd.
pogr
tsca&sal
eywords: Multisensory integration; McGurk illusion; Mismatch negativity; To
. Introduction
The mismatch negativity (MMN) is a well-known electro-hysiological component reflecting preattentive detection of annfrequently presented auditory stimulus (‘deviant’) differingrom a frequently occurring stimulus (‘standard’) (Naatanen &lho, 1995; Ritter, Deacon, Gomes, Javitt, & Vaughan, 1995).eneration of the MMN is believed to reflect the cortical pro-
esses involved in comparing current auditory input with a tran-ient memory trace (lasting ∼10–20 s) of ongoing regularitiesn the auditory environment; when there is a perceptible change,
∗ Corresponding author. Tel.: +1 845 398 6547; fax: +1 845 398 654.E-mail address: [email protected] (J.J. Foxe).
nr&(Seo
028-3932/$ – see front matter © 2006 Published by Elsevier Ltd.oi:10.1016/j.neuropsychologia.2006.03.036
aphy; Preattentive; Audio–visual speech
here is an MMN response (Naatanen, 2001). As such, the MMNerves as an index of auditory sensory (echoic) memory andonstitutes the only available electrophysiological signature ofuditory discrimination abilities (Picton, Alain, Otten, Ritter,
Achim, 2000). Changes along several physical dimensionsuch as duration, intensity, or frequency of sounds can gener-te the MMN, including changes in spectrally complex stimuliike phonemes (Naatanen et al., 1997). Source analysis of mag-etic (Hari et al., 1984; Sams et al., 1985) and electrical scalp-ecordings (e.g. Giard, Perrin, Pernier, & Bouchet, 1990; Scherg
Berg, 1991) as well as intracranial recordings in animals
e.g. Csepe, Karmos, & Molnar, 1987; Javitt, Steinschneider,chroeder, Vaughan, & Arezzo, 1994) and humans (e.g. Rosburgt al., 2005) have shown that the principal neuronal generatorsf MMN are located on the supratemporal plane in auditory
5 opsyc
ce(R
ifwabrolMtttitipanpaa//sScEaD
MaagbPtdttsfMvtCsptaae
ws
2
2
iAsnwitK
2
aEido4tfisid/J1pwtss
t(apwvtbes1wm&dssactivity related to the illusory phonemic change was examined by comparing theresulting ‘auditory’ standard and deviant responses. This subtraction procedure,of course, does not mean that the resulting ERP responses are purely auditory
88 D. Saint-Amour et al. / Neur
ortex, although additional regions including frontal and pari-tal cortices are also likely associated with MMN processingMarco-Pallares, Grau, & Ruffini, 2005; Molholm, Martinez,itter, Javitt, & Foxe, 2005).
Although the MMN is typically elicited by physical changesn the regularity of acoustical signals and attention is not requiredor its generation, evidence suggests that MMN is associatedith purely subjective changes in auditory percepts in the
bsence of any actual acoustical variation. This was first reportedy Sams et al. (1991) using the so-called McGurk illusion, aemarkable multisensory illusion whereby dubbing a phonemento an incongruent visual articulatory speech movement canead to profound illusory auditory perceptions (McGurk &
acDonald, 1976). Depending upon the particular combina-ion of acoustic phoneme and visual speech articulation usedo evoke the McGurk illusion, the resultant percept will tendo be a fusion of the mismatched auditory and visual speechnputs (e.g. auditory /ba/ dubbed onto visual /ga/ results inhe percept of /da/) or will be dominated by the visual speechnput (e.g. auditory /ba/ dubbed onto visual /va/ results in theercept of /va/). Even when initially naıve subjects are fullypprised of the nature of the illusion, the illusion is so domi-ant that subjects continue to report hearing the illusory speechercept rather than the presented auditory speech sound. Samsnd colleagues showed that infrequent deviations in the visualrticulation of audio–visual syllables (acoustic /pa/ and visualka/) interspersed in a sequence of congruent trials (acousticpa/ and visual /pa/) gave rise to magnetic mismatch fields in theupratemporal region (see also Mottonen, Krause, Tiippana, &ams, 2002 for similar results). This MMN-like response asso-iated with the McGurk effect was also found in scalp-recordedRPs with maximal amplitude between 200 and 300 ms afteruditory onset (Colin et al., 2002; Colin, Radeau, Soquet, &eltenre, 2004).The present study aimed to further characterize the McGurk-
MN, using high-density electrical mapping and source-nalysis to investigate the underlying cortical sources of thisctivity. In previous studies, the McGurk-MMN has been investi-ated by comparing the magnetic or electrical responses evokedy the standard and the deviant audio–visual speech stimuli.roblematically, this comparison will yield not only any poten-
ial McGurk-MMN effect, but also any differential responsesue to the physically different visual stimuli used to make uphe standards and deviants. In the present study we ran addi-ional conditions where only the visual stimuli were presentedo that we could (1) subtract out the evoked visual responsesrom the auditory–visual responses from which the McGurk-
MN was derived; (2) characterize the visual activity to theisual-alone stimuli to assess the extent to which they do con-ribute to activity in the latency range of the auditory response.ontrolling for effects due to physical differences in the visual
timuli, we find clear evidence for a McGurk-MMN. That is,erceived phonemic changes in the absence of actual acous-
ic changes elicited the MMN (Colin et al., 2002, 2004). Inddition, topographic mapping and dipole modeling revealeddominance of left hemispheric cortical generators during thearly and late phases of the McGurk-MMN, consistent with the
a
e
hologia 45 (2007) 587–597
ell-known left hemispheric dominance for the processing ofpeech.
. Methods
.1. Participants
Eleven adult volunteers (ages: 19–33 years; mean: 25.6; five males) partic-pated in the experiment and were naive with regard to the intent of the study.fter the study, all subjects were debriefed to ensure that they experienced
trong McGurk illusions.1 The participants reported that they had no hearing oreurological deficits. They possessed normal or corrected-to-normal vision andere right-handed (except for one) as assessed by the Edinburgh handedness
nventory. All subjects provided written informed consent in accordance withhe Declaration of Helsinki, and the Institutional Review Board of the Nathanline Research Institute approved all procedures.
.2. Stimuli and procedure
Stimuli were generated by digitally recording video (frame rate: 25 images/s;udio sample rate: 44.1 KHz in 16 bits) of the natural articulations of a malenglish speaker’s mouth saying the syllables /ba/ and /va/. Although the visual
nformation for both syllables is different in terms of place of articulation, theuration of mouth movement was similar for both stimuli with respect to thenset of the acoustical signal. There was a small difference of approximately0 ms between the onsets of the two visual articulatory movements relative tohe onset of the respective acoustic signals (i.e. −320 ms for /ba/ versus −360 msor /va/). Productions began and ended in a neutral closed mouth position. Thellusory McGurk audio-visual pair was created by synchronously dubbing thepoken syllable /ba/ onto the video of /va/. This particular combination elic-ts a particularly strong McGurk illusion in which the auditory perception isominated by the visual information—that is, observers usually report hearingva/, which corresponds to the visual portion of the bisensory stimulus (e.g.ones & Callan, 2003; Rosenblum & Saldana, 1992; Summerfield & McGrath,984). The auditory /ba/ stimulus (duration: 370 ms; intensity: 60 dB SPL) wasresented binaurally over headphones (Sennheiser-HD600). The visual stimuliere presented to the center of a computer CRT monitor located 100 cm from
he subject (Iiyama VisionMaster Pro 502, 1024 × 768 pixels, 75 Hz). Stimuliubtended 10◦ × 7.5◦ of visual angle and subjects were instructed to look at thepeaking face stimulus and pay attention to the mouth articulation.
McGurk-MMN was generated using an oddball paradigm in two condi-ions: visual alone (/ba/ as ‘standard’ and /va/ as ‘deviant’) and audio–visualcongruent audio /ba/ and visual /ba/ as ‘standard’ and incongruent audio /ba/nd visual /va/ as ‘deviant’). Hence, in the critical audio–visual condition, thehoneme /ba/ was presented auditorily on every trial but was perceived as /va/hen presented with an incongruent visual articulation. The inter-stimulus inter-al was 1630 ms and the probability of a deviant trial was 20%. Because theask involved simple fixation on the speaker’s mouth, very short stimulationlocks (approximately 11⁄2 min each) were administrated to minimize fatigueffects. Visual alone and audio–visual blocks (35–40 stimuli/block) were pre-ented in random order and separated by short breaks for a total of approximately420 trials per condition. The rationale for using the visual alone conditionas twofold. First, we aimed to rule out the possibility that the McGurk-MMNight be attributable to visual mismatch processes (Pazo-Alvarez, Cadaveira,Amenedo, 2003). Second, this condition controls for any sensory response
ifferences due to differential mouth movements in the standard and the devianttimuli. For the main analysis, the standard and deviant visual responses wereubtracted from the corresponding auditory–visual responses. As such, MMN
s they will also include some integrative processing (e.g. Molholm, Ritter,
1 A 12th subject was excluded from the analysis because he reported notxperiencing strong McGurk illusions during post-experiment debriefing.
opsyc
JM
2
<saesnpftstcwat
aVitmaktemurcc
ttp0dBtlM
2
ifidscrmTaMgdepi
m
nbfmrswidfmrtaa
3
a/tvoatubtaFvsdtftsetd
fdvlwsotcacross this 50 ms epoch at Fz (t10 = −2.47, p = 0.033) withpost-hoc tests showing substantially stronger effects at thepeak of the MMN response (p < 0.001; see Fig. 3). We did
D. Saint-Amour et al. / Neur
avitt, & Foxe, 2004), but it does better control for visual evoked activity in thecGurk-MMN.
.3. Data acquisition and statistical analysis
Continuous EEG was acquired from 128-scalp electrodes (impedances5 k�), referenced to the nose, band-pass filtered from 0.05 to 100 Hz, and
ampled at 500 Hz (SynAmps amplifiers, NeuroScan Inc.). Trials with blinksnd eye movements were automatically rejected off-line on the basis of thelectro-occulogram. An artifact criterion of ±75 �V was used at all other scalpites to reject trials with muscular or other noise transient artifacts. The averageumber of accepted sweeps for the deviant responses was 240 (±34). A zero-hase-shift Butterworth digital band-pass filter (0.5–50 Hz, 48 dB) was appliedor ERP peak analysis. The continuous EEG was divided into epochs (the −500o 600 ms surrounding the onset of auditory stimulation) and averaged from eachubject to compute the ERPs. Epochs were first baseline-corrected on a pres-imulus interval of 100 ms before the visual onset in order to subtract the visualondition responses from the audio–visual condition responses. The resultingaveforms were then baseline-corrected over the 100 ms epoch preceding the
uditory stimulus onset (−100 to 0 ms). The MMN was obtained by subtractinghe standard response from the deviant response.
Two methods were used to ascertain the presence of the MMN. A commonpproach (e.g. Ritter, Sussman, Molholm, & Foxe, 2002; Sussman, Ritter, &aughan, 1998; Tervaniemi, Schroger, & Naatanen, 1997) consists of measur-
ng the mean voltage across a 50 ms window centered at the peak latency ofhe grand mean MMN (deviant minus standard) and submitting this dependent
easure to analysis of variance (ANOVA). This was done for four sites, rightnd left mastoids, Fz and Oz. The first three sites are from scalp regions wellnown to show large MMN responses, while a posterior electrode was choseno assess the efficacy of our subtraction procedure in terms of eliminating differ-nces between the waveforms due to visual differences in stimulation. Repeatedeasure analysis of variance (ANOVA) involving additional electrodes was then
sed to test for asymmetry with the following 2 × 5 factors: hemisphere (left,ight) and electrode (FC1/FC2, C1/C2, C3/C4, C5/C6, T7/T8). Statistical signifi-ance was assessed with an alpha level of 0.05 submitted to Greenhouse–Geisserorrections for violation of the sphericity assumption.
To provide a more general description of the spatio-temporal properties ofhe McGurk-MMN, a second method was computed based on point-wise paired-tests between standard and deviant responses for all electrodes at each timeoint. For each scalp electrode, the first time point where the t-test exceeded the.05 p-value criterion for at least 11 consecutive data points (>20 ms at a 500 Hzigitization rate) was labeled as onset of McGurk-MMN response (Guthrie &uchwald, 1991). The resulting statistical cluster plots are a suitable alternative
o Bonferroni correction for multiple comparisons, which would increase theikelihood of type II errors through overcompensation for type I errors (see
urray et al., 2002).
.4. Topographical mapping and source analysis
Exact electrode locations were assessed for each subject on the day of test-ng by 3D-digitization of the locations of the scalp electrodes with respect toduciary landmarks (i.e. the nasion and pre-auricular notches) using a magneticigitization device (Polhemus FastrakTM). Electrode placement was highly con-istent across subjects due to the use of a custom-designed electrode cap thatonstrained inter-electrode spacing and placement. 3D-scalp topographic mapsepresenting interpolated potential distributions were derived from the 128-scalpeasurements and based on the computation of a common average reference.hese interpolated potential maps were displayed on the 3D reconstructions implemented in the Brain Electrical Source Analysis software (BESA 5.1,EGIS Software GmbH, Munich, Germany). Scalp current density (SCD) topo-
raphic mapping was then computed. This method, based on the second spatialerivative of the recorded potential, eliminates the contribution of the reference
lectrode and reduces the effects of volume conduction to the surface recordedotential. This allows for better visualization of the approximate locations ofntracranial generators that contribute to a given scalp recorded ERP.Source localization of the intracranial generators was conducted with dipoleodeling using BESA. This method assumes that there are a limited and distinct
wvo
hologia 45 (2007) 587–597 589
umber of active brain regions over the evoked potential epoch, each of which cane approximated by an equivalent dipole. Dipole generators are placed within aour-shell (brain, cerebrospinal fluid, bone and skin) spherical volume conductorodel with a radius of 90 mm and scalp and skull thickness of 6 and 7 mm,
espectively. The genetic algorithm module of BESA was used to free fit aingle dipole to the peak amplitude of the McGurk-MMN. This initial dipoleas fixed and additional dipoles were successively free fit to assess if they
mproved the solution. When not fixed, the positions and orientations of theipoles are iteratively adjusted to minimize the residual variance between theorward solution and the observed data. Group averaged ERP data were used toaintain the highest possible signal-to-noise ratio as well as to generalize our
esults across individuals. It should be pointed out that in dipole source analysis,he modeled dipoles represent an oversimplification of the activity in the areasnd should be considered as representative of centers of gravity of the observedctivity.
. Results
All 11 subjects included in the analyses reported experiencingstrong McGurk illusion, the majority reporting hearing a clear
va/ with some also reporting hearing occasional derivative frica-ive phonemes (/fa/ or /tha/). The presence of a clear ERP to theisual-alone condition is evident in Fig. 1 where the waveformsbtained for the standard (/ba/) and deviant (/va/) in the visual-lone condition show ongoing visual activity before and afterime point zero (corresponding to the onset of the auditory stim-lus in the audio–visual condition), not only over occipital areasut also over parietal and central regions. This demonstrateshe importance of subtracting out the visual response from theudio–visual response to accurately assess the McGurk-MMN.urther, as can be seen in Fig. 1, there was no suggestion of aisual MMN for the visual alone condition, as evidenced in theubtraction waveform (gray trace).2 The ERPs elicited by theeviant stimuli (red waveforms) are never more negative thanhe ERPs elicited by the standard visual stimuli for more than aew data points (blue waveforms). However, application of the-test method of Guthrie and Buchwald (1991) revealed severalignificant clusters (p < 0.05) over fronto-central scalp during thepoch of the auditory MMN (230–320 ms), with the response tohe standard (“ba”) clearly diverging from that of the deviantue to an additional sharp negative deflection.
Fig. 2 illustrates the standard, deviant and difference wave-orms obtained by subtracting the ERP elicited by the stan-ards (/ba/) from the ERP elicited by the deviant (with theisual-alone control ERPs subtracted out). A typical MMN-ike waveform was found at Fz between 175 and 400 msith peak latency at 290 ms after auditory onset. For each
ubject, the mean voltage across a 50 ms window centeredn the MMN peak (266–316 ms) was measured in the sub-racted waveforms and compared against zero. The t-testonfirmed the presence of a significant MMN-like response
2 It is the prestimulus epoch that we consider here, where a visual MMNould be expected around −150 ms (that is, about 200 ms after the onset of theisual deviant). The statistical cluster plots showed no significant differencesver posterior scalp in this time frame.
590 D. Saint-Amour et al. / Neuropsychologia 45 (2007) 587–597
Fig. 1. Grand mean (n = 11) ERPs for the visual-alone condition. (A) Strong visual responses are evoked by the ongoing videos of speech articulations. Waveformsare plotted for the standard trials (blue line), the deviant trials (red line) and the difference wave (gray line) as obtained by subtracting responses to the standard trialsf n of ts ime aa
naMOtntFArdO
glpg1Sf∼
w(
tttttndoTtifst
rom those to the deviant trials. (B) Topographic mapping shows the distributiocalp. Note that visual activity precedes the 0 ms timepoint, which denotes the trticulations precede the actual onset of any sound.
ot find any evidence for polarity inversion between frontalnd mastoid electrodes, which is more typically found inMN studies using basic acoustical changes such as pitch.nly the left mastoid showed a positive going response in
he latency of the MMN, but this response was not sig-ificantly different from baseline (p = 0.66). Note howeverhat when the current density transform is performed (seeig. 4), SCD maps reveal polarity inversion at the mastoid.s expected, having subtracted out the corresponding visual
esponses, no significant difference between deviant and stan-ard responses was observed over occipital regions (p = 0.71 atz).To explore the spatio-temporal properties of McGurk-MMN
eneration in detail, with a particular focus on determining theaterality of the initial phase of the response, statistical clusterlots were computed (Fig. 3). The earliest statistical evidence foreneration of MMN responses began over left temporal scalp at
74 ms, and remained distinctly left-lateralized until ∼250 ms.ubsequently, a second phase of activity with spread to bilateralronto-central scalp sites began, reaching maximal amplitude at290 ms. Finally, a third distinct phase of the MMN responseop
e
his activity (average of the standard and deviant stimuli) over occipito-parietalt which the auditory stimulus onsets during audio–visual trials, since the visual
as seen, with a return to exclusively left-lateralized scalp sitespeaking ∼375 ms).
The three distinct phases of McGurk-MMN activity are illus-rated in the scalp topographic maps shown in Fig. 4 for threeime points centered at the peak of each of these phases. Toest for laterality effects in the McGurk-MMN, mean ampli-ude values were calculated for three 50 ms time-windows cen-ered at 200, 292 and 375 ms, for ten electrodes along a coro-al line from Fz towards the left and right mastoid. Theseata were then submitted to repeated ANOVAs with factorsf hemisphere and electrode (FC1/FC2, C1/C2, C3/C4, C5/C6,7/T8). A significant main effect of hemisphere was found for
he initial phase of the MMN (F1,10 = 6.1, p < 0.05), driven byts left lateralization as detailed above. As one would expectrom the topographic maps, the second phase of the MMNhowed no such effect (p > 0.1). While the cluster plots andopographic maps indicated left lateralization of the third phase
f the MMN, this did not quite reach significance (F1,10 = 4.3,= 0.06).Although clear left-sided dominance can be observed in thearly and late phases of the mismatch response, the scalp topog-
D. Saint-Amour et al. / Neuropsychologia 45 (2007) 587–597 591
Fig. 2. Grand mean ERPs after subtraction of visual-alone responses fromaudio–visual responses. Waveforms from a pair of electrode sites over mid-line frontal scalp (Fz) and midline occipital scalp (Oz) for the standard trials(blue line), the deviant trials (red line) and the difference wave (gray line) wereota
rbt2atem
v
Fig. 3. Statistical cluster plots of the McGurk-MMN response. Color valuesindicate the p-values that result from point-wise t-tests evaluating the mismatchnegativity effect across post-stimulus time (x-axis) and electrode positions (y-axis). General electrode positions are arranged from frontal to posterior regions(r(
(0
eitl2eaApccawos(
aw(szat
btained by subtracting responses to the standard trials from those to the deviantrials. A clear MMN-like response can be seen in the waveforms at Fz, but nott Oz.
aphy at the peak latency of this effect (292 ms at Fz) suggestsilateral generators. As illustrated in the lower panel of Fig. 4,he corresponding SCD map shows that the main activity at92 ms originates from bilateral temporal cortex. A post hocnalysis was conducted on the SCD values to assess whether
he power (defined as the sum of the absolute voltage for twolectrodes) of the left-mastoid/T7 pair differed from the right-astoid/T8 pair.3 No significant effect of hemisphere was found3 Due to volume conduction, topographic laterality can be obscured in theoltage domain (Foxe & Simpson, 2002). Calculating the SCD removes the
tte
eb
top to bottom of graph) and the scalp has been divided into six general scalpegions. Within each general region, electrode laterality is arranged from rightR) to left scalp (L). Only p-values <0.05 are color-coded.
t(10) = 0.332, p = 0.747), with mean amplitudes of 0.156 and.145 �V/cm2 for the left and right sides, respectively.
Source analysis was conducted on the grand-average differ-nce wave (deviant minus standard responses) to model thentracranial generators of the McGurk-MMN (Fig. 5). Sinceopographic mapping and cluster plots showed a substantiallyateralized distribution over left lateral temporal scalp at around00 ms (Figs. 4 and 5), we started by free fitting a singlequivalent-current dipole for the 190–216 ms time windowcross which this strong unilateral distribution was observed.
stable fit was found in the region of the left transverse tem-oral gyrus of Heschl on the supratemporal plane, in verylose proximity to the superior temporal gyrus (STG) (Talairachoordinates: x = −53, y = −25, z = 13; Brodmann area: 41), andccounted for 65.8% of the variance in the data across this periodith a peak goodness-of-fit of 73% at 204 ms. The location andrientation of this dipole were then fixed before proceeding withource-modeling of the second major phase of MMN activity250–300 ms).
Since the first dipole explained the bulk of left lateralizedctivity for this second phase also, only a single additional dipoleas initially fit across this period. The addition of this dipole
freely fit) improved the explained variance to 88.4% across thisecond epoch with a location in the right STG (x = 47, y = −42,= 11). A third dipole was then added to the model, againllowed to freely fit, and indicated the presence of a generator inhe left superior temporal cortex located slightly deeper and pos-
erior to the first left-side dipole (x = −43, y = −38, z = 16). Thishree-dipole model provided an excellent fit for the 250–300 mspoch, accounting for fully 92.7% of the variance (with a peakffects of volume conduction and results in more focused topographic maps andy extension, in potentially greater sensitivity to possible laterality differences.
592 D. Saint-Amour et al. / Neuropsychologia 45 (2007) 587–597
F the ta
gatafspf
4
eprth(rbtt(
baasotdws
papttDDor
ig. 4. Topographical voltage distribution of the McGurk-MMN activity acrossnd SCD maps (lower panel) during the middle phase.
oodness-of-fit of 94.6% at 264 ms). Opening the epoch upcross the entire MMN epoch (174–384 ms as determined byhe cluster plot analysis), this simple three-dipole model stillccounted for 85% of the variance in the data. Addition of aourth freely fitting dipole across this epoch did not result in anyubstantial improvement in fit (explained variance) and did notroduce a stable fourth generator location, so the presence of aourth source was rejected.
. Discussion
In the current study we measured the spatio-temporal prop-rties of the McGurk-MMN using high-density electrical map-ing. A robust MMN response was uncovered in the latencyange from 175 to 400 ms. Topographic mapping of the ini-ial McGurk-MMN response showed a highly lateralized leftemisphere distribution that persisted for approximately 50 ms175–225 ms). In a second phase of activity, right temporalegions also became active as MMN activity became strongly
ilateralized over fronto-central scalp, reaching maximal ampli-ude at 290 ms. Finally, during a third phase of activity, theopographic distribution revealed a left hemisphere dominance350–400 ms). Source analysis of the McGurk-MMN implicatedotvt
hree main phases of activity identified in the difference waveforms (top panel)
ilateral sources in the temporal lobe just posterior to primaryuditory cortex. While right hemispheric contributions wereccounted for with a single source in the STG, two separateources best accounted for the left hemispheric contributions,ne in the transverse gyrus of Heschl (Broadman area 41) andhe other in STG. These findings support the notion that visuallyriven multisensory illusory phonetic percepts are associatedith an auditory MMN cortical response, and that left hemi-
phere temporal cortex plays a crucial role in this process.Because the McGurk illusion allows the elicitation of a
honetic-MMN in the absence of any acoustic changes, it isn ideal paradigm to dissociate phonologic from acoustic MMNrocessing (Colin et al., 2002). MMN studies suggest the exis-ence of an automatic phonetic-specific trace in the human brainhat is distinct from acoustic change-detection processes (e.g.ehaene-Lambertz, 1997; Naatanen et al., 1997; Sharma &orman, 1999). While acoustic-specific MMN is thought to relyn both hemispheres, with the right hemisphere playing a crucialole, phonetic-specific MMNs appear to occur predominantly
n the left side (Naatanen, 2001). Our results support the exis-ence of phonetic/phonologic MMN processing. Although theoltage maps showed a bilateral fronto-central distribution athe peak latency of the MMN waveform, the earliest and again
D. Saint-Amour et al. / Neuropsychologia 45 (2007) 587–597 593
Fig. 5. Source-analysis of the McGurk-MMN. Dipoles are superimposed on an averaged brain and shown in axial (left), coronal (upper right) and sagittal (lowerright) views (L, left; R, right; A, anterior; P, posterior). Source waveforms are plotted for each of the three generators displayed in the brain slices. The top plot showssource activity for the first left hemisphere dipole, located in the region of the left transverse gyrus. Dashed lines delimit the early left lateralized phase (175–225 ms)o ows ad urce aw -like
tlroabbtW1
tmrpct
f the MMN response, activity that is unique to this source. The middle plot shelimit the second phase of the MMN response, which is evident in both this soas found in more posterior left superior temporal cortex with a very late MMN
he later phase of the McGurk-MMN were found to be stronglyeft-lateralized. Dipole modeling suggests that the most poste-ior of the two left temporal generators is located at the junctionf Brodmann areas 22 and 40, in the vicinity of Wernicke’srea. The existence of such a generator has been hypothesizedy Rinne et al. (1999) in their ERP study of MMN elicited
y non-phonetic and phonetic sounds in which they concludedhat: “. . . additional posterior areas of the temporal cortex (e.g.ernicke’s area) are activated by phonetic stimulation . . .” (p.16). Although precise localization of the intracranial genera-
M
ua
ctivity for the right dipole located in the superior temporal gyrus. Dashed linesnd the left source plotted above. Finally, in the bottom panel, a third generatorresponse (here indicated by the arrow).
ors of the MMN is beyond the spatial resolution of ERP sourceodeling, our results do support the participation of the poste-
ior temporal cortex of the left hemisphere in McGurk-MMNrocessing. This finding is also in agreement with Sams andollaborators’ MEG studies that show magnetic responses inhe supratemporal auditory cortex are associated with infrequent
cGurk perceptions (Mottonen et al., 2002; Sams et al., 1991).Our finding of two sources in the left temporal lobe is partic-
larly interesting because of the hypothesis that speech stimulictivate two MMNs, one acoustic and one phonetic, in the tem-
5 opsyc
p1cwrpimiawaTsAsdishhosaof
aeevdpmfstotp
emsrabaactCapw
m
tAsiCiaeit(odnmptWsfatPeoo
oMisVasFidNM1(
cba1rmSPtsK
94 D. Saint-Amour et al. / Neur
oral lobe of the left hemisphere (Naatanen, 2001; Rinne et al.,999; see also Winkler et al., 1999). One of these MMNs isonsidered to occur with regard to the acoustic change and oneith regard to the phonetic change, the latter being more poste-
ior than the former. The posterior dipole depicted in the presentaper supports the existence of phonetic processing in the vicin-ty of Wernicke’s area, but there is a question as to whether the
ore anterior source should be considered an acoustic MMN,.e. a MMN with an acoustic-level representation. We wouldrgue that it makes sense that the McGurk illusion is associatedith an acoustic MMN in that the illusion is associated with an
uditory perception that differs from the actual auditory input.his suggests that coding of the acoustic features of the auditorytimulus were modified by the visual input to auditory cortex.ssuming this modification occurs prior to input to the MMN
ystem, the acoustic representation of the auditory input wouldiffer from the acoustic representation of the standard, resultingn the activation of an acoustic MMN. However, the general con-ensus is that acoustic MMNs are larger in the right than the leftemisphere (Naatanen, 2001), whereas the initial source foundere was unilateral and located in the transverse temporal gyrusf Heschl, exclusively in the left hemisphere. It is only in theecond phase of MMN processing that we see the emergence ofstrong right hemisphere MMN generator. Why the first phasef the MMN is left-lateralized is not entirely clear, and meritsurther investigation.
Although we did not find clear evidence of polarity inversiont mastoid electrodes for the McGurk-MMN, except for the earli-st phase, scalp current density mapping (SCD) unambiguouslystablished that there was indeed some inversion. In a similarein, Colin et al. (2002), using a design comparable to ours, alsoid not find obvious polarity inversion for the McGurk-MMN butointed out that polarity inversions at the mastoid are often onlyanifest at very high signal-to-noise ratios which can require
ar more deviant trials than were obtained here. Both studiesuggest that polarity reversal for the McGurk-MMN is weakerhan for other contrasts, which in turn suggests that the generatorrientation of this MMN is not equivalent to the orientation ofhe standard MMN evoked by simple auditory deviance whereolarity inversions are typically far more obvious.
Differences in visual stimulation may have erroneously influ-nced previous measures of the McGurk-MMN, and Fig. 1 hereakes it clear that sensory-processing differences are indeed
een for the two visual articulatory stimuli. Colin et al. (2002)eported an earlier McGurk-MMN effect at about 150 ms afteruditory onset. This McGurk-MMN activity, however, may haveeen overestimated because of interference from the prior visualrticulation; the standard and deviant waveforms in the Colin etl. paper do not show a high degree of overlap in the period pre-eding the onset of auditory stimulation and the whole epoch ofhe deviant condition tends to be non-specifically more negative.onsequently, we believe that this “early” mismatch response,lso reported by Sams et al. (1991) with MEG, should be inter-
reted cautiously given that the overlapping visual responsesere not subtracted out.A number of studies have revealed that there is a visual mis-atch system that appears to operate in a similar manner to
ee(a
hologia 45 (2007) 587–597
he auditory mismatch system (see e.g. Heslenfeld, 2003; Pazo-lvarez et al., 2003). Thus, it is curious that there is no obvious
uggestion of a visual MMN in our data (see Fig. 1). A closenspection of the visual-alone data of Sams et al. (1991) andolin et al. (2002, 2004) also shows no obvious visual MMN. It
s not clear why no visual MMN appears to be elicited by visualrticulatory stimuli in oddball designs, but the finding strength-ns our conclusion that the auditory MMN obtained in our studys not affected by an overlapping visual MMN. What then are weo make of the differences that were seen between the standard“ba”) and deviant (“va”) visual stimuli? In the general epochf the auditory MMN (230–320 ms), while the response to theeviant stimulus continues to be positive, a distinct additionalegative deflection emerges in the response to the standard. Theorphology of this difference would appear to rule out P300-like
rocesses as its topography is more fronto-central and it appearso be driven mainly by additional processing of the standard.
hat is more, it should be kept in mind that the deviant visualtimulus was presented some 600 ms before the peak of this dif-erence. Since there is a clear difference between the standardnd deviant beginning at stimulus onset, this divergence betweenhe standard and deviant responses is really too late to reflect a300-like process. The more likely explanation for this differ-nce therefore is basic differences in sensory processing of thengoing visual articulations, which one would fully expect toccur given the rather distinct sensory input patterns.
Another possibility that needs to be considered is that someddball-related N2 or N2b activity was superimposed on theMN. That is, in typical oddball designs where subjects explic-
tly attend for the occurrence of the rare target events, theo-called N2-P3 component complex is evoked (e.g. Simson,aughan, & Ritter, 1977). However, several results suggest thatmajor contribution from N2 processes is unlikely in the presenttudy. First, the complete absence of any P3-like processes (seeig. 2) strongly suggests that subjects were not explicitly attend-
ng to the auditory–visual oddball stimuli. Second, the N2 isistinguishable from the MMN by its scalp topography in that2 has its maximal amplitude at central scalp sites, whereasMN is maximal at frontal sites (Novak, Ritter, & Vaughan,
992), and it is this fronto-central topography that is seen heresee voltage map at 292 ms in Fig. 3).
The existence of an MMN to the McGurk illusion indi-ates that auditory sensory memory processes can be modifiedy visual inputs. Auditory and visual information in speechre undeniably closely linked together (Liberman & Mattingly,985). For example, it is remarkable just how much speechecognition can be accentuated by the visual cues provided byouth articulation under noisy environmental conditions (Ross,aint-Amour, Leavitt, Javitt, & Foxe, in press; see also Sumby &ollack, 1954). In fact, seeing the lip movements of a speaker in
he absence of any auditory stimulation can result in activation ofecondary auditory cortex (Bernstein et al., 2002; Callan, Callan,roos, & Vatikiotis-Bateson, 2001; Calvert et al., 1997; Sams
t al., 1991) and data now appear to unequivocally show thatven primary auditory cortex is activated by seen lip-movementsMacSweeney et al., 2000; Molholm & Foxe, 2005; Pekkola etl., 2005). Such multisensory interactions raise the question of
opsyc
hths1ate1FndBiscaactstatu(tbchrrreiaF2toeMmp
5
tMss2Mpsp
aiaTapao
A
HeDptp(lN
R
B
B
C
C
C
C
C
C
D
F
F
D. Saint-Amour et al. / Neur
ow the auditory cortex is influenced by visual inputs. Accordingo a feedback model, this effect is driven by top-down inputs fromigher order multisensory regions such as the superior temporalulcus onto the early auditory unisensory areas (Calvert et al.,999). An alternative, but more controversial model, is that therere inputs to early sensory regions from other sensory systemshat affect stimulus processing in a feedforward manner (Foxet al., 2000, 2002; Foxe & Schroeder, 2005; Giard & Peronnet,999; Molholm et al., 2002; Murray et al., 2005; Schroeder &oxe, 2002, 2005). In support of this view, monosynaptic con-ections between visual and auditory cortices have recently beenemonstrated anatomically in macaques (Falchier, Clavagnier,arone, & Kennedy, 2002), suggesting direct cortico-cortical
nfluences of visual inputs to auditory cortex. In the audio–visualpeech domain, the fact that seeing the visual articulation pre-edes the auditory signal makes plausible the existence of earlyudio–visual interactions, allowing visual representations to belready available when auditory processing is initiated. Thisould allow the speech system to prime the corresponding audi-ory speech processing, or, in the case of bisensory incongruenceuch as in a McGurk condition, modify it to resolve the percep-ual ambiguity across sensory inputs. In support of this notion,recent ERP study reported audio–visual interaction as early as
he human brainstem response at just 11 ms post-acoustic stim-lation for both congruent and incongruent (McGurk) stimuliMusacchia, Sams, Nicol, & Kraus, 2006). Clearly, effects inhe auditory brainstem such as these must be driven by feed-ack influences due to the preceding visual stimulation. At theortical level, preliminary results from intracranial recordings inuman STG suggest that visual and auditory inputs can interactelatively early in the information-processing stream and giveise to McGurk-MMN responses with similar latencies to MMNesponses obtained with auditory-alone stimuli (Saint-Amourt al., 2005). Interestingly, recent evidence suggests that visualnformation can influence short-term auditory representation inn MMN oddball paradigm even for non-speech stimuli (Besle,ort, & Giard, 2005; Stekelenburg, Vroomen, & de Gelder,004). These findings support the notion that visual and audi-ory information can interact during or before the generationf the MMN. Hence, it is reasonable to assume that the influ-nce of visual inputs on the auditory representation underlyingcGurk-MMN involves feedback processing from higher-orderultisensory regions (e.g. superior temporal sulcus), but also
rocessing at very early stages of the sensory computation.
. Conclusion
Visual influences on auditory speech perception were inves-igated with an auditory-like MMN paradigm. The McGurk-
MN was shown to onset at 175 ms, demonstrating that visualpeech articulations influenced higher order phonemic repre-entations by this time. The amplitude of the MMN peaked at90 ms and extended until 400 ms. Over the time course of the
MN, two distinct topographies were observed with a left-sidedredominance. Along with dipole modeling we showed that aingle left temporal generator is adequate to explain the earliesthase of the MMN and the subsequent response involves two
F
hologia 45 (2007) 587–597 595
dditional generators in the right and left temporal cortex. Signif-cantly, the McGurk-MMN, with sources localized to bilateraluditory cortices, occurred in the absence of acoustic change.his shows that visual influences on auditory speech perceptionre realized in auditory cortex, in the time frame of perceptualrocessing. Further, the obvious left lateralization of the earlynd late phases of the MMN are consistent with the involvementf language representations.
cknowledgments
We would like to express our sincere appreciation to Bethiggins, Deirdre Foxe and Marina Shpaner for their ever-
xcellent technical help with this study. We are most grateful tor. Renee Beland for her valuable and helpful insights on speecherception. Our thanks also go to two anonymous reviewers forheir careful and constructive comments. This work was sup-orted by grants from the National Institute of Mental HealthMH65350 to JJF and WR) and the National Institute of Neuro-ogical Disorders and Stroke (NS30029 to WR and JJF) and aational Research Service Award (MH68174 to SM).
eferences
ernstein, L. E., Auer, E. T., Jr., Moore, J. K., Ponton, C. W., Don, M., &Singh, M. (2002). Visual speech perception without primary auditory cortexactivation. Neuroreport, 13, 311–315.
esle, J., Fort, A., & Giard, M. H. (2005). Is the auditory sensory memorysensitive to visual information? Experimental Brain Research.
allan, D. E., Callan, A. M., Kroos, C., & Vatikiotis-Bateson, E. (2001). Multi-modal contribution to speech perception revealed by independent componentanalysis: A single-sweep EEG case study. Brain Research Cognitive BrainResearch, 10, 349–353.
alvert, G. A., Brammer, M. J., Bullmore, E. T., Campbell, R., Iversen,S. D., & David, A. S. (1999). Response amplification in sensory-specific cortices during crossmodal binding. Neuroreport, 10, 2619–2623.
alvert, G. A., Bullmore, E. T., Brammer, M. J., Campbell, R., Williams, S.C., McGuire, P. K., et al. (1997). Activation of auditory cortex during silentlipreading. Science, 276, 593–596.
olin, C., Radeau, M., Soquet, A., & Deltenre, P. (2004). Generalization of thegeneration of an MMN by illusory McGurk percepts: Voiceless consonants.Clinical Neurophysiology, 115, 1989–2000.
olin, C., Radeau, M., Soquet, A., Demolin, D., Colin, F., & Deltenre, P. (2002).Mismatch negativity evoked by the McGurk–MacDonald effect: A phoneticrepresentation within short-term memory. Clinical Neurophysiology, 113,495–506.
sepe, V., Karmos, G., & Molnar, M. (1987). Evoked potential correlates ofstimulus deviance during wakefulness and sleep in cat—Animal model ofmismatch negativity. Electroencephalographic Clinical Neurophysiology,66, 571–578.
ehaene-Lambertz, G. (1997). Electrophysiological correlates of categoricalphoneme perception in adults. Neuroreport, 8, 919–924.
alchier, A., Clavagnier, S., Barone, P., & Kennedy, H. (2002). Anatomicalevidence of multimodal integration in primate striate cortex. Journal of Neu-roscience, 22, 5749–5759.
oxe, J. J., Morocz, I. A., Murray, M. M., Higgins, B. A., Javitt, D. C., &
Schroeder, C. E. (2000). Multisensory auditory–somatosensory interactionsin early cortical processing revealed by high-density electrical mapping.Brain Research Cognitive Brain Research, 10, 77–83.oxe, J. J., & Schroeder, C. E. (2005). The case for feedforward multisensoryconvergence during early cortical processing. Neuroreport, 16, 419–423.
5 opsyc
F
F
G
G
G
H
H
J
J
L
M
M
M
M
M
M
M
M
M
M
M
N
N
N
N
P
P
P
R
R
R
R
R
R
S
S
S
S
S
S
S
S
96 D. Saint-Amour et al. / Neur
oxe, J. J., & Simpson, G. V. (2002). Flow of activation from V1 to frontal cortexin humans. A framework for defining “early” visual processing. Experimen-tal Brain Research, 142, 139–150.
oxe, J. J., Wylie, G. R., Martinez, A., Schroeder, C. E., Javitt, D. C., Guilfoyle,D., et al. (2002). Auditory-somatosensory multisensory processing in audi-tory association cortex: An fMRI study. Journal of Neurophysiology, 88,540–543.
iard, M. H., & Peronnet, F. (1999). Auditory–visual integration during multi-modal object recognition in humans: A behavioral and electrophysiologicalstudy. Journal of Cognitive Neuroscience, 11, 473–490.
iard, M. H., Perrin, F., Pernier, J., & Bouchet, P. (1990). Brain gen-erators implicated in the processing of auditory stimulus deviance: Atopographic event-related potential study. Psychophysiology, 27, 627–640.
uthrie, D., & Buchwald, J. S. (1991). Significance testing of difference poten-tials. Psychophysiology, 28, 240–244.
ari, R., Hamalainen, M., Ilmoniemi, R., Kaukoranta, E., Reinikainen, K.,Salminen, J., et al. (1984). Responses of the primary auditory cortex topitch changes in a sequence of tone pips: Neuromagnetic recordings in man.Neuroscience Letters, 50, 127–132.
eslenfeld, D. J. (2003). Visual mismatch negativity. In J. Polich (Ed.), Detectionof change: Event-related potential and fMRI findings (pp. 41–60). Dordrecht:Kluwer Academic Publishers.
avitt, D. C., Steinschneider, M., Schroeder, C. E., Vaughan, H. G., Jr., & Arezzo,J. C. (1994). Detection of stimulus deviance within primate primary auditorycortex: Intracortical mechanisms of mismatch negativity (MMN) generation.Brain Research, 667, 192–200.
ones, J. A., & Callan, D. E. (2003). Brain activity during audiovisualspeech perception: An fMRI study of the McGurk effect. Neuroreport, 14,1129–1133.
iberman, A. M., & Mattingly, I. G. (1985). The motor theory of speech per-ception revised. Cognition, 21, 1–36.
acSweeney, M., Amaro, E., Calvert, G. A., Campbell, R., David, A.S., McGuire, P., et al. (2000). Silent speechreading in the absence ofscanner noise: An event-related fMRI study. Neuroreport, 11, 1729–1733.
arco-Pallares, J., Grau, C., & Ruffini, G. (2005). Combined ICA-LORETAanalysis of mismatch negativity. Neuroimage, 25, 471–477.
cGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature,264, 746–748.
olholm, S., & Foxe, J. J. (2005). Look ‘hear’, primary auditory cortex is activeduring lip-reading. Neuroreport, 16, 123–124.
olholm, S., Martinez, A., Ritter, W., Javitt, D. C., & Foxe, J. J. (2005). Theneural circuitry of pre-attentive auditory change-detection: An fMRI studyof pitch and duration mismatch negativity generators. Cerebral Cortex, 15,545–551.
olholm, S., Ritter, W., Javitt, D. C., & Foxe, J. J. (2004). Multisensoryvisual–auditory object recognition in humans: A high-density electrical map-ping study. Cerebral Cortex, 14, 452–465.
olholm, S., Ritter, W., Murray, M. M., Javitt, D. C., Schroeder, C. E., &Foxe, J. J. (2002). Multisensory auditory–visual interactions during earlysensory processing in humans: A high-density electrical mapping study.Brain Research Cognitive Brain Research, 14, 115–128.
ottonen, R., Krause, C. M., Tiippana, K., & Sams, M. (2002). Processingof changes in visual speech in the human auditory cortex. Brain ResearchCognitive Brain Research, 13, 417–425.
urray, M. M., Molholm, S., Michel, C. M., Heslenfeld, D. J., Ritter, W., Javitt,D. C., et al. (2005). Grabbing your ear: Rapid auditory–somatosensory mul-tisensory interactions in low-level sensory cortices are not constrained bystimulus alignment. Cerebral Cortex, 15, 963–974.
urray, M. M., Wylie, G. R., Higgins, B. A., Javitt, D. C., Schroeder, C.E., & Foxe, J. J. (2002). The spatiotemporal dynamics of illusory con-tour processing: Combined high-density electrical mapping, source analysis,
and functional magnetic resonance imaging. Journal of Neuroscience, 22,5055–5073.usacchia, G., Sams, M., Nicol, T., & Kraus, N. (2006). Seeing speech affectsacoustic information processing in the human brainstem. Experimental BrainResearch, 168, 1–10.
S
hologia 45 (2007) 587–597
aatanen, R. (2001). The perception of speech sounds by the human brain asreflected by the mismatch negativity (MMN) and its magnetic equivalent(MMNm). Psychophysiology, 38, 1–21.
aatanen, R., & Alho, K. (1995). Mismatch negativity—A unique measure ofsensory processing in audition. International Journal of Neuroscience, 80,317–337.
aatanen, R., Lehtokoski, A., Lennes, M., Cheour, M., Huotilainen, M.,Iivonen, A., et al. (1997). Language-specific phoneme representationsrevealed by electric and magnetic brain responses. Nature, 385, 432–434.
ovak, G., Ritter, W., & Vaughan, H. G., Jr. (1992). Mismatch detectionand the latency of temporal judgements. Psychophysiology, 29, 398–411.
azo-Alvarez, P., Cadaveira, F., & Amenedo, E. (2003). MMN in the visualmodality: A review. Biology Psychology, 63, 199–236.
ekkola, J., Ojanen, V., Autti, T., Jaaskelainen, I. P., Mottonen, R., Tarkiainen,A., et al. (2005). Primary auditory cortex activation by visual speech: AnfMRI study at 3T. Neuroreport, 16, 125–128.
icton, T. W., Alain, C., Otten, L., Ritter, W., & Achim, A. (2000). Mismatchnegativity: Different water in the same river. Audiology Neurootology, 5,111–139.
inne, T., Alho, K., Alku, P., Holi, M., Sinkkonen, J., Virtanen, J., et al. (1999).Analysis of speech sounds is left-hemisphere predominant at 100–150 msafter sound onset. Neuroreport, 10, 1113–1117.
itter, W., Deacon, D., Gomes, H., Javitt, D. C., & Vaughan, H. G., Jr. (1995).The mismatch negativity of event-related potentials as a probe of transientauditory memory: A review. Ear Hearing, 16, 52–67.
itter, W., Sussman, E., Molholm, S., & Foxe, J. J. (2002). Memory reactiva-tion or reinstatement and the mismatch negativity. Psychophysiology, 39,158–165.
osburg, T., Trautner, P., Dietl, T., Korzyukov, O. A., Boutros, N. N., Schaller,C., et al. (2005). Subdural recordings of the mismatch negativity (MMN) inpatients with focal epilepsy. Brain, 128, 819–828.
osenblum, L. D., & Saldana, H. M. (1992). Discrimination testsof visually influenced syllables. Perceptive Psychophysics, 52, 461–473.
oss, L., Saint-Amour, D., Leavitt, V., Javitt, D., & Foxe, J. J. (in press). Do yousee what I’m saying? Optimal visual enhancement of speech comprehensionin noisy environments. Cerebral Cortex.
aint-Amour, D., Molholm, S., Sehatpour, P., Ross, L., Mehta, A., Schwartz,T., et al. (2005). Multisensory mismatch negativity processes in temporalcortex: A human intracranial investigation of the McGurk illusion. NewYork: Cognitive Neuroscience Society.
ams, M., Aulanko, R., Hamalainen, M., Hari, R., Lounasmaa, O. V., Lu, S.T., et al. (1991). Seeing speech: Visual information from lip movementsmodifies activity in the human auditory cortex. Neuroscience Letters, 127,141–145.
ams, M., Hamalainen, M., Antervo, A., Kaukoranta, E., Reinikainen, K.,& Hari, R. (1985). Cerebral neuromagnetic responses evoked by shortauditory stimuli. Electroencephalographic Clinical Neurophysiology, 61,254–266.
cherg, M., & Berg, P. (1991). Use of prior knowledge in brain electromagneticsource analysis. Brain Topography, 4, 143–150.
chroeder, C. E., & Foxe, J. J. (2002). The timing and laminar profile of converg-ing inputs to multisensory areas of the macaque neocortex. Brain ResearchCognitive Brain Research, 14, 187–198.
chroeder, C. E., & Foxe, J. J. (2005). Multisensory contributions to low-level ‘unisensory’ processing. Current Opinion on Neurobiology, 15, 454–458.
harma, A., & Dorman, M. F. (1999). Cortical auditory evoked potential cor-relates of categorical perception of voice-onset time. The Journal of theAcoustical Society of America, 106, 1078–1083.
imson, R., Vaughan, H. G., Jr., & Ritter, W. (1977). The scalp topography of
potentials in auditory and visual Go/NoGo tasks. ElectroencephalographicClinical Neurophysiology, 43, 864–875.tekelenburg, J. J., Vroomen, J., & de Gelder, B. (2004). Illusory sound shiftsinduced by the ventriloquist illusion evoke the mismatch negativity. Neuro-science Letters, 357, 163–166.
opsyc
S
S
S
T
D. Saint-Amour et al. / Neur
umby, W. H., & Pollack, L. (1954). Visual contribution to speech intelligibilityin noise. The Journal of the Acoustical Society of America, 212–215.
ummerfield, Q., & McGrath, M. (1984). Detection and resolution of
audio–visual incompatibility in the perception of vowels. Quarterly Journalof Experimental Psychology Section A, 36, 51–74.ussman, E., Ritter, W., & Vaughan, H. G., Jr. (1998). Attention affects the orga-nization of auditory input associated with the mismatch negativity system.Brain Research, 789, 130–138.
W
hologia 45 (2007) 587–597 597
ervaniemi, M., Schroger, E., & Naatanen, R. (1997). Pre-attentive process-ing of spectrally complex sounds with asynchronous onsets: An event-related potential study with human subjects. Neuroscience Letters, 227, 197–
200.inkler, I., Lehtokoski, A., Alku, P., Vainio, M., Czigler, I., Csepe, V., et al.(1999). Pre-attentive detection of vowel contrasts utilizes both phonetic andauditory memory representations. Brain Research Cognitive Brain Research,7, 357–369.