eeg evidence for audiovisual speech perception deficits in...

EEGEvidenceforAudiovisualSpeechPerceptionDeficitsinParkinson’s

Disease:AResearchProposal

GeenaBruno

Introduction:

EEGstandsforelectroencephalogram,has32electrodesontheHydroCelGeodesic

SensorNet(HCGSN),andisavailablewithdensitiesof32,64,128,or256channelnet.Itis

connectedtoanamplifierandamplifiedelectricalsignalscomingfromthebrainare

recorded.Researchersareinterestedinthebrain'sresponsetoaspecificstimuli(Khamis-

Dakwar,2012).EEGdoesnotshowwhereactivationoriginates,butrather,thetimingof

howaperson’sbrainrespondstocertainstimuliinataskpresented.Anevent-related

potential(ERP)istheaverageelectricalactivitycorrelatedwiththespecifictypeof

stimuluspresentation;researchershavetosegmentpartsofEEGtogetERPs.ERP

componentsaredefinedintermsofacombinationofpolarity,latency,andscalp

distribution,andaresensorycognitiveandmotorprocessesrelatingtobehaviorand

thought(Kappenman&Luck,2011).Mismatchnegativity(MMN)isanERPcomponentthat

peaksto150–250millisecondswhenthereisachangeofstimulus(Khamis-Dakwar,2012);

forexample,ifthestimulusisanoddballparadigm,havingastandardanddeviantsound

(ta,ta,ta,da,ta).MMNalsohasthelargestamplitudeatfrontalandcentralelectrodesites

oftheskull,andthepatientdoesnotneedtorespondinordertogetanMMNresponseon

theEEG(FonaryovaKeyet.al,2005).Thisisgreatforbabieswhocannotrespondontheir

own,orforpatientswhohavecognitivedeficits.

Motorspeechdisordersaredifficultiesrelatingtoproblemsofmovement,resulting

fromaneurologicaldisorderorinjurythataffectsmotorplanning,programming,

coordination,timing,andexecutionusedforspeech;thesedisorderscanalsoaffect

respiration,resonation,phonation,andarticulation(OwensJr.et.al,2014)Thebasal

ganglia(BG)isasubcorticalareaofthebrainthatregulatesmotorfunctioningand

maintainspostureandmuscletone.Ithasdirectandindirectpathways,andifdamagedit

resultsineitherslowedmovementsasinParkinson’sdiseaseorinvoluntarymovementsas

inHuntington’sdisease(OwensJr.et.al,2014).TheBGisalsoinvolvedinonline

monitoringofauditoryfeedbackduringspeech(DeKeyseret.al,2016).Damagetothe

neuralcircuitryconnectingtheprefrontalandBGareasofthebraincausesanincreasein

syntacticandvoiceonsettimingerrorsonsingle-wordproductiontasks(Walsh&Smith,

2011).

Dysarthriaisagroupofspeechdisordersresultingfromdisturbancesinthecentral

andperipheralnervoussystemsthatcontrolmusclesofspeechproduction.Specifically,in

Parkinson’sdiseaseitisdescribedashypokineticdysarthria.Hypokineticdysarthriaisa

BGcontrolcircuitpathologyevidentinvoice,articulation,prosody.Ithaseffectsofrigidity,

difficultyinitiatingmovement,reducedforceandrangeofmotioninarticulators,slowbut

sometimesfastrepetitivemovementsonspeech,reducedvocalloudnesswithharsh–

hoarsequality,slowspeakingratewithburstofrapid-firearticulation,excessiveinlong

pauses,prolongedsyllables,mono-loudness,andreducedphonationtime(Freed,2000;

Owenset.al,2014).IndividualswithParkinson’sdiseasealsohavehigherlipaperture

variability,portrayinglessconsistentoralmotorcoordinationthanthecontrolparticipants

(Walsh&Smith,2011).Sensorimotorintegrationdeficitsmayresultfromdysfunctionsof

feedforwardandfeedbackcontrolinpeoplewiththisdisease(DeKeyseretal,2016).

Importantly,peoplediagnosedwithParkinson’sdiseasehavepoortemporaldiscrimination

fortactile,auditory,andvisualstimuli,alsoknownasproprioceptivedeficits(Duffy,2005),

whichwillhaveaneffectonauditory-visualintegration.

Researchshowsthatmultisensoryintegrationincreasesspeedofdetection,

enhancessensorysensitivity,andcorrectlyidentifiesevents:“Multisensoryintegrationis

mosteffectiveandthereforeelicitmaximalbehavioralenhancementswhenlessintenseor

weakandambiguousindividualstimuliareapplied”(Freiherret.al,2013).Thisisnotonly

evidentinpeoplewithadisorderordisease,butalsoinnormalhearingpeople;whenyou

areatabarorloudrestaurant,yourelyonvisualcueslikelookingatsomeone'smouthto

helpyouunderstandwhatthepersonissayingtoyou.ThisisalsocalledthePrincipleof

InverseEffectiveness;multisensoryintegrationhelpscounteracttheconsequencesof

unisensorydeterioration(Freiherret.al,2013).Also,substantialworkinthespeech

productionandspeechperceptionliteraturesupportsthenotionthatspeechperception

andproductionhaveabidirectionalinfluenceononeanother(DeKeyseret.al,2016).To

addtothis,visualarticulatoryinformationintegrateswiththemotorsystemduringspeech

production(Veneziaet.al,2016).

OnewaytoelicitanMMNresponseistousetheMcGurkeffect,whichiswhatis

beingusedinthisresearchproposal.TheMcGurkeffectisaperceptualillusionthatshows

theinfluenceofvisualspeechinformationontheperceptionofspeech(Cienkowski&

Carney,2002).Whatthepatientseesinfluenceswhatthepatienthears(Franciscoet.al,

2017).TheMcGurkEffectshowsthatmannerofarticulationandvoicingaretransmitted

mostefficientlyfromtheauditoryportionofthestimulusandplaceofarticulationis

transmittedbestbythevisualportion(Hessler,et.al,2013).

Thisstudywillinvestigatethepossibledifferencesinaudiovisualspeechperception

processinginindividualswithParkinson’sdiseaseandhealthyindividualsasindexedby

theMMNcomponentrecordedusinganEEGnet.

Methods

A. Participants

Therewillbe10individualsinthePDgroupand10healthyindividualsinthe

controlgroup.HealthyindividualsarematchedwithpatientswithPDbygenderandage.

PatientswithPDwerediagnosedbyaneurologistusingtheUnifiedParkinson’sDisease

RatingScale.Hypokineticdysarthriawasdiagnosedbyspeechlanguagepathologistsusing

theSentenceIntelligibilityTestandArizonaBatteryforCommunicationDisordersof

Dementia.Eachparticipantreceivedaspeechevaluationthatincludedacasehistoryofhis

orherspeechandlanguage,anoralmechanismexamination,perceptualjudgmentsof

respiration,phonation,resonance,articulation,andprosodyduringsustainedphonation,a

diadochokinetictask,andreadingandspontaneousspeech.Participantinterviewsand

reviewofPDparticipants’medicalrecordswereusedtodeterminethatallparticipantshad

negativehistoriesforheadtraumaandforpre-existingcommunication,memory,

neurologic,orpsychiatricproblems.

B.EEG

High-densityEEGwillberecordedfromparticipantswhiletheyareexposedto

experimentalstimuli.IntheAVMcGurkparadigm,participantsviewastandard

presentationofcongruentauditoryandvisualinformation(e.g.articulationof/ba/)

interspersedwithadeviantpresentationofincongruentvisual(e.g.articulationof/ga/)

dubbedovertheoriginalaudiostimulus.InEEGexperimentsutilizingtheMcGurkeffect,

thecongruentaudiovisualpresentationof/ba/ispresentedrepeatedlyasthestandard

stimulus.InfrequentpresentationoftheMcGurkstimulus,theincongruentaudio

presentationof/ba/pairedwithvisualpresentationof/ga/,generatestheMMN(Saint-

Amouret.al,2007).Inthisparadigmthepresentedaudioisconsistently/ba/andonlythe

visualstimuluschanges.

AnadditionalAVconditionutilizinganinverseMcGurkdeviantwillbeimplemented

toexploretheeffectsofincongruenceasmediatedbymodality.TheinverseMcGurk

condition(AVinverse,labeledAI)utilizescongruentpresentationofauditoryandvisual

/ba/asastandardstimuluswithachangetoauditory/ga/,whilemaintainingvisual/ba/

forthedeviantstimulus.Healthyindividualsperceivetheauditoryaspectofthestimulus,

ordonotfuseresponsesintoasinglepercept,butratherperceivebothsounds

simultaneously(/b-ga/).TheAIconditioninthepresentstudywouldprovideadirect

contrastwiththeAVcondition,elucidatingtheinfluenceofmodalityonincongruent

deviancedetection.

Avisual-only(VO)controlconditionisalsonecessarytoensurethatthederived

MMNisduetoAVintegrationprocesses(visualinformationchangingtheauditorypercept)

ratherthanresponsestochangeinvisualstimulus(SaintAmouretal.,2007).

Thedatawillbeanalyzedwithingroupsandbetweenconditionsaswellasacross

conditionsandbetweengroups,withinthetimewindowsofinterest,toevaluatewhether

thereweresignificantdifferencesinMMNpeakamplitude.

Condition Standard80%

Deviant20%

DeviantPercept

AV auditory/ba/andvisual/ba/

auditory/ba/andvisual/ga/

/da/

AI auditory/ba/andvisual/ba/

Auditory/ga/andvisual/ba/

/ba/or/b-ga/

VO Visual/ba/

Visual/ga/ /ga/

Figure 1

Thisexperimentwillutilizea128-channelEEGsystem(ElectricGeodesics,Inc.).The

128electrodesarearrangedinapredictablegeodesicpositionrelativetooneanotherina

sensornet.Theelectrodesareheldtogetherbyafineelastomerandcontainasilver

chloride-platedcarbonfiberembeddedinaplasticsubstrate.Eachelectrodehassponge

insertsthataresoakedinanelectrolytesolutionofpotassiumchlorideandwaterbefore

usetoensureoptimumconductivity.Eachparticipantwillwearanetthatfitsthe

circumferenceoftheirhead,whichwillhavebeenmeasuredbytheresearcher.Inaddition,

measurementsaretakentolocatethevertexoftheskullinordertoaccuratelyplacethe

net.Afterthenetisplacedontheparticipant,itisidealtohavetheindividualsitinachair

inasound-attenuatedroom.Theparticipantsitsinfrontofacomputermonitorthat

presentsthestimuliandthesensornetconnectstoacalibratedamplifier.

Figure2

C.Stimuli

Theauditory-visual(AV)stimulithroughoutthisexperimentwerecreatedtoevoke

theMcGurkMMN.Stimuliwerecreatedbydigitalrecordingofafemalenativespeakerof

AmericanEnglishsaying/ba/and/ga/.Digitalvideo(CanonVixiaHFR50)andcorrelating

audio(BlueMicYetiPro,www.bluemic.com)wererecordedatasamplingrateof44.1KHz

andaframerateof24images/second;theylatertrimmedforatotaldurationof300ms

pertoken.Theplacesofarticulationfor/ba/and/ga/differgreatly.Sincetheauditory

distinctioninthisparadigmdependsonplaceofarticulation,videosegmentsbeganinthe

preparatoryarticulatoryposition—closedlipsfor/ba/andopenlipsfor/ga/.Thespeaker

wasinstructedtoopenhermouthminimally.Visualinspectionofvideosegmentsensured

thatjawcoordinationwasconsistentbetweenthe/ba/and/ga/videosegments.Video

framewascroppedusingAppleiMovietorevealonlythespeaker’smouthinorderto

constrainthevisualpresentationandtoavoideye-movementartifactsduringEEG

recording.TheaudiotrackswereseparatedfromthevideoandeditedinPraatwith50ms

rise/falltoavoidclickartifactsintherecording,andamplitudesnormalizedto70dB.The

vowelsegmentfromone/ba/recordingwasremovedinPraatandthespliced/a/segment

wasusedwiththeonsetsforboth/b/and/g/,sotheonlydifferenceintheaudioisthe

consonantsegment.Theaudiotrackfor/ba/wasdubbedoverthevideotracksofboth

/ba/and/ga/,creatingcongruent(auditory/ba/,visual/ba/)andincongruentMcGurk

(auditory/ba/,visual/ga/)AVstimuliinAppleiMovie.OnsetoftheAVstimulibeginswith

thecontrastivearticulatoryposition,closedlipsfor/ba/andopenmouthfor/ga/.TheAI

conditionwassimilarlycreated,withaninverseMcGurkdeviant(auditory/ga/dubbed

ontovisual/ba/).

Visual-only(VO)stimuliconsistedofthesame300ms/ba/and/ga/videotracks

withaudioremoved.EEGepochsweresegmentedtocoincidewiththeonsetofthe

auditorycomponentofstimulustomorespecificallyexaminetheneurophysiological

responsetothevisualinfluenceonauditoryperception(Hessleretal.,2013).

Figure 3

a. OddballParadigm

StimulifortheAV,AI,andVOconditionswillbepresentedinanoddball

paradigm.Thisiswhenthestandardswerepresentedfor80%oftrialsandthedeviantsfor

20%oftrials.Eachconditionhas450totaltrials,with360standardsand90deviants.

Stimuliwillbepresentedpseudorandomlyinordertoensurethatatleasttwostandards

camebeforeeverydeviantandthatdeviantswerenotplayedconsecutively.The

interstimulusinterval(ISI)forallconditionsis600ms.Presentationofeachconditionis

counterbalancedbetweenparticipantswithineachgroup.

Figure 4

B. Experiment/BehavioralMeasuresandProcedures

a.BehavioralMeasures

1.TheSentenceIntelligibilityTest(SIT)has1,100sentences(100

sentencesrangingfromfiveto15wordsinlength).Participantswillbeevaluatedusing10

to15sentencestotestintelligibilityandwillbeaskedtorepeatsentencesreadbythe

researcher.Thiswillhelptomakesureparticipantshaveahighintelligibilityrateto

understandwhattheyneedtodothroughouttheexperimentandtheinstructionsthey

needtofollow(Yorkston,Beukelman,&Tice,1996).

2.TheArizonaBatteryforCommunicationDisorders(ABCD)isa

comprehensiveassessmentandscreeningofdementia.Itincludes14subteststhatevaluate

verbalepisodicmemory,linguisticcomprehension,visuospatialconstruction,etc.The

ABCDwasstandardizedonAlzheimer'sandParkinson’sdiseasepatients(Bayles&

Tomoeda,1991).Itisimportantthatpatientshaveatleast80%ormoreontheABCDin

ordertoparticipateinthisstudy.

c.Participationintheexperimentshouldinvolvetwovisits:

Visit1:

1. Participantswillbeshownthelab,equipment,andproceduresbyan

experiencedspeech-languagepathologistwithexperiencediagnosingand

treatingindividualswithPD.CaregiverswillaccompanypatientswithPDand

questionswillbeencouragedthroughoutthisfamiliarizationperiod.

2. Participantswillbepresentedwithaconsentformthattheparticipant,

caregiver(ifnecessary),andresearcherhavetosign.Thespeech-language

pathologistgivestimeforparticipantstoreadtheformandalsoprovidesa

verbalexplanation.Last,theparticipantandcaregiverareinformedofany

riskfactorsandoftheiroptiontowithdrawatanytimeduringthe

experiment.

3. Followingcompletionofconsentprocedures,participantsparticipatedin

administrationofstandardizedtests

4. Visit1willvaryintimedependingoneachparticipant’sneeds

Visit2:

1.Afterassessingiftheparticipantswereappropriateforthestudy,the

participantswillbeinvitedbacktocompletetheexperimentaltasks.Onthesecondlab

visit,aconsentform(thesameasforvisit1)willbepresentedagainandtherewillbe

opportunitiesforquestions.

2.Hearingmustbescreenedat500,1000,2000,and4000Hzwithinarange

of40–20dBbeforethestartoftheexperiment.

3.Thecircumferenceoftheheadwillbemeasuredtochoosethecorrectnet

sizeandthevertexwillbemarkedontheskulltoensurethatalltheelectrodesare

correctlyplaced.

4.Theparticipantwillbeseated80cmfromthecomputermonitor,ina

sound-attenuatedroom.Theparticipantswillbegiveninsertheadphones;thered-

codedinsertgoesintotherightearandtheblueintheleft.Avideocamerawillbein

theroomtogivetheresearchervisualinformationabouttheparticipantduringthe

experiment.Theparticipantwillberemindedtosignalatanytimeduringthe

experimentifheorshedoesnotwishtocontinue.Theamplifiermustbechecked

andcalibratedbeforethenetisconnected,andimpedances(lossofsignalbetween

scalpandsensor)mustbemeasured.Inordertoimproveimpedances,the

electrodesneedtobeadjustedasnecessarysothattheyareingoodcontactwiththe

participant’sscalp.

5.ExperimentalEEGtaskswillbepresentedinrandomorderand

counterbalancedacrossparticipants.Taskswillbepresentedinshortrunsofless

than10minutestominimizefatigueandreducehabituationthatinterfereswith

MMNelicitation.Participantswillbeencouragedtotakeshortbreaksbetweenruns.

6.Aftercompletingtheexperimentaltasks,thespeech-languagepathologist

willremovetheEEGnetandtheparticipantwillbeaskedquestions.Visit2maylast

approximately60minutesorlonger,dependingonparticipant’sneedsandnumber

anddurationofbreaks.

ProposedResults:

Speech-productionimpairmentinPDmayimpacthigher-levellinguisticand

cognitiveprocessingasindexedbytheMcGurkMMN.Thecomparisongroupisexpectedto

showtheclassicMMNinresponsetotheMcGurkeffectintheAVconditionbutnottheAI

condition.ItisanticipatedthatresponsesinthePDgroupwillbemorevariablethaninthe

comparisongroup.ThismeansthatindividualswithPDnotgettinganearlyautomatic

fusionMMNresponseintheAVmayhaveadeficitinaudiovisualintegration.Weexpectthe

PDgroup’sresponsetobesimilarintheAVandAIconditions,indicatingthattheycan

detecttheoddballdeviantbutdonotexperienceAVintegration.ResultsforVOstimuliwill

beconsistentwithinthePDgroupandcomparisongroup,demonstratingthattheresults

arenotduetosimplechangedetectionofthevisualstimulus.

Resultswillalsorevealasmalleramplitudeandlongerlatencyresponsein

individualswithPDascomparedtocontrols.ThismeansthatpatientswithPDwillhavea

slowerresponsetimeandhaveatendencytowardlatergoingresponse.Thiswillchange

theirMMNresponsetolooklikeaP300response(anotherERPcomponentderivedfrom

EEGmeasurements).TheseresultsexplainthatindividualswithPDhaveadeviancein

detectionofstimuliandaninsensitivitytomodality.

Figure 5 (examples of MMN and P300 responses)

Figure 6 (Jahshan et. al, 2012) MMNisrecordedatlargestoverthefrontocentralelectrodes,regardlessofwhere

theprocessishappening.Theimagebelowisarepresentationofthisimplication.

Figure7

DiscussionandClinicalImplication:

TheresultsoftheproposedstudywillhelpusunderstandifdeficitsinAV

integrationarepartofParkinson’sdiseaseoraresultoftheongoingexperienceofliving

withaspeechimpairment.Asstatedinliterature,patientswithPDhaveproprioceptive

deficitsthatcausepoortemporaldiscriminationfortactile,auditory,andvisualstimuli

(Duffy,2005).ThiscanhaveaneffectonAVintegration,andtherapymaynotchangethese

deficits.Also,inmanytherapytechniquesforpatientswithspeech,language,andhearing

disorders,speechtherapistsemphasizetheimportanceoftheprincipleofinverse

effectiveness,orusingmultiplesensestocounteractthenegativeeffectsofweakeningone

sensorysystem(Freiherr,2013).Whilewetendtousevisualarticulatorycuesduring

speechtherapy,wedon’tknowifvisualarticulatoryinformationisfacilitativeor

overloadingthesysteminpatientswithPD.

References

Bayles,K.A.,&Tomoeda,C.K.(1991).ArizonaBatteryforCommunicationDisordersofDementia.

Tucson,AZ:CanyonlandsPublishing.

Cienkowski,K.M.,&Carney,A.E.(2002,November).Auditory-visualspeechperceptionand

aging.EAR&HEARING,23(5),439-449.

DeKeyser,K.,Santens,P.,Bockstael,A.,Botteldooren,D.,Talsma,D.,DeVos,S.,…DeLetter,

M.(2016,October).Therelationshipbetweenspeechproductionandspeechperception

deficitsinParkinson’sdisease.JournalofSpeech,Language,andHearingResearch,59,915-

931.

Duffy,J.R.(2005)MotorSpeechDisorders:Substrates,DifferentialDiagnosis,andManagement

(2nded.).ElsevierMosby.

FonaryovaKey,A.P.,Dove,G.O.,&Maguire,M.J.(2005).LinkingBrainwavestotheBrain:An

ERPPrimer.DEVELOPMENTALNEUROPSYCHOLOGY,27(2),183-215.DOI:

10.1207/s15326942dn2702_1

Francisco,A,A.,Jesse,A.,Groen,M.A.,&Mcqueen,J.M.(2017,January).Ageneral

audiovisualtemporalprocessingdeficitinadultreaderswithdyslexia.Journalof

Speech,Language,andHearingResearch,60,144-158

Freed,D.(2000).Motorspeechdisorders:diagnosisandtreatment.SanDiego,CA:Singular

ThomsonLearning.

Freiherr,J.,Lundstrom,J.N.,Habel,U.,&Reetz,K.(2013,December).Multisensoryintegration

mechanismsduringaging.FrontiersinHumanNeuroscience,7,1-6.

Hessler,D.,Jonkers,R.,Stowe,L.,&Bastiaanse,R.(2013).Thewholeismorethanthesumofits

parts–AudiovisualprocessingofphonemesinvestigatedwithERPs.Brainand

language,124(3),213-224.

Kappenman,E.S.,&Luck,S.(2011).ERPComponents:TheUpsandDownsofBrainwave

Recordings.TheOxfordHandbookofEvent-RelatedPotentialComponents,3-30.

doi:10.1093/oxfordhb/9780195374148.013.0014

Khamis-Dakwar,R.(2012).Translationaldisplayofneurophysiologicalinvestigations.In

CommunicationSciencesandDisorders.In:Goldfarb,R.(Ed.).TranslationalSLP/A:

EssaysinhonorofDr.SadanandSingh.PluralPublishing,Inc.IntroductionToERP(2012)

Owens,R.E.,Jr.,Metz,D.E.,&Farinella,K.A.(2014).IntroductiontoCommunication

Disorders:ALifespanEvidence-BasedPerspective,FifthEdition.(5thed.).Pearson.

Saint-Amour,D.,DeSanctis,P.,Molhoim,S.,Ritter,W.,&Foxe,J.J.(2007,February).Seeing

voices:High-intensityelectricalmappingandsource-analysisofthemultisensory

mismatchnegativityevokedduringtheMcGurkIllusion.Neuropsychologia,45(3),

587-597

Venezia,J.H.,Fillmore,P.,Matchin,W.,Isenberg,A.L.,Hickock,G.,&Fridriksson,J.(2016).

Perceptiondrivesproductionacrosssensorymodalities:Anetworkforsensorimotor

integrationofvisualspeech.NeuoroImage,(126),196-207.

Walsh,B.,&Smith,A.(2011,June).Linguisticcomplexity,speechproduction,and

comprehensioninParkinson’sDisease:BehavioralandphysiologicalIndices.Journalof

Speech,Language,andHearingResearch,54,787-802.

Yorkston,K.M.,Beukelman,D.,&Tice,R.(1996).SentenceIntelligibilityTestforWindows

[Computersoftware].Lincoln,NE:TiceTechnology.