Multisensory Integration in the Brain

Paul J. Laurienti, M.D., Ph.D.

Department of Radiology

McGurk Illusion

http://www.wam.umd.edu/~vvw/McGurk.htm

Why Multisensory?

1. Survey of Language Knowledge LOUISA C. MOATS, Ed.D 2. VALERIE G. TUCKER, M.Ed., CALT President, The Alliance for Accreditation and Certification of Multisensory Structured Language Education.3. Panel of Experts Moderator: VALERIE G. TUCKER, M.Ed. A panel of experts will address the National Reading Panel Competencies and give teachers practical

ideas for implementing multisensory structured activities for each identified area.4. Capturing Elusive Images: Practical Strategies for Building Orthographic Recall MICHELE BERG, Ph.D5. Teaching the Teachers: Effective Models for Colleges and Universities6. Dyslexia and the English Language Learner ELSA CÁRDENAS-HAGAN, Ed.D,7. Beyond Basic Phonics: The MultiSyllabic “Missing Link” to Literacy POLLY BAYRD, M.A and KRISTIN JOHNSON, M.A.8. Strengthening Vocabulary with Direct Instruction and Morphology BRYAN K. CLAYBACK, B.A. et al.9. Expressive Phonological Impairment and Spelling: Impact of a Multisensory Intervention DEBORAH USREY, Ed.D10. Teaching Non-Literate and Brain-Injured Adults Successfully with Sounds in Syllables JENNIE LYNCH, B.A. and SANDRA DILLON, M.A.11. Effects of a Multisensory-Based Comprehensive Reading District Reform Model PAUL WORTHINGTON12. The ABCs of Grammar Instruction: Keeping it Structured, Sequential and Multisensory PHYLLIS C. HUTSON13. Hands-On, MultiSensory Reading Instruction GERI MARSHALL MOHLER, Ed.D. and BETH ANN BADER-PAETSCHOW, Ed.D.14. The Association Method: Successful Strategies for Oral and Written Language Intervention MAUREEN K. MARTIN, Ph.D.15. Make it Fun! Make it Challenging! Make it Multisensory! SUE GNAGY FEGAN, M.S., F/AOGPE and S C OTT MURRILL, M.A.16. Considerations of Private Practice ELLEN J. LINSKY, Ed.D., M/AOGP17. Implementing Undergraduate and Graduate Multisensory Structured Language Teacher Training Programs: Two Examples KEICHER and SCHNEIDER18. The Student Notebook: An Important Component of Multisensory Structured Language Instruction. KAREN K. LEOPOLD, M.Ed.19. The Roots of Algebra Revealed: Concepts Before Procedures MARILYN ZECHER, M.A., C.A.T.20. Preparing Preschool Children to Read: A Randomized Trial Study MARCIA DAVIDSON, Ph.D., JENNAE BULAT, M.A., M.B.A., and LAUREL COCO, M.A.21. Part B: The Painted Essay: A Multisensory Approach for Essay Writing and Assessment CARLENE BADEAU, B.A.22. Part A: Reading Remediation in the Middle School Through Explicit Phonics Instruction GERI MARSHALL MOHLER, Ed.D23. The Slingerland Multisensory Approach to Reading BONITA COLONY and SUE WINTERS-BROW N, M.Ed.24. Teaching the Teachers: Effective Models for Colleges and Universities MAUREEN K. MARTIN, Ph.D. et al25. Fun and Games in an OG Lesson COREY ZYLSTRA26. Does Multisensory Instruction Meet the Criteria for Scientifically-Based Reading Research (SBRR)? DEBORAH F. KNIGHT, Ph.D. and MARGIE GILLIS, Ph.D27. Building Bridges Through Collaboration: Reading Disabilities Prevention and Intervention DEBORAH MIDDEL–KAT Z E N M E Y E R, M.A. et al28. Prevention and Early Intervention: Multisensory Instruction in K-3 General Education Classrooms ROBIN CARLO, M.A29. Slingerland Multisensory Approach for Upper Elementary through High School Students S U SAN HEINZ, Ph.D.

Why Multisensory?

We live in a “multisensory” world

• Information can be processed on a modality-specific basis

• Information converges and is integrated in the brain

• Performance enhancement is larger for multisensory than unisensory stimuli

Color Discrimination Task

(Multisensory)

(Unisensory)

Multisensory Enhancement

0

100

200

300

400

500

600

700

Auditory MultisensoryVisual

*

Cross-modal

0

100

200

300

400

500

600

700

Visual 1 Visual 2 Dual-Visual

ns

Dual-visual

Factors Modulating Multisensory Integration

• Temporal/spatial coincidence

• Semantic congruence

• Sensory attention– Selective– Divided

Temporal Coincidence

Hairston et al. Exp Brain Res (2005) 166: 474–480

+

Temporal Coincidence

Hairston et al. Exp Brain Res (2005) 166: 474–480

+

Temporal Coincidence

Hairston et al. Exp Brain Res (2005) 166: 474–480

Factors Modulating Multisensory Integration

• Temporal/spatial coincidence

• Semantic congruence

• Sensory attention– Selective– Divided

0

100

200

300

400

500

600

700

Semantic Congruence

Visual Multisensory-Congruent

Multisensory-Incongruent

0

100

200

300

400

500

600

700

0

100

200

300

400

500

600

700

Factors Modulating Multisensory Integration

• Temporal/spatial coincidence

• Semantic congruence

• Sensory attention– Selective– Divided

Selective Attention – Multisensory Processing

0.6

0.8

1

1.2

1.4

1.6

1.8

2

2.2

300 350 400 450 500 550 600 650 700 750 800 850

RT (ms)

Ca

pa

cit

y

divided

attend_vis

attend_aud

integration

Selective Attention – Multisensory Processing

• Selective attention suppresses multisensory integration

Unpublished data removed

Behavioral studies

• Redundant multisensory benefits >> redundant unisensory benefits

• Multisensory Integration is modulated stimulus features

– Timing

– Location

– Meaning

• Congruent – enhancements

• Incongruent- decrements

• Sensory attention modulates multisensory integration

Multisensory Brain Regions

• Superior temporal sulcus– STS (BA 22)

• Inferior temporo-occipital – BA 19/37

• Angular gyrus– BA 39/40

• Cingulate gyrus– BA 32/9/10

Brodmann Areas

http://spot.colorado.edu/~dubin/talks/brodmann/brodmann.html

Superior Temporal Sulcus

Calvert et al, Current Biology 2000, 10:649–657

Beauchamp et al. – Nature Neuroscience 2004, 1190-1192

Atteveldt et al. Neuron 2004 43, 271–282.

Inferior Temporo-occipital

L. Cohen et al. NeuroImage 23 (2004) 1256–1270

Hashimoto et al, Neuron, Vol. 42, (2004) 311–322

Beauchamp et al, Neuron, Vol. 41, 809–823, March 4, 2004,

Angular and Cingulate Gyri

Saito et al Cereb Cortex. 2005, 15:1750-60 Nonlinguistic

Linguistic

Laurienti et al. (2003)

de Zubicaray et al. (2002)

Unisensory Brain Regions

Brodmann Areas

http://spot.colorado.edu/~dubin/talks/brodmann/brodmann.html

• Do the unisensory cortices interact?

• How does attention modulate unisensory activity?

• Are unisensory interactions altered in dyslexia?

Unisensory Brain Regions

• Cross-modal deactivations are modulated by selective attention

Unpublished data removed

AuditionVision

Laurienti et al. (2002)

Phoneme Recognition Task

• Dyslexic readers do not deactivate visual cortex during an auditory processing task

Unpublished data removed

Imaging Multisensory Processes

• Multiple multisensory cortical regions

• Factors that modulate behavior (coincidence, congruence, attention) also modulate fMRI

• Unisensory and multisensory regions are closely situated

• Cross-modal deactivations

• Altered multisensory processing in dyslexia