Neural Bases of Syntax and Semantics: State of the Art

EALing minicourse

Liina Pylkkänen

Departments of Linguistics and PsychologyNew York University

A theoretically grounded initial to-do list for theCognitive Neuroscience of Syntax and Semantics

• Morphology– What neural activity decomposes complex words into pieces

and what activity composes the parts back to together?

• Syntax– What neural mechanisms combine the elementary building

blocks of syntax into complex structural representations?

• Semantics– What neural mechanisms combine the elementary building

blocks of semantics into complex semantic representations?

The questions main stream Cognitive Neurosciencehas instead focused on

• Morphology– What neural activity distinguishes irregular from regular

inflection?– How does the brain react to morphological ill-formedness?

For regulars vs. irregulars?

• Syntax– How does the brain react to syntactic ill-formedness?– What brain regions show more activity for complex

structures, like ones involving long-distance dependencies?

• Semantics– How does the brain react to semantic ill-formedness?

• Example: He spread the warm bread with socks.• Contrast to what “semantic” means in Linguistics.

These four lectures

• Morphology (Days 1 & 2)– What neural activity decomposes complex words into pieces [and what

activity composes the parts back to together]?

• Syntax (Days 2 & 3)– What neural mechanisms combine the elementary building blocks of

syntax into complex structural representations?

• Semantics (Day 4)– What neural mechanisms combine the elementary building blocks of

semantics into complex semantic representations?

• NB: The vast majority of the studies I’ll discuss usevisual stimuli. So unless otherwise announced, thesubject is reading.

Plan for today

• Behavioral evidence for form-baseddecomposition

• Cognitive neuroscience of the visual word formarea and fusiform gyri more generally

• Two studies on the neural bases of earlydecomposition– Pylkkänen & Zweig (2009)– Solomyak & Marantz (2009)

Do morphologically related words elicit repetitionpriming effects or cumulative similarity effects?

Rastle et a. (2000) tested for the priming effects of Meaning similarity: cello - VIOLIN

Form/sound similarity: typhoid - TYPHOON

Meaning + form similarity: screech - SCREAM

Morphological relatedness: adapter - ADAPTABLE

Identity: church - CHURCH

Lowercase visual prime followed by uppercase visual target. Lexical decisions to target only. Priming assessed with respect to unrelated controls.

<unrel> - VIOLIN

<unrel> - TYPHOON

<unrel> - SCREAM

<unrel> - ADAPTABLE

<unrel> - CHURCH

Additional factor that was manipulated:

Stimulus Onset Asynchrony (SOA)

The interval between the onset of the prime and the onset of the target.

cello VIOLIN

0 50 100 150 200 250 300 350Time

[msec]

PRIME TARGET

SOA = 200ms

Priming effects differ in how fast they develop and how long they last. Rastle et al. used 3 different SOA’s:

43 msec 73 msec 230 msec

Results (Rastle et al. 2000)

How much faster or slower were subjects’ lexical decisions to the related thanto the unrelated conditions?Positive numbers = priming. Negative numbers = inhibition.

Prim

ing

(Unr

elat

ed -

Rel

ated

) / m

s

Positive primingall around.

Positive primingall around.

Slowly developingpriming effect.

Slowly developingpriming effect.

No reliableeffects.

Conclusions

The effect of morphological relatedness is Indistinguishable from the effect of identity.Unexplainable in terms of combined form and

meaning similarity. Follows straightforwardly from the decomposition

theory but not from the storage theory.

Behavioral evidence for across the boardpresemantic decomposition from masked priming

The prime is sandwiched between a forward pattern mask and the targetstimulus, which acts as a backward mask:E.g.,: mask (500 ms) #####      

prime (40-50 ms)  horse     target (500 ms) HOUSE

At such short SOAs, subjects are not consciously aware of the prime. Nevertheless, form-based and repetition priming can be obtained. No semantic priming (SOA too short).

Masked priming demo:http://www.u.arizona.edu/~kforster/priming/masked_priming_demo.htm

Davis, Rastle, & New (2004):

In a masked priming paradigm:morphology: CLEANER - clean primingapparent morphology: CORNER - corn equivalent primingno morphology: BROTHEL - broth no priming

All apparently morphologically complex words are decomposed in early stages ofvisual word perception.

Behavioral evidence for across the boardpresemantic decomposition from masked priming

What are the brain correlates of early form-baseddecomposition?

• The activity should be– Prelexical– But “high-level” enough such that it is able to detect

morpheme shapes/letter combinations

• Plausible candidate:– MEG M170, corresponding to activity from the

Visual Word Form Area (VWFA) of fMRI studies.

MEG and fMRI

• Magnetoencephalography (MEG)– Measures the magnetic fields generated by neuronal currents.– Millisecond temporal resolution.– Relatively good spatial resolution (~1cm for cortex).

• Functional magnetic resonance imaging– Measures blood oxygenation associated with increased neural

activity.– Excellent spatial resolution.– Temporal resolution not adequate for studying the dynamics

of language processing.

Typical MEG response to visual wordsM100 M170 M250 M350

100-150ms 150-200ms 200-300ms 300-400ms

Effects of low-levelvisual features such

as size and luminance.

“Visual featureanalysis”

Left:Letter-string effects

“Visual word form area”

Right:Face perception

“Fusiform face area”

?

Lexical effects:Frequency

((Embick Embick et al., 2000, et al., 2000, CBRCBR))

Morpheme repetition(Pylkk(Pylkkäänen et al., 2002; nen et al., 2002; Stockall Stockall &&

MarantzMarantz, 2006, , 2006, Mental LexiconMental Lexicon))

Morpheme frequency((Fiorentino Fiorentino && Poeppel Poeppel, 2008, , 2008, LCPLCP;;PylkkPylkkäänen et al., 2004, nen et al., 2004, CognitionCognition))

“Morpheme access”

McCandliss, Cohen and Dehaene, The visual word form area: expertise for reading in the fusiform gyrus. Trends Cogn Sci. 2003Jul;7(7):293-299.

Left fusiform gyrusThe Visual Word Form Area, VWFA

• Activated by words andpseudowords to a greaterdegree than other similarcontrol stimuli, such asconsonant strings (trwbrts).

• Insensitive to variance incase: tAbLe.– VWFA doesn’t just respond

to perceptual familiarity.

Lexical decisions on bimorphemic (teacher), monomorphemic (switch), & orthographic controls (winter)

What are the earliest effects of morphological

complexity in MEG? (Zweig & Pylkkänen, 2008, LCP)

Lexical decisions on bimorphemic (teacher), monomorphemic (switch), & orthographic controls (winter)

Larger M170 amplitudes for bimorphemic words:

But surprisingly, the effect was right-lateral…

What are the earliest effects of morphological

complexity in MEG? (Zweig & Pylkkänen, 2008, LCP)

hunterhunter silversilverfolderfolder switchswitch

ComplexComplex SimpleSimple

reliverelive rejectreject

PrefixedPrefixed SimpleSimple

Prefix follow-up

• Lexical decisions on– bimorphemic (relive),– monomorphemic (salute), &– orthographic controls (reject)

• Larger M170 amplitudes for bimorphemic words bilaterally:

Conclusions• Decomposition does not appear to affect lexical decision times.• Decomposition effects at the M170.• Extends to opaque forms, as predicted by masked priming data (folder, header)

[I didn’t show you these data]• No decomposition effect for pseudoaffixed forms such as winter.• Higher-level object recognition areas appear to look for morpheme shapes.• Decomposition occurs when the form exhaustively decomposes to independently

occurring forms.• Interesting cases:

– lov-er vs. kill-er (missing e)– drop-p-er vs. kiss-er (extra consonant)

• Masked priming evidence suggests that these decompose “normally,”even when there’s no semantic relationship (McCormick, Rastle & David,JML, 2007)

– toler-able (bound stem) (Solomyak & Marantz, 2009)

Solomyak & Marantz (2009):Studying decomposition in single trial data

• The usual averaging techniques (designed to increase signalto noise ratio) of imaging and electromagnetic data lose ahuge amount of information about single trials.

• But in recent years, several groups have developedtechniques for using evoked responses in single trials as thedependant measure, similarly to analyses of reaction timedata using multi-level regression models.

• Instead of trying to control for various stimulus variablesacross conditions, these variables are entered as factors inthe regression model.

Solomyak & Marantz (2009):Studying decomposition in single trial data

• To assess decomposition in single trial data, decomposition has totreated as some type of continuous variable.

• Traditionally it’s been thought that stem frequency is the mostimportant determinant of decomposition.

• But Hay and Baayen have argued that what’s more important is theconditional probability of the affixed form given the stem.– High probability: items occur together a lot.

• S&M tagged all their stimuli for this factor and contrasted it withorthographic transition probability, i.e., the likelihood of the first twoletters of the suffix, given the last two letters of the stem.

3 ROIs

- No effectsmorphological factors.

- reliable correlationswith letter stringfrequency andtransition probabilityfrom one string to thenext.

- higher amplitudesfor highermorphologicalconditional probability

- effects obtainsacross free andbound roots, althoughis marginal withinbound roots only

- higher amplitudesfor higher lemmafrequency

M100 M170 M250

Wrap-up

• Study of isolated words suggests that syntax-relevant computation starts in high-level objectrecognition area around 150-200ms.

• The parser over-parses, i.e., decomposes evenwhen that analysis is “wrong” (corner).

• So far decomposition studies have primarily(maybe exclusively) used isolated words.

• Tomorrow’s topic will relate to decompositionin sentential contexts.