Language

1 Overview.2 Biology and Basic Representations.3 Distributed Lexicon & Dyslexia.4 Reading & Past Tense.5 Semantics & Sentences.

1 / 33

Overview of Models

Distributed lexicon (ortho, phono, sem): reading & dyslexia.

Orthography to phonology in reading: regularities andexceptions.

Semantics to phonology and past tenseoverregularizations.

Semantic representations from word co-occurrences.

Sentence-level processing and the sentence gestalt.

2 / 33

Biological Substrates of Language

teethlips

tongue

phar

anx

hard palatevelum(soft palate)

nasal cavity

alveolarridge

epiglottis

glottis(vocal cords)

uvula Broca’s Wernicke’s

Frontal Parietal

Occ

ipita

l

Temporal

3 / 33

Phonology Features: Vowels

<- front | back ->1 2 3 4 5 6 7

ˆ +---------------------+u 1 E--A_ Up 2 \ i --\ u |__hi_ˆ__

3 e \ --Y-- / Od \ \ \-/ |o 4 \ \ ˆ / \--ow 5 @ -I- W |__lo_v__n \ \ / |v 6 ----- a ------+

round/flat, long/short

4 / 33

Phonology Features: Consonants

place: labial, labio-dental, dental, alveolar, palatal, velar, glottal

manner: plosive, fricative, semi-vowel, liquid, nasal

vocalized, not vocalized

5 / 33

Distributed Lexicon Model & Dyslexias

Orthography Phonology

Semantics

Hidden Hidden

Hidden

Phonological: nonwords (“nust”) impaired.

Deep: phono + semantic errors (“dog” as “cat”) +visual errors (“dog” as “dot”).

Surface: nonwords OK + semantic access impaired +difficulty reading exception words (“yacht”) + visualerrors.

6 / 33

The Model

t w y

a d e g k n r s

i l n o p r s v

l o r t a c e g

p r s t w a e i

c d e f g h l n

OS_Hid

Semantics

SP_Hid

w

s t

p r

l n

h k

f g

− d

t w

p r

l n

h k

f g

− d

t w

p r

l n

h k

f g

− d

i o

a e

O U

E I

@ A

z

t v

r s

n p

k l

g j

− d

z

t v

r s

n p

k l

g j

− d

z

t v

r s

n p

k l

g j

− d

Orthography OP_Hid Phonology

Cleanup self-connections.7 / 33

Corpus and SemanticsConcrete/Abstract Semantics

Y

X0.00

5.00

10.00

15.00

20.00

25.00

30.00

35.00

40.00

0.00 10.00 20.00 30.00 40.00

tentflea

lassfacegrin

wavereedloon

haredeerhind

tartflan

pleahint

planflaw

lackneed

deedtactease

pastgainloss

factrole

stayhire

loanrent

costwage

ropepostlockcase

starcoatflag

20 concrete (more features), 20 abstract.8 / 33

Simulating DyslexiaLayer(s) lesioned Dyslexia TypeComp Sem SurfaceComp Dir DeepOSHid SurfaceSP Hid SurfaceOPHid PhonoOSHid + Comp Dir DeepSP Hid + Comp Dir DeepOPHid + Comp Sem Surface

Orthography Phonology

Semantics

Hidden Hidden

Hidden

9 / 33

Semantic Pathway Lesions, Intact Direct

0.0 0.2 0.4 0.6 0.8Lesion Proportion

05

101520

Err

ors

VisualVis + SemSemanticBlendOther

05

101520

Err

ors

0.0 0.2 0.4 0.6 0.8Lesion Proportion

Concrete Abstract

OS_Hid

SP_Hid SP_Hid

OS_Hid

10 / 33

Semantic Pathway Lesions, Lesioned Direct

0.0 0.2 0.4 0.6 0.8Lesion Proportion

05

101520

Err

ors

VisualVis + SemSemanticBlendOther

05

101520

Err

ors

0.0 0.2 0.4 0.6 0.8Lesion Proportion

Concrete Abstract

OS_Hid

SP_Hid SP_Hid

OS_Hid

11 / 33

Direct Pathway Lesion

0.0 0.2 0.4 0.6 0.8Lesion Proportion

05

101520

Err

ors

VisualVis + SemSemanticBlendOther

05

101520

Err

ors

0.0 0.2 0.4 0.6 0.8Lesion Proportion

Concrete Abstract

Full Sem Full Sem

No Sem No Sem

12 / 33

Language II: Reading and Past Tense

1 Reading.2 Past Tense.

13 / 33

Reading: Regularities & Exceptions

Regularities in pronunciation are often partial, contextdependent:i in mint, hint.. vs mind, find.. except: pint

Exceptions are extreme of context dependent.

Seidenberg & McClelland (1989) used extremely contextdependent wickelfeatures: _think_ = _th , thi , hin , ink ,and nk_

PMSP used hand-coded context independent inputs (+ someconjunctions).

Need a range of context dependency for regulars andexceptions.

14 / 33

Reading as Object Recognition

Tradeoff between dependence & independence similar to objectrecognition: need invariance but also need to bind features.

words...words

worw

orrds d

hidden

phonemes

15 / 33

Reading Model

Ortho

Ortho_Code

Hidden

Phon

16 / 33

Nonword Performance

Regularity tests (Glushko): bint→ /bint/

Pseudo-homophones (McCann & Besner):phoyce→ /fYs/, choyce→ /CYs/

Matched regularity/exception cases (Taraban):High freq: poes→ /pOz/, goes→ /gOz/, does→ /dˆz/Low freq: mose→ /pOs/, poes→ /pOz/, lose→ /lUz/

Nonword Set Model PMSP PeopleGlushko regulars 95.3 97.7 93.8Glushko exceptions raw 79.0 72.1 78.3Glushko exceptions alt OK 97.6 100.0 95.9McCann & Besner ctrls 85.9 85.0 88.6McCann & Besner homoph 92.3 n/a 94.3Taraban & McClelland 97.9 n/a 100.01

17 / 33

Past Tense

Simple production model: semantics to phonology, withinflections:

Semantics

Phonology

pasttense

ed

(strong correlation)

18 / 33

Past Tense: U-Shaped Curve

This is the interesting target developmental phenomenon:

2 3 4 5 6 7 8 9 10Years of Age

0.90

0.92

0.94

0.96

0.98

1.001−

Ove

rreg

Rat

eOverregularization in Adam

Eventualcorrect

performanceassumed

19 / 33

U-Shaped History

Initially: overzealous rule usage.

Then: Rumelhart & McClelland, U-shaped curve based onnetwork processing of regularities. BUT: trained irregulars first,then regs.

(much controversy ensues)

Later: Plunkett et al., etc, manipulate enviro in graded way.

Problem: backprop is gradient descent!

20 / 33

U-Shaped Model in Leabra

Competition & Hebbian learning produce network that is indynamic balance between reg & irreg mappings.

Small tweaks can shift it one way or the other! (priming model).

Hebbian sensitivity to correlational regularity.

21 / 33

The Past Tense Model

Semantics

Hidden

Phonology

22 / 33

The Past Tense Model

Inflection Reg sfx Regular/Irregular examples

Base – I walk to the store daily.I go to the store daily.

Past -ed I walked to the store yesterday.I went to the store yesterday.

3rd pers sing -s She walks to the store daily.She goes to the store daily.

Progressive -ing I am walking to the store now.I am going to the store now.

Past participle -en I have walked to the store before.I have gone to the store now.

23 / 33

Past Tense Results

0 25000 50000 75000Number of Words

0.0

0.2

0.4

0.6

0.8

1.0

1−O

R, R

espo

nses

Overregularization in Bp

ORResponses

b)

a)

0 25000 50000 75000Number of Words

0.0

0.2

0.4

0.6

0.8

1.0

1−O

R, R

espo

nses

Overregularization in Leabra

ORResponses

24 / 33

Past Tense Results

Bp L H0 L H001 L H0050

25

50

75

100

Res

pons

es

Early Correct Responding

To 1st ORTo 2nd OR Bp L H0 L H001 L H005

0

50

100

150

200

Ove

rreg

ular

izat

ions

Total Overregularizations

25 / 33

Language III: Semantics and Sentences

1 Semantics.2 Sentences.

26 / 33

Distributed Semantics

orthography phonology

action oriented

tactile

visual

auditory

kinestheticform

color

3−D

telephonefoot

brake

kettle

velv

et

cloudsthunder

Semantics is distributed across specialized processing areas.27 / 33

Model: Correlational Semantics

Use Hebbian learning to encode structure of wordco-occurrence.

Same idea as:

V1 receptive field learning: learn the strong correlations.

Latent Semantic Analysis (LSA) (but avoids “blobs” ofPCA).

Input

Hidden

28 / 33

Sentences

Traditional approach:S

(subject)

Art N

The boy

V NP(direct object)

chases the cats

NP VP

Alternative approach:

Distributed reps of sentence meaning: The sentence Gestalt!

Parallel to object recognition issues: 3D structural model vs.distributed reps that distinguish different objects.

29 / 33

Toy World

People: busdriver (adult male), teacher (adult female),schoolgirl, and pitcher (boy).

Actions: eat, drink, stir, spread, kiss, give, hit, throw, drive, rise.

Objects: spot (the dog), steak, soup, ice cream, crackers, jelly,iced tea, kool aid, spoon, knife, finger, rose, bat (animal), bat(baseball), ball, ball (party), bus, pitcher, and fur

Locations: kitchen, living room, shed, and park.

Syntax: Active & Passive, phrases.

30 / 33

Network

Input

Encode

Gestalt Gestalt Context

Decode

Role Filler

31 / 33

Tests

Task SentenceRole assignment

Active semantic The schoolgirl stirred the kool-aid with a spoon.Active syntactic The busdriver gave the rose to the teacher.Passive semantic The jelly was spread by the busdriver with the knife.Passive syntactic The teacher was kissed by the busdriver.(control) The busdriver kissed the teacher.

Word ambiguity The busdriver threw the ball in the park.The teacher threw the ball in the living room.

Concept instantiation The teacher kissed someone (male).Role elaboration The schoolgirl ate crackers (with finger).

The schoolgirl ate (soup).Online update The child ate soup with daintiness.

(control) The pitcher ate soup with daintiness.Conflict The adult drank iced-tea in the kitchen (living-room).

32 / 33

Gestalt Representations

SG Gestalt PatternsY

X0.00

1.00

2.00

3.00

4.00

5.00

6.00

7.00

8.00

9.00

10.00

11.00

12.00

13.00

14.00

15.00

16.00

0.00 2.00 4.00 6.00 8.00

sc_at_so.

bu_at_so.

te_at_so.

pi_at_so.

sc_at_st.

bu_at_st.

te_at_st.

pi_at_st.

pi_dr_ic.

sc_dr_ic.

bu_dr_ic.

te_dr_ic.

pi_st_ko.

bu_st_ko.

te_st_ko.

sc_st_ko.

33 / 33