Universität des Saarlandes Seminar: Language Prediction and Integration Jesús Calvillo

When Reading…

Fixations (information intake)

Saccades

When Reading…

Fixations (information intake)

Saccades

Duration

When Reading…

Fixations (information intake)

Saccades

Duration

Word Level Factors Token Frequency Word Length N-gram probabilities

When Reading…

Fixations (information intake)

Saccades

Duration

Word Level Factors Token Frequency Word Length N-gram probabilities

Sentence Level Factors Syntax Semantics …

When Reading…

Fixations (information intake)

Saccades

Duration

Word Level Factors Token Frequency Word Length N-gram probabilities

Sentence Level Factors

Syntax (Surprisal) Semantics …

When Reading…

Fixations (information intake)

Saccades

Duration

Word Level Factors Token Frequency Word Length N-gram probabilities

Sentence Level Factors

Syntax (Surprisal) Semantics …

Predictability??

Introduction

Surprisal

Dependency Parsing Surprisal

Experiments Surprisal and Predictability Surprisal and Eye-movement data

Conclusions

For a language, Surprisal is a measure of the information conveyed by any given linguistic unit (e.g. phoneme, word, utterance) in context.

Human sentence processing complexity metric. Assumes a Probabilistic Grammar.

A probabilistic grammar assigns exactly one structure to unambiguous

sentences.

Before the ending of a sentence, several structures might be compatible to the so far heard words.

After each word, certain amount of probability mass is discarded, reducing the number of possibly compatible structures.

Surprisal is the amount of probability mass eliminated after each word, or the amount of information that the word introduced.

Intuitively, surprisal increases when a parser is required to build some low-probability structure.

Der alte Kapitaen goss stets ein wenig Rum in seinen Tee. The old captain poured always a little Rum in his tea.

144 German sentences enriched with eye-tracking and n-gram information. Predictability study. Native speakers were asked to guess the next word

given its left-context (Incremental Cloze task)

272 participants for predictability ▪ High school students ▪ University students ▪ Adults as old as 80 years

83 complete predictability protocols for each word.

Every word (except the first of each sentence) is associated with an empirical word-predictability value Є [0,1] with mean=0.20 and sd=0.28

Values submitted to logit transformation to correct for the dependency between mean probabilities and the associated standard deviation

Linear Mixed-Effects Model

*Optimized with Deviance Information Criterion

*The DIC was 2229 for the simpler model, versus 2220 for each of the two more complex models.

Although surprisal seems to be irrelevant to predict cloze probabilities, it may bring in linguistic factors that are not captured by conscious reflection about upcoming words.

Humans tend to read more slowly under conditions of cognitive duress.

For example, readers make regressive eye movements more often and go more slowly during the disambiguating region of syntactically-ambiguous sentences.

144 German sentences Eyetracking data collected from

▪ 33 students (Age mean= 21.9 sd= 2.2 range:19-28 years)

▪ 32 adults (Age mean=69.9 sd=3.9 range:65-83 years)

One goal was to represent a large variety of grammatical structures.

Participants were instructed to read for comprehension.

Predictors: N-gram factors Word length Empirical Predictability

+Syntactic Surprisal

Deviance Information Criterion to compare the

simple model vs the model with surprisal.

For first-fixation durations, only those values were analyzed that were non-identical to single-fixation durations.

Reading times below 50 ms were removed.

Dependent measures were log transformed.

*For virtually every dependent measure, the predictive error (DIC) was lower in the more complex model that included surprisal.

Early and late fixation-duration-based measures exhibited clear effects of unigram and bigram frequency, and logit predictability. with the exception of first-fixation duration in which

predictability and length were not significant.

For fixation durations, surprisal had a significant effect in the predicted direction. (longer durations for higher surprisal values) with the exception of cfg-surprisal on RRTs .

*Dependency grammar based surprisal presents a significant effect, while pcfg based surprisal is not significant.

Surprisal values are significant predictors of reading times and regressions. Even when predictability, n-gram frequency and word length are

taken into account.

Surprisal did not appear to have a significant effect on empirical predictability (cloze probabilities).

Surprisal appears as factor for both early and late measures, with comparable magnitudes. Hard to reconcile with a simple identification of early measures with

syntactic parsing costs and late measures with durations of post-syntactic events.

Syntactic parsing costs can be estimated using probabilistic knowledge of grammar.

Boston, Marisa Ferrara, John T. Hale, Reinhold Kliegl, Umesh Patil, and Shravan Vasishth. 2008. Parsing costs as predictors of reading difficulty: An evaluation using the Potsdam Sentence Corpus. Journal of Eye Movement Research 2(1): 1-12.

Kliegl, R., Grabner, E., Rolfs, M., & Engbert, R. 2004.Length, frequency, and predictability effects of words on eye movements in reading. European Journal of Cognitive Psychology, 16, 262–284.

Christiane Wotschack. 2009. Eye Movements in Reading Strategies.

Slides by Matthew Crocker , 2013.

Wikipedia: http://en.wikipedia.org/wiki/Deviance_information_criterion http://en.wikipedia.org/wiki/Logit http://en.wikipedia.org/wiki/Mixed_model