Slide 1

Slide 2

Technology Based Assessment of Language and Literacy Patti Price - Intro - Decoding - Comprehension - Opportunities/Discussion PPRICE Speech & Language Technology www.pprice.com

Slide 3

Primates Gorillas Humans Stone tools FireShelter Spears Language: ~ 50,000 years ago Clothing Burial Hunting Art Farming, 10,000 years ago Bronze, 5500 years ago Written language: ~5 6,000 years ago Intro Decoding Comprehension Discussion

Slide 4

Pre-language: Information access by direct experience Oral language: Information access also from someone in same time and place with same language Written language: Information access not limited by time and place Multimedia: The power of direct experience, but not limited by time, place, language, learning style, sensory abilities Intro Decoding Comprehension Discussion

Slide 5

Written and Spoken Language are Different These differences are important in learning to read. How difficult it is to spell (many letter combos to one sound) How difficult it is to read (many sounds for the same letter) How closely your dialect matches written conventions Got to say this for you, you got guts. Guts and no brains. But guts alone dont mean nothin. I gotta say this for you-- you have guts. Guts and no brains. But guts alone mean nothing. Ive got to say this for you: You have guts. You have guts and no brains, but guts alone mean nothing. Intro Decoding Comprehension Discussion

Slide 6

Differences Between Speech and Writing Vocabulary Distinct words Sentence length Sentence structure Grammar Prosody WrittenSpoken things stuff 20,000+ 2,000+ 10 30 words ? Complex shallow Conservative Liberal A., ! ? ; : () Rich Intro Decoding Comprehension Discussion

Slide 7

~130 words (1) Ive been talking Ive been multiplying matrices already, but certainly time for me to discuss the rules for matrix multiplication. (2) And the interesting part is the many ways you can do it, and they all give the same answer. (3) So its and theyre all important. (4) So matrix multiplication, and then, uh, come inverses. (5) So were uh, we mentioned the inverse of a matrix, but theres thats a big deal. (6) Lots to do about inverses and how to find them. (7) Okay, so Ill begin with how to multiply two matrices. (8) First way, okay, so suppose I have a matrix A multiplying a matrix B and giving me a result well, I could call it C. (9) A times B. Okay. (10) Uh, so, l- let me just review the rule for w- for this entry. Transcript from a linear algebra lecture (From Glass, Hazen, Lee and Wang, Analysis and Processing of Lecture Audio Data, 2004) Example ~45 words Weve been multiplying matrices. Now lets discuss the rules for matrix multiplication. Note that there are many ways to do it and get the same answer. One way to multiply two matrices, A and B, to get a new matrix C is the following rule: Intro Decoding Comprehension Discussion

Slide 8

Overview of TBALL T echnology B ased A ssessment of L anguage & L iteracy Child-friendly testing system Measure & analyze child reading & pronunciation 5-8 year olds Native speakers of English & Spanish Multidisciplinary collaboration: o Engineering o Education o Computer science o Linguistics o Neuroscience UCLA, USC, UCB Intro Decoding Comprehension Discussion

Slide 9

8 TBall Goals and Components Reliable and objective automatic reading assessments Comprehensive instructional framework for Diagnosis Intervention Three main components: Childrens interface Teacher interface Assessment module o Decoding (DAM) o Comprehension (BARLA) Intro Decoding Comprehension Discussion

Slide 10

9 Decoding Comprehension Need to be balanced In alphabet-based languages, hints about pronunciation If you know the language you are reading, the hints may be enough If your first language is another language, and/or if you know the LTS rules in another language, there may be problems. MOSTLY depends on successful decoding Much more to learn than LTS rules Intro Decoding Comprehension Discussion

Slide 11

Decoding Necessary (but not sufficient) for reading comprehension Necessary for reliable and efficient processing of text Strongly related to speed of initial reading acquisition A predictor of difficulties in comprehension development Accounts for much variance in reading ability at all ages Intro Decoding Comprehension Discussion

Slide 12

English Letter to Sound Rules Example: pronunciations of o Pronunciations of the letter o observed in the 100 most frequent words of English 9 OW: go, home, most, no, only, over, so (know, own) 6 UW: into, to, who, do, (too, you) 6 AH: from, of, other, another, (some, come) 6 AO: for, or, on, often, (your, more) 5 OU: (about, hour; down, how, now) 3 UH: (good, goodbye would) 3 -: (people, work, world) 2 AA: got, not 1 W AH: one Intro Decoding Comprehension Discussion

Slide 13

Written Language, Complexity LanguageLetterSound Englishc/s/, /k/ hot, car, walk, ah/aa/ Spanisha/aa/ Frencha/aa/ allo, pot, eau/ow/ SpellingSound ache + past = ached/ey k/ + /t/ aim + past = aimed/ey m/ + /d/ paint + past = painted/p ey n t/ + /ax d/ Intro Decoding Comprehension Discussion

Slide 14

Decoding Assessment Are the words read correctly? Decodability: closer to one letter one sound is better In sample of 100 most frequent words in grade 1 material m, j, b, d pretty straight forward (always /m jh b d/) Digraphs (2 letters -> 1 sound): -ng, -ck, (wh-) 2 letters -> 2 sounds (or): th -> /dh, th/ (thy, thigh) 1 letter -> 2 sounds (and): x -> /k s/ (fox, box) G: get, thing, laugh, large E: because, get, been, they, great, often O: go, do, from, for, how, good, people, got, one Intro Decoding Comprehension Discussion

Slide 15

Decoding Assessment Some words are more decodable than others High frequency words tend to be less decodable High frequency words tend to be sight words (not decoded but recognized instantly) (this is important) Decoding skills are also important If a word is well-known by a child, it may be treated as a sight word and NOT decoded (tyrannosaurus) Therefore, unless we know which words are sight words for a child, its hard to assess decoding skills Nonsense words must be decoded Decoding Assessment Measure Intro Decoding Comprehension Discussion

Slide 16

Decoding Issues: Dialect, Idiolect and Assessment If the child has trouble with the letter 'r' (common in English), and says 'cow' when shown 'car' is it a misreading? If the child is from Boston and reads 'Carl' as 'Call' is it a misreading? What if the teacher has never heard anyone from Boston? if the child sees the word 'ask' and says 'aks'? if the child is a native speaker of another language, e.g., Spanish, and says 'seat' when presented the word 'sit', is it a misreading? if the child is already learning to read, e.g., Spanish, and sees the word 'sore' and pronounces it 'sore-ay', is it a misreading? SOLUTION: Map the child's system, whatever it is Augment the dictionary to diagnostic labels. Intro Decoding Comprehension Discussion

Slide 17

Dialect Modeling in ASR Dialect is a system The system is not well modeled in ASR These are being merged These are being merged When the point is the Contrast system When the point is the Contrast system Head sections adapted from Gunnar Fants Acoustic Theory of Speech Production Intro Decoding Comprehension Discussion

Slide 18

Slide 19

Decoding Issues: Dialect, Idiolect and Assessment Grape g r ey p Grape g w ey p Grape g r aa p ey Bird b er d Bird b uh d Bird b iy r d Rocks w aa k s Horse h ow s Three th w iy Run w ah n Intro Decoding Comprehension Discussion

Slide 20

Decoding Results nug: /n uw g/ - Sp. LTS, phonology, or short/long? jore, tay, bap: /hh ao r ey, t ay, b aa p/ - Sp. LTS yan, vag, hine: /zh aa n, b aa g, ay n/ - Sp. Phon. rin, bap, mot: /r iy n, b ey p, m ow t/ - short/long jore: /zh ow r ey/ - mixed Spanish phonology, LTS jick: /hh ih s k/ - Spanish LTS, misapplied English LTS Intro Decoding Comprehension Discussion

Slide 21

Comprehension Assessment GRADE 2: Kim has a dog. The dog's name is "Dot." Dot is a very white dog, and he has a black dot on his leg. Sam, Ben, and Kim like to play with Dot. They play ball with Dot all day long. Ben lost the ball in the mud, and Dot went into the mud to get the ball. Now, Dot is a brown dog from the mud! What does Dot look like? A Dalmatian Brown I dont remember Like a dog A white dog A white dog with a black dot on his leg White and a dot on his leg Well hes white and he has a brown dot on his leg Like a rat Please describe Dots physical appearance? Intro Decoding Comprehension Discussion

Slide 22

Motivation/ Desiderata Equivalent measures for reading or listening (to compare) Separate understanding of probe from understanding of text (probe as picture adds interest and is not text or speech) Diagnose reading error patterns (by selection of alternate items) Easy to automate (compared to most comprehension tests) Items ascend in difficulty (several errors in a row stops the test) Many items (to allow retesting without repeating items) Equal number of plausible/implausible items (to mitigate guessing) PIAT-R (Peabody Individual Achievement Test Revised) is best match, but Not automated Reading only They did not respond to our request Intro Decoding Comprehension Discussion

Slide 23

Intro Decoding Comprehension Discussion Vocabulary Syntax Morphology Accent adjustment Prosody Find words Print adjustment Punctuation Decoding (LTS) In your first language, this might be easier In a new language, this might be easier

Slide 24

BARLA Example 1. (pot and box reversed) 2.(dog in box) 3.(box on boy) 4.(correct) The pot is on a box. Hear OR Read: The pot is on a box. Intro Decoding Comprehension Discussion

Slide 25

BARLA Example Here is a fire. 1.tire (rhymes) 2.fire (correct) 3.fur (vowel differs) 4.fairy(shares, f, i, r) Hear or read: Here is a fire. Intro Decoding Comprehension Discussion

Slide 26

Decoding Opportunity We sampled the space with 25 items High Frequency Word: tell, nap Low Frequency Word: fell, zap Nonsense Word: zell, fap However, desirable would be Frequent retesting (perhaps in a game environment) Diagnostic of which decoding skills need more work Items need to be generated and coded for diagnosis Intro Decoding Comprehension Discussion

Slide 27

An Opportunity! Intro Decoding Comprehension Discussion ContentGamePedagogy User Model Letter sounds, English: a /ae/ b /b/ c /k/, /s/ z /z/ By font, audio files Levels: Lower case simple to complex Upper case Digraphs Move up if: 100% correct Move down if: 3 wrong in a row Last played Sounds mastered Diagnosis Preferences Engine b p d b p d

Slide 28

An Opportunity! Intro Decoding Comprehension Discussion ContentGamePedagogy User Model Vocabulary, English: bat Levels: Lower case simple to complex Upper case Digraphs Move up if: 100% correct Move down if: 3 wrong in a row Last played Sounds mastered Diagnosis Preferences Engine b p d b t a

Slide 29

A Comprehension Opportunity! Data collected for sentences analogous to PIAT-R But not analyzed Paradigm ready to hatch more uses For more information about the project: http://nautilus.icsl.ucla.edu/tball/ For more information about the data: iseli@ucla.edu iseli@ucla.edu Intro Decoding Comprehension Discussion

Slide 30

An Opportunity! Intro Decoding Comprehension Discussion Natural language processing and reading augmentation

Slide 31

Simplify Vocabulary By "augmenting human intellect" we mean increasing the capability of a man to approach a complex problem situation, to gain comprehension to suit his particular needs, and to derive solutions to problems. Increased capability in this respect is taken to mean a mixture of the following: more-rapid comprehension, better comprehension, the possibility of gaining a useful degree of comprehension in a situation that previously was too complex, speedier solutions, better solutions, and the possibility of finding solutions to problems that before seemed insoluble. And by "complex situations" we include the professional problems of diplomats, executives, social scientists, life scientists, physical scientists, attorneys, designers--whether the problem situation exists for twenty minutes or twenty years. We do not speak of isolated clever tricks that help in particular situations. 1. Select Text2. Select View3. View By "augmenting human intellect" we mean increasing the capability of a man to approach a complex problem situation, to gain comprehension to suit his particular needs, and to derive solutions to problems. Increased capability in this respect is taken to mean a mixture of the following: more-rapid comprehension, better comprehension, the possibility of gaining a useful degree of comprehension in a situation that previously was too complex, speedier solutions, better solutions, and the possibility of finding solutions to problems that before seemed insoluble. And by "complex situations" we include the professional problems of diplomats, executives, social scientists, life scientists, physical scientists, attorneys, designers--whether the problem situation exists for twenty minutes or twenty years. We do not speak of isolated clever tricks that help in particular situations. Simplify vocabulary Simplify syntax Simplify both Simplify vocabulary Simplify syntax Simplify both By "adding to people's brains" we mean adding to the way we deal with difficult things, to add to understanding according to our needs, and to figure out solutions to problems. Intro Decoding Comprehension Discussion

Slide 32

Simplify Syntax By "augmenting human intellect" we mean increasing the capability of a man to approach a complex problem situation, to gain comprehension to suit his particular needs, and to derive solutions to problems. Increased capability in this respect is taken to mean a mixture of the following: more-rapid comprehension, better comprehension, the possibility of gaining a useful degree of comprehension in a situation that previously was too complex, speedier solutions, better solutions, and the possibility of finding solutions to problems that before seemed insoluble. And by "complex situations" we include the professional problems of diplomats, executives, social scientists, life scientists, physical scientists, attorneys, designers--whether the problem situation exists for twenty minutes or twenty years. We do not speak of isolated clever tricks that help in particular situations. 1. Select Text2. Select View3. View By "augmenting human intellect" we mean increasing the capability of a man to approach a complex problem situation, to gain comprehension to suit his particular needs, and to derive solutions to problems. Increased capability in this respect is taken to mean a mixture of the following: more-rapid comprehension, better comprehension, the possibility of gaining a useful degree of comprehension in a situation that previously was too complex, speedier solutions, better solutions, and the possibility of finding solutions to problems that before seemed insoluble. And by "complex situations" we include the professional problems of diplomats, executives, social scientists, life scientists, physical scientists, attorneys, designers--whether the problem situation exists for twenty minutes or twenty years. We do not speak of isolated clever tricks that help in particular situations. Simplify vocabulary Simplify syntax Simplify both Simplify vocabulary Simplify syntax Simplify both By "augmenting human intellect" we mean increasing the capability of a person to approach a complex problem situation. We also mean how a person gains comprehension to suit his particular needs and derives solutions to problems. Intro Decoding Comprehension Discussion

Slide 33

Simplify Vocab & Syntax By "augmenting human intellect" we mean increasing the capability of a man to approach a complex problem situation, to gain comprehension to suit his particular needs, and to derive solutions to problems. Increased capability in this respect is taken to mean a mixture of the following: more-rapid comprehension, better comprehension, the possibility of gaining a useful degree of comprehension in a situation that previously was too complex, speedier solutions, better solutions, and the possibility of finding solutions to problems that before seemed insoluble. And by "complex situations" we include the professional problems of diplomats, executives, social scientists, life scientists, physical scientists, attorneys, designers--whether the problem situation exists for twenty minutes or twenty years. We do not speak of isolated clever tricks that help in particular situations. 1. Select Text2. Select View3. View By "augmenting human intellect" we mean increasing the capability of a man to approach a complex problem situation, to gain comprehension to suit his particular needs, and to derive solutions to problems. Increased capability in this respect is taken to mean a mixture of the following: more-rapid comprehension, better comprehension, the possibility of gaining a useful degree of comprehension in a situation that previously was too complex, speedier solutions, better solutions, and the possibility of finding solutions to problems that before seemed insoluble. And by "complex situations" we include the professional problems of diplomats, executives, social scientists, life scientists, physical scientists, attorneys, designers--whether the problem situation exists for twenty minutes or twenty years. We do not speak of isolated clever tricks that help in particular situations. Simplify vocabulary Simplify syntax Simplify both Simplify vocabulary Simplify syntax Simplify both By "adding to people's brains" we mean adding to how we deal with difficult things. We also mean adding to understanding that is suited to our needs, and to how we can figure out solutions to problems. Intro Decoding Comprehension Discussion

Slide 34

Simplify Vocabulary & Syntax: Result By "adding to people's brains" we mean adding to how we deal with difficult things. We also mean adding to understanding that is suited to our needs, and to how we can figure out solutions to problems. By adding to what we can do, we mean a mix of several things. Faster and better understanding are examples. We also mean making it possible to get some understanding when before it was too difficult. And, we mean faster solutions, better solutions, and being able to find solutions that didn't seem possible before. By "difficult situations" we mean the problems people have in different kinds of jobs. The jobs can be in the government, running businesses, being scientists, lawyers, or designers. The problems can be there for twenty minutes or twenty years. We do not mean clever tricks that help in only some situations. We mean a way of life. In this new way of life many things are useful. Useful things include hunches, cut-and-try, things we can't touch, and our "feel for a things. These things are more useful with great ideas, ways to talk about the ideas and ways to write them, intelligent ways of doing things, and powerful technologies to help us. The number of people and what they create are growing fairly quickly. But our problems are getting harder even more quickly. Also, the importance of finding solutions becomes greater because things are changing more quickly than they used to change. Things that before happened only in one place now are spreading to the whole world. This also makes it important to find solutions. If we could make our brains better in this way, any intelligent person would run as fast as possible to get it. To make them want to run, we need to show a way to do it. We also need to show what we might get out of it. Intro Decoding Comprehension Discussion

Slide 35

Summary of Views and Reading Ease Original Words239 Sentences7 Word/sentence34.1 Characters/ word 5.6 Reading Ease10.7% Flesch-Kincaid Grade Level 12.0 Vocabu -lary SyntaxBoth 275249315 71522 39.216.614.3 4.75.64.4 42.3%28.2%71.7% 12.0 6.8 Intro Decoding Comprehension Discussion

Slide 36

Questions? Intro Decoding Comprehension Discussion