kathy mccoy artificial intelligence natural language processing applications for people with...
Post on 19-Dec-2015
222 views
TRANSCRIPT
Kathy McCoyKathy McCoy
Artificial IntelligenceArtificial Intelligence
Natural Language ProcessingNatural Language Processing
Applications for People with Applications for People with DisabilitiesDisabilities
Primary Research AreasPrimary Research Areas
Natural Language GenerationNatural Language Generation – problem – problem of choice. of choice. Deep Generation --- structure and content of Deep Generation --- structure and content of
coherent textcoherent text Surface Generation – particularly using TAG Surface Generation – particularly using TAG
(multi-lingual generation and machine translation)(multi-lingual generation and machine translation)
Discourse ProcessingDiscourse Processing Second Language AcquisitionSecond Language Acquisition Applications for people with disabilitiesApplications for people with disabilities
affecting their ability to communicateaffecting their ability to communicate
ProjectsProjects Augmentative CommunicationAugmentative Communication – –
Center for Applied Science and Center for Applied Science and Engineering in Rehabilitation (ASEL) – Engineering in Rehabilitation (ASEL) – Word Prediction and Contextual Word Prediction and Contextual Information (Keith Trnka, (Jay McCaw), Information (Keith Trnka, (Jay McCaw), Chris Pennington, Debbie Yarrington)Chris Pennington, Debbie Yarrington)
ICICLEICICLE – CALL system for teaching – CALL system for teaching English as a second language to ASL English as a second language to ASL natives (Rashida Davis, Charlie natives (Rashida Davis, Charlie Greenbacker)Greenbacker)
Text SkimmingText Skimming – for someone who is blind – for someone who is blind to skim a document to find an answer to a to skim a document to find an answer to a question (Debbie Yarrington).question (Debbie Yarrington).
Generating Textual Summaries of GraphsGenerating Textual Summaries of Graphs – (Sandee Carberry, Seniz Demir)– (Sandee Carberry, Seniz Demir)
Developing Developing Intelligent Intelligent
Communication Communication Aids for People Aids for People with Disabilitieswith DisabilitiesKathleen F. McCoyKathleen F. McCoy
Computer and Information Sciences & Center for Applied Science and Engineering in Rehabilitation
University of Delaware
Augmentative Augmentative CommunicationCommunication
Intervention that gives non-speaking person Intervention that gives non-speaking person an alternative means to communicatean alternative means to communicate
User PopulationUser Population May have severe motor impairmentsMay have severe motor impairments
Unable to speakUnable to speak Unable to writeUnable to write Cannot use sign languageCannot use sign language
May have cognitive impairments and/or May have cognitive impairments and/or developmental disabilitiesdevelopmental disabilities
May be too young to have developed May be too young to have developed literacy skillsliteracy skills
Predicting Fringe Predicting Fringe VocabularyVocabulary
Word Prediction of Spelled Words Word Prediction of Spelled Words (infrequent context-specific words)(infrequent context-specific words)
MethodsMethods Statistical NLP MethodsStatistical NLP Methods Learning from the context of the Learning from the context of the
individualindividual Other Contextual CluesOther Contextual Clues
Geographic Location, Time of Day, Geographic Location, Time of Day, Conversational Partner, Topic of Conversational Partner, Topic of Conversation, Style of the DocumentConversation, Style of the Document
Can we do better??Can we do better??
Intuitively all possible words do not Intuitively all possible words do not occur with equal likelyhood during a occur with equal likelyhood during a conversation.conversation.
The topic of the conversation affects The topic of the conversation affects the words that will occur. the words that will occur. E.g., when talking about baseball: ball, E.g., when talking about baseball: ball,
bases, pitcher, bat, triple….bases, pitcher, bat, triple…. How often do these same words occur How often do these same words occur
in your algorithms class?in your algorithms class?
Topic ModelingTopic Modeling
Goal: Automatically identify the topic Goal: Automatically identify the topic of the conversation and increase the of the conversation and increase the probability of related words and probability of related words and decrease probability of unrelated decrease probability of unrelated words.words.
QuestionsQuestions Topic RepresentationTopic Representation Topic IdentificationTopic Identification Topic ApplicationTopic Application Topic Language Model UseTopic Language Model Use
Topic ApplicationTopic Application
How do we use those similarity How do we use those similarity scores?scores?
Essentially weight the contribution Essentially weight the contribution of each topic by the amount of of each topic by the amount of similarity that topic has with the similarity that topic has with the current conversation.current conversation.
Current WorkCurrent Work
What happens with significantly larger What happens with significantly larger corpora?corpora?
What other kinds of tuning to the user What other kinds of tuning to the user can we do:can we do: RecencyRecency StyleStyle
Does keystroke savings translate into Does keystroke savings translate into communication rate enhancement?communication rate enhancement?
Problem:Problem: Blind and dyslexic individuals cannot skim textBlind and dyslexic individuals cannot skim text
Example: “What’s the syntax for calling a function Example: “What’s the syntax for calling a function with template parameters?” (skimming through with template parameters?” (skimming through code)code)
““Why was Ayers Rock renamed?”Why was Ayers Rock renamed?” ““What type of tree produces leaves with three What type of tree produces leaves with three
distinct shapes?”distinct shapes?” ““Where can I find more information about Portugal?”Where can I find more information about Portugal?”
People who cannot read text rely on People who cannot read text rely on screen readers (Jaws, Window-Eyes)screen readers (Jaws, Window-Eyes) braille outputbraille output
more difficult to come by more difficult to come by extremely bulky to carry aroundextremely bulky to carry around
Example of Jaws Output at 400 Example of Jaws Output at 400 wpmwpm
LinkLink““What psychological and philosophical significance should we attach to What psychological and philosophical significance should we attach to
recent efforts at computer simulations of human cognitive capacities? In answering recent efforts at computer simulations of human cognitive capacities? In answering this question, I find it useful to distinguish what I will call "strong" AI from "weak" or this question, I find it useful to distinguish what I will call "strong" AI from "weak" or "cautious" AI (Artificial Intelligence). According to weak AI, the principal value of the "cautious" AI (Artificial Intelligence). According to weak AI, the principal value of the computer in the study of the mind is that it gives us a very powerful tool. For example, computer in the study of the mind is that it gives us a very powerful tool. For example, it enables us to formulate and test hypotheses in a more rigorous and precise fashion. it enables us to formulate and test hypotheses in a more rigorous and precise fashion. But according to strong AI, the computer is not merely a tool in the study of the mind; But according to strong AI, the computer is not merely a tool in the study of the mind; rather, the appropriately programmed computer really is a mind, in the sense that rather, the appropriately programmed computer really is a mind, in the sense that computers given the right programs can be literally said to understand and have other computers given the right programs can be literally said to understand and have other cognitive states. In strong AI, because the programmed computer has cognitive states, cognitive states. In strong AI, because the programmed computer has cognitive states, the programs are not mere tools that enable us to test psychological explanations; the programs are not mere tools that enable us to test psychological explanations; rather, the programs are themselves the explanations.rather, the programs are themselves the explanations.
I have no objection to the claims of weak AI, at least as far as this article is I have no objection to the claims of weak AI, at least as far as this article is concerned. My discussion here will be directed at the claims I have defined as those of concerned. My discussion here will be directed at the claims I have defined as those of strong AI, specifically the claim that the appropriately programmed computer literally strong AI, specifically the claim that the appropriately programmed computer literally has cognitive states and that the programs thereby explain human cognition. When I has cognitive states and that the programs thereby explain human cognition. When I hereafter refer to AI, I have in mind the strong version, as expressed by these two hereafter refer to AI, I have in mind the strong version, as expressed by these two claims.claims.
I will consider the work of Roger Schank and his colleagues at Yale (Schank I will consider the work of Roger Schank and his colleagues at Yale (Schank & Abelson 1977), because I am more familiar with it than I am with any other similar & Abelson 1977), because I am more familiar with it than I am with any other similar claims, and because it provides a very clear example of the sort of work I wish to claims, and because it provides a very clear example of the sort of work I wish to examine. But nothing that follows depends upon the details of Schank's programs. The examine. But nothing that follows depends upon the details of Schank's programs. The same arguments would apply to Winograd's SHRDLU (Winograd 1973), Weizenbaum's same arguments would apply to Winograd's SHRDLU (Winograd 1973), Weizenbaum's ELIZA (Weizenbaum 1965), and indeed any Turing machine simulation of human ELIZA (Weizenbaum 1965), and indeed any Turing machine simulation of human mental phenomena.”mental phenomena.”
Proposed Solution:Proposed Solution:
A system that takes a question and a A system that takes a question and a document or a few documents, and document or a few documents, and returns a small set of text links where returns a small set of text links where potential answers to the question might potential answers to the question might be foundbe found
In order to accomplish this, we will In order to accomplish this, we will potentially use:potentially use: Techniques used in existing Question Techniques used in existing Question
Answering systemsAnswering systems Data collected from skimming text with an Data collected from skimming text with an
eye tracking deviceeye tracking device
WhatWhatArtArtMiddleMiddleinfusedinfusedpurpose purpose with with also servedalso servedpeople believedpeople believedwriting doeswriting doeswho readwho readSculpture. The missionSculpture. The missionas well as decorateas well as decorateBiblical talesBiblical taleslessons tolessons towerewerechurch sculpture; church sculpture; animalsanimalslifelife““Green man” peeringGreen man” peeringcarefullycarefullywroughtwroughtforthforthRomanesque eraRomanesque eraclassicalclassicalconventionsconventionsof figuresof figures
RomanesqueRomanesqueAt the beginningAt the beginningera the style ofera the style ofarchitecturearchitecturethat was in voguethat was in vogueKnown as Romanesque Known as Romanesque because it copied the patternbecause it copied the patternproportionproportionof the architectureof the architecturethe Roman Empirethe Roman Empirechief characteristics of the chief characteristics of the Romanesque style wereRomanesque style werevaults, round arches,vaults, round arches,and few windowsand few windowsThe easiest point to lookThe easiest point to lookfor is the rounded arch, seen for is the rounded arch, seen in door openingsin door openingswindowswindowsIn generalIn generalchurches were heavychurches were heavyCarrying about them an airCarrying about them an airsolemnity andsolemnity andThese earlyThese earlytapestries ortapestries orlook closelylook closely
werewereFrance called it “gothic”France called it “gothic”was a reference was a reference Ransacked RomeRansacked RometwilighttwilightarchitecturalarchitecturalRomanesqueRomanesquevaultsvaultsincorporatedincorporatedof windowof windowThe easiest point of The easiest point of archarchdoors. Alsodoors. Alsolater Gothiclater Gothicveryveryespecially theespecially thethethechurcheschurchesoutdo eachoutdo eachofofFor theFor theconstruction, througtconstruction, througtThe architectThe architectsame placesame place
Problem:Problem: Blind and dyslexic individuals cannot skim textBlind and dyslexic individuals cannot skim text
Example: “What’s the syntax for calling a function Example: “What’s the syntax for calling a function with template parameters?” (skimming through with template parameters?” (skimming through code)code)
““Why was Ayers Rock renamed?”Why was Ayers Rock renamed?” ““What type of tree produces leaves with three What type of tree produces leaves with three
distinct shapes?”distinct shapes?” ““Where can I find more information about Portugal?”Where can I find more information about Portugal?”
People who cannot read text rely on People who cannot read text rely on screen readers (Jaws, Window-Eyes)screen readers (Jaws, Window-Eyes) braille outputbraille output
more difficult to come by more difficult to come by extremely bulky to carry aroundextremely bulky to carry around
Example of Jaws Output at 400 Example of Jaws Output at 400 wpmwpm
LinkLink““What psychological and philosophical significance should we attach to What psychological and philosophical significance should we attach to
recent efforts at computer simulations of human cognitive capacities? In answering recent efforts at computer simulations of human cognitive capacities? In answering this question, I find it useful to distinguish what I will call "strong" AI from "weak" or this question, I find it useful to distinguish what I will call "strong" AI from "weak" or "cautious" AI (Artificial Intelligence). According to weak AI, the principal value of the "cautious" AI (Artificial Intelligence). According to weak AI, the principal value of the computer in the study of the mind is that it gives us a very powerful tool. For example, computer in the study of the mind is that it gives us a very powerful tool. For example, it enables us to formulate and test hypotheses in a more rigorous and precise fashion. it enables us to formulate and test hypotheses in a more rigorous and precise fashion. But according to strong AI, the computer is not merely a tool in the study of the mind; But according to strong AI, the computer is not merely a tool in the study of the mind; rather, the appropriately programmed computer really is a mind, in the sense that rather, the appropriately programmed computer really is a mind, in the sense that computers given the right programs can be literally said to understand and have other computers given the right programs can be literally said to understand and have other cognitive states. In strong AI, because the programmed computer has cognitive states, cognitive states. In strong AI, because the programmed computer has cognitive states, the programs are not mere tools that enable us to test psychological explanations; the programs are not mere tools that enable us to test psychological explanations; rather, the programs are themselves the explanations.rather, the programs are themselves the explanations.
I have no objection to the claims of weak AI, at least as far as this article is I have no objection to the claims of weak AI, at least as far as this article is concerned. My discussion here will be directed at the claims I have defined as those of concerned. My discussion here will be directed at the claims I have defined as those of strong AI, specifically the claim that the appropriately programmed computer literally strong AI, specifically the claim that the appropriately programmed computer literally has cognitive states and that the programs thereby explain human cognition. When I has cognitive states and that the programs thereby explain human cognition. When I hereafter refer to AI, I have in mind the strong version, as expressed by these two hereafter refer to AI, I have in mind the strong version, as expressed by these two claims.claims.
I will consider the work of Roger Schank and his colleagues at Yale (Schank I will consider the work of Roger Schank and his colleagues at Yale (Schank & Abelson 1977), because I am more familiar with it than I am with any other similar & Abelson 1977), because I am more familiar with it than I am with any other similar claims, and because it provides a very clear example of the sort of work I wish to claims, and because it provides a very clear example of the sort of work I wish to examine. But nothing that follows depends upon the details of Schank's programs. The examine. But nothing that follows depends upon the details of Schank's programs. The same arguments would apply to Winograd's SHRDLU (Winograd 1973), Weizenbaum's same arguments would apply to Winograd's SHRDLU (Winograd 1973), Weizenbaum's ELIZA (Weizenbaum 1965), and indeed any Turing machine simulation of human ELIZA (Weizenbaum 1965), and indeed any Turing machine simulation of human mental phenomena.”mental phenomena.”
Proposed Solution:Proposed Solution:
A system that takes a question and a A system that takes a question and a document or a few documents, and document or a few documents, and returns a small set of text links where returns a small set of text links where potential answers to the question might potential answers to the question might be foundbe found
In order to accomplish this, we will In order to accomplish this, we will potentially use:potentially use: Techniques used in existing Question Techniques used in existing Question
Answering systemsAnswering systems Data collected from skimming text with an Data collected from skimming text with an
eye tracking deviceeye tracking device
WhatWhatArtArtMiddleMiddleinfusedinfusedpurpose purpose with with also servedalso servedpeople believedpeople believedwriting doeswriting doeswho readwho readSculpture. The missionSculpture. The missionas well as decorateas well as decorateBiblical talesBiblical taleslessons tolessons towerewerechurch sculpture; church sculpture; animalsanimalslifelife““Green man” peeringGreen man” peeringcarefullycarefullywroughtwroughtforthforthRomanesque eraRomanesque eraclassicalclassicalconventionsconventionsof figuresof figures
RomanesqueRomanesqueAt the beginningAt the beginningera the style ofera the style ofarchitecturearchitecturethat was in voguethat was in vogueKnown as Romanesque Known as Romanesque because it copied the patternbecause it copied the patternproportionproportionof the architectureof the architecturethe Roman Empirethe Roman Empirechief characteristics of the chief characteristics of the Romanesque style wereRomanesque style werevaults, round arches,vaults, round arches,and few windowsand few windowsThe easiest point to lookThe easiest point to lookfor is the rounded arch, seen for is the rounded arch, seen in door openingsin door openingswindowswindowsIn generalIn generalchurches were heavychurches were heavyCarrying about them an airCarrying about them an airsolemnity andsolemnity andThese earlyThese earlytapestries ortapestries orlook closelylook closely
werewereFrance called it “gothic”France called it “gothic”was a reference was a reference Ransacked RomeRansacked RometwilighttwilightarchitecturalarchitecturalRomanesqueRomanesquevaultsvaultsincorporatedincorporatedof windowof windowThe easiest point of The easiest point of archarchdoors. Alsodoors. Alsolater Gothiclater Gothicveryveryespecially theespecially thethethechurcheschurchesoutdo eachoutdo eachofofFor theFor theconstruction, througtconstruction, througtThe architectThe architectsame placesame place
Current DirectionsCurrent Directions
Have collected eye-tracking data from close Have collected eye-tracking data from close to 100 people (on several documents each)to 100 people (on several documents each)
Analysis quite interesting – enough data to Analysis quite interesting – enough data to find patterns in where the skimmers are find patterns in where the skimmers are looking.looking.
Analyzing data with “text tiling methods” to Analyzing data with “text tiling methods” to pick out places in the text where “same pick out places in the text where “same thing” being discussed.thing” being discussed.
Incorporate question extraction techniquesIncorporate question extraction techniques How to present this to the user?How to present this to the user?
ModelingModeling the the Acquisition of English Acquisition of English in the ICICLE Systemin the ICICLE System
Kathleen F. McCoyDepartment of Computer and Information
SciencesUniversity of Delaware
PeoplePeople
Current PeopleCurrent People Rashida DavisRashida Davis Charlie GreenbackerCharlie Greenbacker
OthersOthers Chris Pennington, Dan Blanchard, Mike Bloodgood, Chris Pennington, Dan Blanchard, Mike Bloodgood,
Greg Silber, Meghan Boyle, Mohamed Mostagir, Greg Silber, Meghan Boyle, Mohamed Mostagir, Stephanie Baker, Heejong Yi, David DermanStephanie Baker, Heejong Yi, David Derman
Graduates: Matthew Huenerfauth, Jill Janofsky, Graduates: Matthew Huenerfauth, Jill Janofsky, Lisa Masterman Michaud, Litza Stark, David Lisa Masterman Michaud, Litza Stark, David SchneiderSchneider
The ICICLE ProjectThe ICICLE Project
IInteractive nteractive CComputer omputer IIdentification and dentification and
CCorrection of orrection of LLanguage anguage EErrorsrrors Interactive writing tutor for native Interactive writing tutor for native
signers of American Sign Language signers of American Sign Language (ASL)(ASL)
Purpose: analyze student-written Purpose: analyze student-written English texts and provide individualized English texts and provide individualized feedback and instruction on grammarfeedback and instruction on grammar
The ICICLE ProjectThe ICICLE Project
system provides student with system provides student with tutorial instruction on the errorstutorial instruction on the errors
student has opportunity to make student has opportunity to make corrections and request re-corrections and request re-analysisanalysis
TheICICLESystem
student provides piece of textstudent provides piece of text system analyzes text for system analyzes text for
grammatical errorsgrammatical errors
Cycle of user input, system Cycle of user input, system responseresponse
Current ImplementationCurrent Implementation
the student enters text
here
the system shows which
sentences have errors
explanations shown here
Writing From Deaf Writing From Deaf StudentsStudents
Literacy is a serious issue for the Deaf Literacy is a serious issue for the Deaf population.population.
Lots of variation in level of acquisition.Lots of variation in level of acquisition. Marked Differences from writing of Marked Differences from writing of
hearing peers.hearing peers. Dropped be: Dropped be: She really pretty.She really pretty. Missing Possessives: Missing Possessives: She age is 13.She age is 13. Subject/verb agreement, plural markers, Subject/verb agreement, plural markers,
determiners: determiners: She really like go with friend to She really like go with friend to mall.mall.
Work on ICICLEWork on ICICLE
Previous work focused on developing Previous work focused on developing grammar and mal-rules and modeling grammar and mal-rules and modeling the user’s level of acquisition (so the user’s level of acquisition (so different analyses can be found different analyses can be found depending on it)depending on it)
Current WorkCurrent Work Tutorial ResponsesTutorial Responses Probabilistic Parsing – need help!Probabilistic Parsing – need help! NEED SYSTEM HELP!!!!!NEED SYSTEM HELP!!!!!
What Mal-Rules do We What Mal-Rules do We Use?Use?
BeginnerBeginner:: Over-application of auxiliary Over-application of auxiliary IS, missing simple present IS, missing simple present morphology:morphology: She She teachesteaches piano on Tuesdays. piano on Tuesdays.
IntermediateIntermediate:: Botched progressive Botched progressive tense:tense: She is teachShe is teachinging piano on Tuesdays. piano on Tuesdays.
AdvancedAdvanced:: Botched passive voice: Botched passive voice: She is She is taughttaught piano on Tuesdays. piano on Tuesdays.
“She is teach piano on Tuesdays.”
Current DirectionsCurrent Directions
Have collected eye-tracking data from close Have collected eye-tracking data from close to 100 people (on several documents each)to 100 people (on several documents each)
Analysis quite interesting – enough data to Analysis quite interesting – enough data to find patterns in where the skimmers are find patterns in where the skimmers are looking.looking.
Analyzing data with “text tiling methods” to Analyzing data with “text tiling methods” to pick out places in the text where “same pick out places in the text where “same thing” being discussed.thing” being discussed.
Incorporate question extraction techniquesIncorporate question extraction techniques How to present this to the user?How to present this to the user?