An Evaluation Tool for Natural An Evaluation Tool for Natural Language Processing Language Processing

SystemsSystems

Audrey N. Mbeje

Department of Computer Science Ball State University

November 09, 2000

Contents

V. IntroductionVI. Problem DescriptionVII.Significance of the StudyVIII.Definition of TermsIX. Computational LinguisticsX. ContextXI. Literature ReviewXII.MethodologyXIII.Anticipated ResultsXIV.Time ScheduleXV.DeliverablesXVI.Future Research &

Conclusion

Problem Description

Problem Background:Human interactive discourse provides many challenges for natural language processing (NLP) systems. One ofthe main challenges is representing the speaker’s intended meaning in its context. Thus the focus ofcurrent research on NLP has been to develop thetechnology that will enable the computer to understandnews events in the context they occur in the real world.

The evolving technology, however, is linguistically inclined and is less concerned about the quality of thesoftware. Additionally, it does not reflect uniform principles of software evaluation.

Goal: The goal of the proposed study is to improve thequality of the natural language processing technologyby assessing NLP system inventions for linguistic and technical quality assurance before they are implemented.

We are suggesting a natural language processing system evaluation tool that will provide both the linguistic and software quality assurance. The proposed study is based on the assumption that progress in developing NLP technology depends on using evaluation methods that better model the speakers’ natural discourse and the quality software.

Significance of the Study

•The study will benefit the theory of natural languageprocessing, particularly the research area concerned with context in NLP systems.

•The study is proposing an integration of linguistic principles and software design principles in NLP systemsevaluation which would be a contribution in the currentprogress in NLP technology.

•The proposed tool will improve the NLP system usability by offering quality assurance for reliability and validity of the software technically and linguistically.

Definition of Terms

1. Computational Linguistics:-Discipline between linguistics and computer sciencewhich is concerned with the computational aspects ofhuman language faculty.

-Belongs to the cognitive sciences, artificial intelligence (AI) specifically.

-Has two componentsapplied and theoretical

Definition of Terms (cont’d)

-With the applied component the interest is in thepractical outcome of modeling human languageuse. The goal is to create software products that have some knowledge of human language.

-The theoretical aspect deals with issues of formaltheories about the linguistic knowledge that ahuman needs for generating and understandinglanguage.

(The proposed evaluation tool is intended for the appliedcomponent of CL.)

Definition of Terms (cont’d)

2. Context:-Rough definition of the term

-We say that an utterance x presupposes a fact y,if uttering x only makes sense if the context (e.g., world knowledge or earlier utterance in the sameconversation) provides enough information to conclude that y is the case. Consider example 2a

2a. Mary’s husband is out of town.The noun phrase presupposes Mary is married.

Computational linguists are concerned with making NLPsystems understand such contextual information.

Literature Review

•Much research on the problem of in-depth story understanding by computer was performed starting in the 1970’s. •In the 1990’s the interest shifted towards information extraction and word sense disambiguation.

•The end of the 1990 marked another shift in focus backto in-depth story understanding by the computer.

•McCarthy (1990) discusses the problem of getting the computer to understand the following text from the New York Times:

A 61-year old furniture salesman was pushed down the shaft of a freight elevator yesterday in his downtown Brooklyn store by two robberswhile a third attempted to crush him with the elevator car because they were dissatisfied with the $1,200 they had forced him to give them. Thebuffer springs at the bottom of the shaft preventedthe car from crushing the salesman John J. Hug, afterhe was pushed from the first floor to the basement.

The car stopped about 12 inches above him as he flattened himself at the bottom of the pit.

(Mueller, 1999)

McCarthy’s concern was beyond mere word sensedisambiguation and information extraction. He suggested that the computer should be able to demonstrate such contextual questions as:

Who was in the store when the events began?Who had the money at the end?What would have happened if Mr. Hug had notflattened himself at the bottom of the pit? etc.

Literature Review (cont’d)

Current research on contextual understanding is concerned with such problems as the one stated above.

Several NLP systems have been suggested whose orientations is mainly linguistic.

This study is suggesting an evaluation tool for suchNLP systems integrating linguistic and technical principles, namely, speed.

Methodology

•Create an algorithm simulating aspects of human language faculty, namely, speed and ability to decode contextual discourse.

-Evaluation technologies to evaluate the NLP systems for context decoding and speed using existing evaluation technology.

Methodology (cont’d)

-Do the same test using the proposed tool.

-Compare the results

Note: The proposed evaluation tool will be evaluatedfor validity and reliability before its implementation using outside researcher’s evaluation tool.

Anticipated Results

•The proposed tool should effectively evaluate NLPsystems for context and speed.

Time Schedule

August - November: Proposal Writing & Presentation

November - December: Proposal Review

January – March: Literature Review

April – July: Data Gathering Evaluation Tool Designing Evaluation Tool Testing

August - November: Thesis Writing & Defense

Deliverables

1. Natural Language Processing Evaluation Tool2. Research Presentation at a Conference3. Research Publication

Conclusion and Future Research

•Computing context of a natural language discourse is an essential task for a natural language processing system.

•The proposed evaluation tool for NLP system will havea potential for modification to incorporate new design principles for improved usability.

The End******** ********