nlp
DESCRIPTION
NATURAL LANGUAGE PROCESSING IN ARTIFICIAL INTELLIGENCETRANSCRIPT
Natural
Language
Processing
N HINIISHANT
By
What is Natural Language?
Language is meant for Communicating about the world.
By studying language, we can come to understand more about the world.
Refers to the language spoken by people, e.g. English, Japanese, Swahili, etc.
What Is Natural Language?
One of the aims of Artificial Intelligence (AI) is to build machines that can "understand" commands in natural language, written or spoken.
A computer that can do this requires very powerful hardware and sophisticated software.
At the present time, this is at the early stages of development.
Introduction to NLP
It is not an easy task to teach a person or computer a natural language.
The main problems are syntax (the rules governing the way in which words are arranged), and understanding context todetermine the meaning of a word.
To interpret even simple phrases requires a vast amount ofknowledge.
The basic goal of Natural language Processing is to enable aperson to communicate with a computer in a language that they use in their everyday life.
Natural Language And Computer Language
Natural language are those that we use for communicating with each other, eg. Arabic, English, French, Japanese, etc.
Natural language are expressive and easy for us to use.
Computer languages are those that we use for controlling the operations of a computer, eg. Prolog, C, C++, C#, Java, Python,…,etc.
Computer languages are easy for a computer to understand, but they are not expressive.
What is Natural Language Processing?
”Natural language processing (NLP) is a field of computer science, artificial intelligence (also called machine learning), and linguistics concerned with the interactions between computers and human (natural) languages.
Specifically, the process of a computer extracting meaningful information from natural language input and/or producing natural language output ”
Why Natural Language Processing?
kJfmmfj mmmvvv nnnffn333
Uj iheale eleee mnster vensi credur
Baboi oi cestnitze
Coovoel2^ ekk; ldsllk lkdf vnnjfj?
Fgmflmllk mlfm kfre xnnn!
Computers Lack Knowledge
•Computers “see” text in English the same you have seen the previous text!
•People have no trouble understanding language• Common sense knowledge• Reasoning capacity• Experience
•Computers have • No common sense knowledge• No reasoning capacity
Why Natural Language Processing?
Huge amounts of data
Internet = at least 20 billions pagesIntranet
Applications for processing large amounts of texts require NLP expertise
•Classify text into categories•Index and search large texts•Automatic translation•Speech understanding• Understand phone conversations
•Information extraction• Extract useful information from
resumes•Automatic summarization• Condense 1 book into 1 page
•Question answering•Knowledge acquisition•Text generations / dialogues
Why
is
Computer Processing
of
Human Language Difficult?
How can a machine understand these differences?
Decorate the cake with the frosting.Decorate the cake with the kids. Throw out the cake with the frosting.Throw out the cake with the kids.
Solution is
NATURAL LANGUAGE PROCESSING
How To Tackle These Problems?
Goals Of Natural Language Processing?
•Scientific Goal
• Identify the computational machinery needed for an agent to exhibit various forms of linguistic behavior
•Engineering Goal
• Design, implement, and test systems that process natural languages for practical applications
Where does it fit in the CS taxonomy?
Computers
Artificial Intelligence AlgorithmsDatabases Networking
Robotics SearchNatural Language Processing
InformationRetrieval
Machine Translation
Language Analysis
Semantics Parsing
Methods In Natural Language Processing
•Natural Language Understanding(NLU)The NLU task is understanding and
reasoning while the input is a natural language
•Natural Language Generation(NLG)NLG is a subfield of Natural Language Processing
NLG is also referred to text generation
Linguistic And Language Processing
Linguistic is the science of language. It study includes
•Sounds(phonology),
•Word formation(morphology),
•Sentence structure(syntax),
•Meaning(semantics) and Understanding(pragmatics) etc
Levels Of Linguistic Analysis
Steps in Natural Language Processing
Steps in Natural Language Processing
Morphological Analysis
Syntactic Analysis
Semantic Analysis
Discourse integration
Pragmatic Analysis
Natural Language Processing is done at 5 Levels
Morphological Analysis
Individual words are analyzed into their components and non-word tokens such as punctuation are separated from the words.
Morphology is the structure of words.
It is concerned with inflection.
It is also concerned with derivation of new words from existing ones.
In NLP, words are also known as lexicon items and a set of words form a lexicon.
WHY IS IT IMPORTANT
Why it is important?
Any NL analysis system needs a lexicon {a module that tells what words there are and what properties they have}.
Simplest model is a full form dictionary that lists every word explicitly.
Simply expanding the dictionary fails to take advantages of the regularities.
No dictionary contains all the words one is likely to encounter in real input.- Languages with highly productive morphology (e.g. Finnish, where a verb can have many thousands of forms.)- Noun Compounding
Morphological Analysis: Example
Suppose we have an English interface to an operating system and the following sentence is typed:
I want to print Bill’s .init file.
Morphological analysis must do the following things:
Pull apart the word “Bill’s” into proper noun “Bill” and the possessive suffix “’s”Recognize the sequence “.init” as a file extension that is functioning as an adjective in the sentence.
Syntactic Analysis
•Here the analysis is of words in a sentence to know the grammatical structure of a sentence.
•The words are transformed into structures that show the words relate to each others.
•Some word sequences may be rejected if they violate the rules of the language for how words may be combined.
•Example : “Boy the go the to store”
Syntactic Analysis : Example
S -SentenceNP -Noun PhraseVP -Verb PhraseDet-DeterminerN -Noun
John hit the ball
Semantic Analysis
Semantic analysis is concerned with the meaning of the language.
The first step in any semantic processing system is to look up the individual words in a dictionary(or lexicon) and extract their meanings.
Semantic Analysis
Unfortunately, many words have several meanings, for example, the word ‘diamond’ might have the following set of meanings:
(1) a geometrical shape with four equal sides.(2) a baseball field(3) an extremely hard and valuable gemstone
Semantic Analysis
The process of determining the correct meaning of an individual word is call word sense disambiguation or lexical disambiguation.
It is done by associating, with each word in the lexicon, information about the contexts in which each of the word’s senses may appear.
To select the correct meaning for the word ‘diamond’ in the sentence “Joan saw Susan’s diamond shimmering from across the room”.
It is necessary to know that neither geometrical shapes nor baseballfields shimmer, whereas gemstones do (process of elimination).
Semantic Analysis
Other useful semantic markers are
PHYSICAL-OBJECTANIMATE-OBJECTABSTRACT-OBJECT
Using these markers, the correct meaning of ‘diamond’ in the sentence “I dropped my diamond” can be computed.
As part of the lexical entry, the verb ‘drop’ will specify that its object must be a PHYSICAL-OBJECT.
Semantic Analysis
Finally, we have to process the text at sentence level. There are four approaches to this. •semantic grammar •case grammar •conceptual parsing •approximately compositional semantic interpretation.
Discourse Integration
Once the correct reference for it is known, we can also determine exactly which it is being referred to.
The meaning of individual sentence may depend on the sentences that precede it and may influence the meaning of the sentences that follow it
Example : “You wanted it”
Pragmatic Analysis
The final step in pragmatic processing is to translate, from the knowledge based representation to a command to be executed by the system.
This is an additional stage of analysis concerned with the pragmatic use of the language.This is important in the understanding of texts and dialogues.
The idea is, what was said is reinterpreted to determine what was actually meant.
Example: “Do you know what time it is?”Should be interpreted as a request.
Real World NLP Application
Machine Translation
Information Retrieval / NL interface
Information Visualization
Autonomous interacting bots
Grammar Checking Systems
Speech Recognition Systems / Speech Synthesizers
Document Summary Systems
Machine Translation: Deluxe Universal Translator
Information Retrieval: Buzzcity
AltaVista Search Engine
AltaVista Search Engine
Information Visualization: Cartia’s Themescape
Autonomous interacting bots: Eliza’s grand-daughter -Lisa
http://stuff.simplenet.com/files/doorsam/lisa18.zip
Grammar Checking Systems: MS Word Grammar Checker