practical natural language processing from theory to industrial applications

Practical Natural Language ProcessingFrom Theory to Industrial Applications

Jaganadh Ghttp://jaganadhg.in

[email protected]

Karpagam UniversityCoimbatore

19th March 2012

Jaganadh G Practical Natural Language Processing

About me !!

Working in Natural Language Processing, MachineLearning, Data Mining etc...

Passionate about Free and Open source :-)

When gets free time teaches Python, Speaks about FOSSand blogs athttp://jaganadhg.in

I am a computational linguist / Linguist and Indologist,Book reviewer

Software Engineer by Profession

Question ??

Have you ever used any Natural Language Processing basedtools/services?

Question ??

What is Natural Language Processing (NLP) ?

Aim : To build intelligent systems that can interact withhuman beings as like human beings

A sub-field of Artificial Intelligence (AI)

Inter-disciplinary subject (Language + Linguistics +Statistics + Computer Science + .. )

Natural Language

Refers to the language spoken by people, e.g.English,Japanese, Tamil, Malayalam as opposed to artificiallanguages, like C++, Java, etc.

Natural Language

Definition

Natural Language Processing

Natural Language Processing is a theoretically motivated rangeof computational techniques for analyzing and representingnaturally occurring texts/speech at one or more levels oflinguistic analysis for the purpose of achieving human-likelanguage processing for a range of tasks or applications.

NLP was considered as an academic discipline beforesome 10 to 20 years.

Now concepts from NLP is applied in variety ofComputing Platforms and Services

Practical NLP ?

Problem

Before going to some theory can we have some funnypractical problems to solve ?

Picture Courtesy: http://twitpic.com/1y21qm/full

Practical NLP ?

Problem

Practical NLP ?

Problem

Practical NLP

Problem

Tweet-a-Toddy receives thousands of tweets per day

Tweets requesting home deliveryTweets about quality of productsTweets related to enquirers

They requires following things to be automated

Identify tweet categoryProcess home-delivery requestEvaluate quality related tweets

How to find a solution for Tweet-a-Toddy

Practical NLP

Problem

Practical NLP

Problem

Tweets requesting home delivery

Tweets about quality of productsTweets related to enquirers

Practical NLP

Problem

Tweets requesting home deliveryTweets about quality of products

Tweets related to enquirers

Practical NLP

Problem

Practical NLP

Problem

Practical NLP

Problem

Identify tweet category

Process home-delivery requestEvaluate quality related tweets

Practical NLP

Problem

Identify tweet categoryProcess home-delivery request

Evaluate quality related tweets

Practical NLP

Problem

Practical NLP

Problem

Solution

Any Solutions

Some thoughts

Text Classification

Entity Identification

Information Extraction

Sentiment Analysis

Parsing, gammer ...

Regex (Regular Expressions)

Solution

Any Solutions

Some thoughts

Text Classification

Sentiment Analysis

Parsing, gammer ...

Solution

Any Solutions

Some thoughts

Text Classification

Sentiment Analysis

Parsing, gammer ...

Solution

Any Solutions

Some thoughts

Text Classification

Sentiment Analysis

Parsing, gammer ...

Solution

Any Solutions

Some thoughts

Text Classification

Sentiment Analysis

Parsing, gammer ...

Solution

Any Solutions

Some thoughts

Text Classification

Sentiment Analysis

Parsing, gammer ...

Solution

Any Solutions

Some thoughts

Text Classification

Sentiment Analysis

Parsing, gammer ...

Solution

Any Solutions

Some thoughts

Text Classification

Sentiment Analysis

Parsing, gammer ...

Another Practical Question

Everybody might have used spell checker available in wordprocessing systems like OpenOffice.org or Microsoft WordAny guess on how to develop a spell checker system ?

Solutions

Word List

Structure of words

Dynamic Programming (Edit Distance)

Solutions

Word List

Structure of words

Solutions

Word List

Structure of words

Solutions

Word List

Structure of words

Another Practical Question ...

Context Sensitive Spell-checking

Identifying and suggesting spelling of words based on contextHow ??

Solutions

Statistical Models

Word category based suggestions

Solutions

Statistical Models

Solutions

Statistical Models

Solutions

Statistical Models

Can Machines Translate ??

Answer !!!

Why NLP ?

Because ”Information is Power !!!”

Every day wast amount of text and speech data is beingproduced

Internet == at least 40 Million pages

Picture Courtesy: http://soundsgood.in/wikipediafat print book/

Why NLP ?

History

Second World War !!!

Machine Translation

Most promising imperfect technology

Moves from Lab to Industry to Layman

History

Machine Translation

History

Machine Translation

History

Machine Translation

History

Machine Translation

History

Machine Translation

NLP Really Hard to Achieve?

NLP delas with human languagesHuman Language is dynamic and mysterious !!!

Communication in Human Language

NLP delas with human languagesHuman Language is dynamic and mysterious !!!

Communication in Human Language

Levels of Knowledge encoding in Language Data

Tasks in NLP

Broad Areas

Text Processing

Speech Processing

Tasks in NLP

Broad Areas

Text Processing

Speech Processing

Tasks in NLP

Broad Areas

Text Processing

Speech Processing

Major tasks in Text Processing

Word Level Analysis

Morphological SynthesisPart of Speech TaggingStemmingLemmatization

Sentence Level Analysis - Syntactical Parsing

Discourse Analysis - Semantic Processing

Word Level Analysis

Morphological Synthesis

Part of Speech TaggingStemmingLemmatization

Word Level Analysis

Morphological SynthesisPart of Speech Tagging

StemmingLemmatization

Word Level Analysis

Morphological SynthesisPart of Speech TaggingStemming

Lemmatization

Word Level Analysis

Morphology

The branch of linguistics that studies word structures.

To a computer program a word is : ???

Morphological analysis can be explained as: the process ofanalyzing words to identify its constituents

Computational Analysis of Morphology

Morphological Analysis

Morphological Generation

Stemming

Lemmatization

Morphology

Stemming

Lemmatization

Morphology

Stemming

Lemmatization

Morphology

Stemming

Lemmatization

Morphology

Stemming

Lemmatization

Morphology

Stemming

Lemmatization

Morphology

Stemming

Lemmatization

Morphology

Stemming

Lemmatization

Practical Question from Morphology

Approximate number of word forms that can be derived from

the word”maram”

Parts of Speech Tagging

POS tagging is the process of marking up the words in a text(corpus) as corresponding to a particular part of speech, basedon both its definition, as well as its context.Ram goes to school.Ram/NNP goes/VBZ to/TO school/NN ./.

Words are ambiguous !!!!e.g. book, cricket, bank

Parts of Speech Tagging

POS tagging is the process of marking up the words in a text(corpus) as corresponding to a particular part of speech, basedon both its definition, as well as its context.Ram goes to school.Ram/NNP goes/VBZ to/TO school/NN ./.

Words are ambiguous !!!!e.g. book, cricket, bank

Syntactical Parsing

Parsing

In computer science and linguistics, parsing, or, more formally,syntactic analysis, is the process of analyzing a text, made of asequence of tokens (for example, words), to determine itsgrammatical structure with respect to a given (more or less)formal grammar.

Sentences are ambiguous !!!!

Syntactical Parsing

Parsing

In computer science and linguistics, parsing, or, more formally,syntactic analysis, is the process of analyzing a text, made of asequence of tokens (for example, words), to determine itsgrammatical structure with respect to a given (more or less)formal grammar.

Sentences are ambiguous !!!!

Semantics

Study of meaning ans its structure

Word meaning is ambiguous !!!!E.g. marriage

Semantics

Study of meaning ans its structure

Word meaning is ambiguous !!!!E.g. marriage

Where can I apply this techniques?

Machine Translation Systems

Search Engine

Spell-checker

Grammar Checker

..........

Search Engine

Spell-checker

Grammar Checker

..........

Search Engine

Spell-checker

Grammar Checker

..........

Search Engine

Spell-checker

Grammar Checker

..........

Search Engine

Spell-checker

Grammar Checker

..........

Other Interesting Tasks

Named Entity Identification

Information Retrieval

Text Classification and Clustering

Speech Processing

Two Major Areas

Text to Speech

Speech Recognition

Practical Applications

Technology for Visually Challenged People

Mobile Phones

Speech Enabled Web

Vehicle Mounted GPS Navigator

Speech Processing

Two Major Areas

Text to Speech

Speech Recognition

Practical Applications

Technology for Visually Challenged People

Mobile Phones

Speech Enabled Web

Vehicle Mounted GPS Navigator

Commerical NLP Applications

What Industry Looks

Components of Word Processors

Custom Search Systems

Text Summarization

Speech Systems

Question Answering Systems

What Industry Looks

Text Summarization

Speech Systems

What Industry Looks

Text Summarization

Speech Systems

What Industry Looks

Text Summarization

Speech Systems

What Industry Looks

Text Summarization

Speech Systems

What Industry Looks

Text Summarization

Speech Systems

What Industry Looks

Text Summarization

Speech Systems

What Industry Looks

Text Summarization

Speech Systems

What Industry Looks

Text Summarization

Speech Systems

Future of NLP

Future!!!

Semantics oriented technologies

NLP in other domains

Bio-Medical

Forensic Science

Education

Politics

E-governance

Business Development

Marketing

and where ever we use language !!!

Natural Language Processing in India

Academic Institutions

IIT Kanpur, Kharagpur, Bombay

IIIT hydrabad

IISc Bangalore

AU-KBC Chennai

Amritha University Ettimadai, Coimbatore

IIITMK, Trivandrum

Central University, Hydrabad

JNU, Delhi

Tamil University, Thanjore

Natural Language Processing in India

Industry

Microsoft

Yahoo!

365Media Pvt. Ltd.

Inside View

Thaazza

AIAIO Labs

Questions ??

References

Daniel Jurafsky,James H. Martin, SPEECH andLANGUAGE PROCESSING, 2nd Edition.

U.S. Tiwary, Tanveer Siddiqui , Natural LanguageProcessing and Information Retrieval

Finally

Questions ??

References

Daniel Jurafsky,James H. Martin, SPEECH andLANGUAGE PROCESSING, 2nd Edition.

U.S. Tiwary, Tanveer Siddiqui , Natural LanguageProcessing and Information Retrieval

Finally

practical natural language processing from theory to industrial applications

Technology

practical nlp problem

funny practical problems

thousands of tweets

home deliverytweets

quality of productstweets

withhuman beings

following things

intelligent systems