two and a half approaches to natural language processing

Two and a half approaches to natural language

processing in computational biology

Kevin Bretonnel CohenBiomedical Text Mining Group Lead

Center for Computational PharmacologyUCHSC, Fitzsimons [email protected]

There’s natural language processing in computational

biology??

What’s it doing there?

(One lab’s) funding for NLP in computational biology

• INIA (Neuroinformatics of Alcoholism) $5M, 5 years

• Wyeth Genomics Institute ($200K, 2 years)

• National Library of Medicine ($4.2M, 3 years)

• …

Why biologists care

• High-throughput data interpretation• Literature search• Annotation• Database construction

But, I’m a computer scientist (mathematician, engineer…)

• Hard, but might be possible• Might be harder in molbio than in

“General English”• Might be more possible in molbio than

in “General English”

ResourcesThe big drawing point for “bionlp”

• DataUMLS, Gene OntologyPubMedLabelled training data from “bake-offs”

• Tools lvg (“Lexical Variant Generator”), entity identification systems, …

Job market

• Academia: greatUS, Europe

• Industry: even better, although not molbio-specific

Overview

• Some definitions• Specific tasks• General issues• Two (and a half) approaches

Natural language vs. artificial language

• 2 + 3 * 4== 14!= 20

Natural language vs. artificial language

• 2 + 3 * 4== 14!= 20

• fatty acids and cholesterol(fatty acids) and cholesterolfatty (acids and cholesterol)

The basic applications (biologist’s-eye view)

• High-throughput data interpretation• Literature search• Annotation• Database construction

The basic tasks(computer-scientist’s-eye view)

Information extractionEntity identificationIndexing

What processing means

• High-levelEntity identificationInformation extractionInformation retrieval

• Lower-levelTokenizationPart-of-speech taggingSyntactic analysis

These are prerequisites to (most of) the others…

Information extraction

Given the template:

BINDING_EVENTBinder:Bound:


…and the input:

Met28 binds to DNA.


…return:

BINDING_EVENT

Binder: Met28Bound: DNA

Second thing that makes it hard: morphosyntactic variability

Met28 binds to DNA…binding of Met28 to DNA……Met28 and DNA bind……binding between Met28 and DNA……Met28 is sufficient to bind DNA……DNA bound by Met28…


…binding of Met28 to DNA……binding under unspecified conditions

of Met28 to DNA……binding of this translational variant of

Met28 to DNA……binding of Met28 to upstream regions

of DNA…


…binding under unspecified conditions of this translational variant of Met28 to upstream regions of DNA…

Even before that, though…

HSP60Hsp-60heat shock protein 60CerberuswinglessKen and Barbiethe

Even before that, though…

HSP60Hsp-60heat shock protein 60CerberuswinglessKen and Barbiethe

Entity identification

How they connect

• Entity identification

• Information extraction

• Indexing

• Literature search

• Annotation

• Database construction

…and none of those are what we REALLY want…

• “text data mining:” finding knowledge that isn’t explicitly stated

Question-answering“Missing links” SummarizationInference

Why is NLP hard?

Why is NLP hard?

• 2 + 3 * 4== 14!= 20

• fatty acids and cholesterol(fatty acids) and cholesterolfatty (acids and cholesterol)

Types of ambiguity

• Part of speech: Is this a noun, a verb, an adjective…

• Lexical: which word is this?• Structural: what’s the relationship

between these words/phrases?• Semantic, pragmatic, discourse-level…

Part-of-speech ambiguity

• adjective • noun

• heat shock protein 60heat/VERB shock/VERB protein/NOUNheat/NOUN shock/ NOUN protein/NOUN

Lexical ambiguity

• Flying mammal • Sporting equipment

Lexical ambiguity

Lexical ambiguitya verb with 14 meanings

• 7 a : to assemble and set alight the materials for (a fire) b : to set in order <make beds>

• 3 a : to bring into being by forming, shaping, or altering material : FASHION <make a dress>

http://www.m-w.com/cgi-bin/dictionary?book=Dictionary&va=fashion

• hunkhuman natural killer (cell type)HUN kinase (gene/protein)radiological/orthopedic classification schemepiece of something

A reminder about what syntactic structure is

• Relations between phrases (groups of words)

I dislike stupid people like you I find them boring



[I dislike stupid people][like you I find them boring]



[I dislike stupid people like you][I find them boring]



I saw the man with the binoculars




Paraphrase: using the binoculars, I saw the man




Paraphrase: the man that I saw had some binoculars

Structural ambiguity


• Clerk interprets:• Verb (blouse) (in

window)

• She means:• Verb (blouse (in

window))


• Sly dog means:Verb (in (house (on

bed)))

• She interprets:• Verb (in house) (on

bed)

• NaCT is expressed in liver, testis and brain in rat and shows preference for citrate over dicarboxylates.


• (liver), (testis) and (brain in rat)• liver, (testis and brain in rat)• (liver, testis and brain in rat)


• shows preference for (citrate over dicarboxylates)

• shows preference (for citrate) (over dicarboxylates)

Other syntactic ambiguity(ellipsis)

Two approaches to NLP

Rule-based Statistical/machine learning

First approach to NLP

• Rule-based• AI, linguistics• Patterns (regular, context-free…)• Procedures

Rule-based: regex

• Patterns (regular, context-free, …)

• Procedures

$geneName = “[A-Za-z]+-?[0-9]”;

$input =~ /interaction of ($geneName) with ($geneName)/;

$interactionAssertion->setGene1($1);

$interactionAssertion->setGene2($2);

Rule-based: CFGs


• Procedures

NounPhrase -> NounPhrase+ Conjunction NounPhrase

NounPhrase -> Predeterminer Determiner+ Adjective+ Noun

Rule-based: procedural


• Procedures

if (currentWordEndsWith-ing) {

if (previousWordIsThe) {

if (nextWordIsOf) {

Rule-based approachesWhy they work

• Patterns are realPsychologicallyFormally adequate (mostly)

• Intuition works• No need for training data

Rule-based approachesWhy they’re hard

• Knowledge takes time to get• Process of developing large rule sets

can be slowConsider English syntax…

Second approach to NLP

• Mosteller & Wallace• Bayesian• Other machine learning techniques

Statistical approaches

Bayesian statistics: Why this is funny

Statistical/ML approaches

• Frame the NLP task as a series of classification problems

Which POS is this?Which word meaning?Which phrasal grouping?

Specifics of statistical approaches

• Training data: 100 examples of “bat”• 95 times, meaning is “sporting

equipment”• 5 times, meaning is “mammal”


• You see “bat:” what does it probably mean?


• 95 examples mean “sporting implement”

• 95 times, word “ball” also appears


• 5 examples mean “mammal”• Each time, word “wizard” also appears


• You see the word “bat,” and you also see the word “wizard…”

Statistical approachesWhy they work

• Statistics can be proxy for knowledge• Some interesting stuff is frequent

enough to be tractable

Statistical approachesWhy they’re hard

• Problem: sparse data

Frequency

Rank


• Solutions: smoothing, back-off


• Problem: labelled training data is expensive


• Solutions: spend moneyfigure out how to use other peoples’“weakly labelled” data

Rule-based or statistical: what to do??

Rule-based vs. statistical approaches

• Picking one:Is it cheaper to label more training data, or to put time into developing patterns?


• Combine them:Do both together/iterativelyStatistical solution first, then rule-based post-processing

the 2.5th

approach


• Researcher’s answer:Use one as the baseline for the other


Domain specificity makes both of them more

tractable


Sublanguage model

Two extended examples

POS tagging: why you need it

• All syntax is built on it• Overcome sparseness problem by

abstracting away from specific words• Potential basis for entity

identification

What “POS tagging” is

• POS: part of speech• School: 8 (noun, verb, adjective,

interjection…)• Real life: 40 or more

How do you get from 8 to 80?

• Noun • NN (noun, singular or mass)• NNS (plural noun)• NNP (proper noun)• NNPS (plural proper noun)

How do you get from 8 to 80?

• Verb • VB (base form)• VBD (past tense)• VBG (gerund)• VBN (past participle)• VBP (singular present-tense non-

3rd-person)• VBZ (3rd-person singular present

tense)

How do you define noun, verb, etc.?

• Semantic: “A noun is a person, place, or thing…”“A verb is…”

• Distributional characteristics:

“A noun can take the plural and genitive morphemes”“A noun can appear in the environment All of my twelve hairy ___ left before noon”

POS tagging defined (or, why’s it interesting?)

• Given:Time flies like an arrow, but fruit flies like a banana.

• Do:Time flies/VBZ like/IN an arrow, but fruit flies/NNS like/VBP a banana.

A statistical approach: TnT• Second-order Markov model

• Smoothing by linear interpolation of ngrams

• λ estimated by deleted interpolation• Tag probabilities learned for word endings;

used for unknown words

TnT• Ngram: an n-tag or n-word sequence• N = 1

DETNOUNrole

• BigramsDET NOUNNOUN PREPOSITIONa role

• Trigrams

The Brill Tagger

The Brill tagger

• Iterative error reduction1. Assign most common tags, then2. Evaluate performance, then

The Brill tagger

• Iterative error reduction1. Assign most common tags, then2. Evaluate performance, then3. Propose rules to fix errors4. Evaluate performance, then5. If you’ve improved, GOTO 3, else END

The Brill tagger

• Change Determiner Verb “of”• …to…• Determiner Noun “of”

The/Determiner running/Verb of/IN

The/Determiner running/Noun of/IN

EI as POS tagging

• Retinoic acid downmodulateserythroid differentiation and GATA1 expression in purified adult-progenitor culture. (Labbaye et al. 1994)

• Retinoic/JJ acid/NNdownmodulates/VBZ erythroid/JJdifferentiation/NN and/CCGATA1/NN expression/NN in/INpurified/VBN adult-progenitor/JJculture/NN ./. (Labbaye et al. 1994)

• Retinoic/JJ acid/NNdownmodulates/VBZ erythroid/JJdifferentiation/NN and/CCGATA1/GENE expression/NN in/INpurified/VBN adult-progenitor/JJculture/NN ./. (Labbaye et al. 1994)

Rule-based post-processing

• 14 simple patterns for adjusting boundariesif current_word == (gene OR mutant etc.) && previous_word_tag == GENE then current_word_tag = GENE

if current_word =~ /digit|Roman numeral|Greek letter/ && previous_word_tag == genethen current_word_tag = GENE

Results (BioCreative)Precision and Recall : Official Score

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0Recall

Pre

cis

ion

No post-processing

Closed Division

Open Division

High

Median

Low

How to get started

• Certificate program project• BIOI 7713 project• “Bake-offs”

BioCreativeTREC

• BIOI 7791, Spring quarter, HSC/Fitz• NLP meetings: 2 p.m. Wednesdays,

Room 6106, South Tower

Coursework

• CSCI 5832, Natural language processingJim MartinBoulder campus, Spring semesterhttp://www.cs.colorado.edu/~martin/csci5832.html

• Independent Study• BIOI 7791• BIOI 7713

http://www.cs.colorado.edu/~martin/csci5832.html

http://www.cs.colorado.edu/~martin/csci5832.html

Miscellaneous extra slides for answering questions, etc.

How to read a paper in “BioNLP”

• Evaluation setExclusions

Evaluation

• P/R/F-measure

The other application of NLP in computational biology

• Sequence dataHow do you know a gene/promoter/whatever when you see one?Grammar-based approaches (David Searls)HMM’s

<happy-face photo>

One year later…

A sad story: physicians don’t buy a lot of NLP software

Another sad story: trying to sell “gisting” to physicians

Sold for $400K: 14.5 or 2.9¢ on the dollar…

Salesperson’s thought process

You have a problem

I can solve it for you

Physician’s thought process

Have I been sued over how I do this?

I have a problem

I don’t have a problem

YesNo

Can you solve it for me?

Cooccurrence

Pleonastic vs. referential ambiguity

Edit distance

two and a half approaches to natural language processing

Documents