noun compound interpretation - iit bombay you have a dictionary of words ... automatic noun compound...

Noun Compound Interpretationby

Girishkumar Ponkiya

Supervisor: Prof. Pushpak Bhattacharyya

Co-supervisor: Mr. Girish K Palshikar (TRDDC, Pune)

June 21, 2015

Outlines

• Introduction

• Problem Definition

• Two Approaches• Automatic Rule Based

• Deep Learning

• An interesting problem

2Noun Compound Processing21/6/2016

Introduction

• Noun Compound (NC): sequence of two or more nouns that act as a single noun.

Example: apple pie, student protest

• Task: interpret the meaning of English NCs (bi-gram)• Labeling: relationship of modifier with the head noun.

Apple pie : Made-Of

• ParaphrasingApple pie : “a pie made of apple” or “a pie with apple flavor”

Noun Compound Processing 321/6/2016

Motivational Example

• “Honey Singh became the latest victim of celebrity death hoax.”

• Machine Translation:• [Hindi] हनी स िंघ प्रस द्ध व्यक्ति की मौत के बारे में अफवाह के ताजा सिकार बने।

“Hanī siṅgha prasid'dha vyakti kī mauta kē bārē mēṁ aphavāha kē tājā śikāra banē.”

• Question Answering:• What type of rumor was spread about Honey Singh?

• Text Entailment:• H: Honey Singh is dead. (False)


More Examples..

• “Japan May exports fall on earthquake disruption”The Economic Times (June 20, 2016)

• Noun order in a compound:Mosquito malaria v/s malaria mosquito

Adult male rat v/s male adult rat

• Groping is important in long sequences:"plastic water bottle" v/s "water bottle cap"


Noun Compound Processing

1. IdentificationIdentification of noun compounds from a sentence (or text)

• Student protest is a common thing in West Bengal.• Kindly throw used plastic water bottles in the trash.• The term weblog was first coined in 1997, 3 years after the first dynamic website was created.• Some kelp products are snake oil, but the good ones promote plant growth.

2. Parsing (if necessary)tumor suppressor protein ⇒ [ tumor suppressor ] protein

3. InterpretationWe are going to discuss this in this presentation


Problem Definition

• Given: a two-word English noun compound

• Output: assign an abstract label (relationship of modifier with the head)

• Dataset:1. Kim and Baldwin (2005)

• “KB05” (20 relations; 2084 compounds)

• Best result: 53.38% accuracy (using WordNet similarity)

2. Tratz and Hovy (2010)• “TH10” (43/37 relations; 19158 compounds)

• Best result: 79.3% accuracy (multiclass SVM), and 77.7 average F-score (using neural network)


Semantic Relationsused by Kim and Baldwin (2005)


Challenges

• The compounding process is highly productiveIn BNC, 60.3% of total noun compounds appears only once (Baldwin and Tanaka, 2004)

• The semantic relation is implicitThe relation of a modifier noun with the head noun in a compound is not mentioned explicitly.

• Contextual pragmatic factors influence the interpretationWhy students are ‘beneficiary’ in student price?


Illustrative Literature SurveyN

ou

n C

om

po

un

d

Inte

rpre

tati

on Attributional Methods

(Based on the similarity between the respective components of

compounds)

Relational Methods(Model relation directly)

Probabilistic Modeling𝑃(𝑟|𝑛1, 𝑛2) or 𝑓(𝑛1, 𝑝, 𝑛2):tri-gram

using predicates

ParaphrasingUsing some patterns, or verbs and

predicates

Explicit ParaphrasingFrom predefines patterns, fine the

best for given NC

Implicit ParaphrasingCreate a vector of frequency of each paraphrase, and use an algorithm to

classify.

Noun Compound Processing 10

Rule Based System

Deep Learning

21/6/2016

Our Approaches

• Rule Based• Extracted rules using CN2

• Probabilistic inferencing using MLN (Markov Logic Network)

• Deep Learning• Learn embedding for noun compounds

• A new problem• Coherence in noun compounds


Example: student protestHypernym paths for each of the components

• entity.n.01,

• physical_entity.n.01,

• causal_agent.n.01,

• person.n.01,

• enrollee.n.01,

• student.n.01

.

• entity.n.01,

• abstraction.n.06,

• psychological_feature.n.01,

• event.n.01,

• act.n.02,

• speech_act.n.01,

• objection.n.02,

• protest.n.01


Example: student protestHypernym paths for each of the components, with default value for the rest

1. entity.n.012. physical_entity.n.013. causal_agent.n.014. person.n.015. enrollee.n.016. student.n.017. X8. X9. x

1. entity.n.012. abstraction.n.063. psychological_feature.n.014. event.n.015. act.n.026. speech_act.n.017. objection.n.028. protest.n.019. X


CN2: inputs

• Attribute file: description and domain of each attribute

• Example file: Vector for each example

• Params:

• Error function: laplacian, naïve

• Star size: for bean search

• Threshold: stop condition


CN2: Outputs

• Rule file: ordered/unordered rules

• An example rule:

IF "u.5" = N_08050385 AND "v.6" = N_00575741

THEN labels = agent [0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0]

N_08050385 : polity.n.02

N_00575741 : work.n.01

• Evaluation matrix


CN2 Results


MLN (Alchemy)

Training

• Input:• MLN definition file

• Domain(s) of variables (optional)• predicate declaration• logic in terms of formula

• Database file• Evidence and outcomes

• Output:• MLN file with weight assigned to

each of the formulas

Testing

• Input:• MLN file with weight assigned to

each of the formulas

• Database file: evidence only

• Output:• Probability for each ground atom

of outcome predicate


MLN file format// Domain for each of the attributes (optional)

u0 = {}

u1 = {}

...

labels = {AGENT, BENEFICIARY, CAUSE, ..., TOPIC}

// Class

Outcome(nc, labels!)

// Predictors

U0Value(nc, u0)

U1Value(nc, u16)

...

// Rules

V6Value(x, "creation.n.01") ^ U2Value(x, "group.n.01") => Outcome(x, AGENT)

...


MLN Results


Our Approaches






Word/Text Representation

• 1-hot representation

• Bag-of-words representation

• Don’t capture similarity between synonyms• Two words with different surface forms have orthogonal representation, i.e.,

dot product is 0.

• Dimension of a vector equals to number of words in vocabulary, which is in terms of thousands in most!!

• Can we have better presentation which can capture “similarity” between words?


Word Embeddings

Suppose you have a dictionary of words

• 𝑑 typically in range of 50 to 1000

• Similar words should have similar embeddings (share latent features)


The 𝑖𝑡ℎ word in the dictionary is represented by an embedding:𝑤𝑖 ∈ 𝑅𝑑

i.e., 𝑑-dimensional vector which is learnt!!

21/6/2016

Embedding of 115 Country Names(Bordes et al., ’11)


Deep Neural Network (DNN)

• Deep Learning has been used for learning high-dimensional continuous-valued vector for words

• Idea is: can we learn similar embedding for the semantic relation in noun compounds?

• “Fine tuning” of generic word embedding – constructed independently – has improved the performance (Collobert et al., 2011)• Fine tuning: task specific improvement of generic word embedding


DNN: Architecture (Dima and Hinrichs, ’15)


Results (TH10)

26Noun Compound Processing

Average Precision: 0.78Average Recall: 0.78Average F-score: 0.78

21/6/2016

“Nearest Neighbors” of a compound: 1/2robot arm (WHOLE+PART_OR_MEMBER_OF)

Using Component Similarity

• robot spider

• foot arm

• service arm

• mouse skull

• machine operator

• elephant leg

• car body

Using Compound Similarity

• dinosaur wing

• airplane wing

• mouse skull

• jet engine

• airplane instrument

• pant leg

• fighter wing


“Nearest Neighbors” of a compound: 2/2hillside home (LOCATION)

Using Component Similarity

• waterfront home

• brick home

• trailer home

• winter home

• boyhood home

• retirement home

• summer home

Using Compound Similarity

• waterfront home

• patio furniture

• fairway bunker

• beach house

• basement apartment

• ocean water

• beach resort


Analysis (zero f-score classes)


Representation at Hidden Layers


First Hidden Layer Second Hidden Layer Output Layer

21/6/2016

Our Approaches






The current lingering question

• Any sequence of nouns cannot be a noun compound

• For example:• Student protest

• Towel juice

• Towel namkeen

• So, question is: what is that thing that decides such coherence in noun compound?


References – 1/2

• Timothy Baldwin and Takaaki Tanaka. Translation by machine of complex nominals: Getting it right. In Proceedings of the Workshop on Multiword Expressions: Integrating Processing, pages 24—31. Association for Computational Linguistics, 2004.

• Renu Balyan and Niladri Chatterjee. Translating noun compounds using semantic relations. Computer Speech & Language, 2014.

• Ken Barker and Stan Szpakowicz. Semi-automatic recognition of noun modier relationships. In Proceedings of the 17th international conference on Computational linguistics-Volume 1, pages 96—102. Association for Computational Linguistics, 1998.

• Cristina Butnariu and Tony Veale. A concept-centered approach to noun-compound interpretation. In Proceedings of the 22nd International Conference on Computational Linguistics-Volume 1, pages 81—88. Association for Computational Linguistics, 2008.

• Peter Clark and Robin Boswell. Rule induction with cn2: Some recent improvements. In Machine learning—EWSL-91, pages 151—163, 1991.

• Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. Natural language processing (almost) from scratch. The Journal of Machine Learning Research, 12:2493{2537, 2011.

• Corina Dima and Erhard Hinrichs. Automatic noun compound interpretation using deep neural networks and word embeddings. IWCS 2015, page 173, 2015.


References – 2/2

• Pamela Downing. On the creation and use of English compound nouns. Language, pages 810—842, 1977.

• Su Nam Kim and Timothy Baldwin. Automatic interpretation of noun compounds using WordNet similarity. In Natural Language Processing—IJCNLP 2005, pages 945—956. Springer, 2005.

• Preslav Nakov. On the interpretation of noun compounds: Syntax, semantics, and entailment. Natural Language Engineering, 19(03):291—330, 2013.

• Parag Singla. Markov Logic: Theory, Algorithms and Applications. PhD thesis, University of Washington Graduate School, 2009.

• Stephen Tratz. Semantically-enriched parsing for natural language understanding. University of Southern California, 2011.

• Stephen Tratz and Eduard Hovy. A taxonomy, dataset, and classier for automatic noun compound interpretation. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 678—687. Association for Computational Linguistics, 2010.


Thank youContact: [email protected]


noun compound interpretation - iit bombay you have a dictionary of words ... automatic noun compound...

Documents