Upload File

uncertainty in neural network word embedding: exploration of threshold for similarity

  • Home
  • Science
  • Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity
45
Navid Rekabsaz ([email protected]) Mihai Lupu ([email protected]) Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity NeuIR at SIGIR, 21st July 2016 Navid Rekabsaz, Mihai Lupu, Allan Hanbury @NRekabsaz [email protected]

Upload: navid-rekabsaz

Post on 14-Apr-2017

253 views

Category:

Science


1 download

Report

TRANSCRIPT

Page 1: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

Navid Rekabsaz ([email protected]) Mihai Lupu ([email protected])

Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

NeuIR at SIGIR, 21st July 2016

Navid Rekabsaz, Mihai Lupu, Allan Hanbury

@NRekabsaz [email protected]

Page 2: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

Similar Terms

Page 3: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

Similar Terms

books 0.82foreword 0.77author 0.74published 0.73preface 0.69republished 0.68reprinted 0.68afterword 0.67memoir 0.67

bookcorpulent 0.44hideous 0.43unintelligent

0.42wizened 0.42catoblepas 0.42creature 0.42humanoid 0.41grotesquely 0.41tomtar 0.41

dwarfish

Page 4: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

TopN:

Similar Terms

books 0.82foreword 0.77author 0.74published 0.73preface 0.69republished 0.68reprinted 0.68afterword 0.67memoir 0.67

bookcorpulent 0.44hideous 0.43unintelligent

0.42wizened 0.42catoblepas 0.42creature 0.42humanoid 0.41grotesquely 0.41tomtar 0.41

dwarfish

Page 5: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

TopN:

Similar Terms

Using threshold in SIGIR 2016, tuned by brute-force search: Scalable Semantic Matching of Queries to Ads in Sponsored Search Advertising.Mihajlo Grbovic et al.Robust and Collective Entity Disambiguation through Semantic Embeddings.Stefan Zwicklbauer et al.

books 0.82foreword 0.77author 0.74published 0.73preface 0.69republished 0.68reprinted 0.68afterword 0.67memoir 0.67

bookcorpulent 0.44hideous 0.43unintelligent

0.42wizened 0.42catoblepas 0.42creature 0.42humanoid 0.41grotesquely 0.41tomtar 0.41

dwarfish

Page 6: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

Contribution

Page 7: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

Our Contribution • Analytical exploration of a general threshold on term similarity

• Defined for all the terms in the lexicon • Tested on Ad Hoc retrieval

• Showing the advantage of threshold-based rather than TopN-based approach

Contribution

Page 8: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

Roadmap • Uncertainty in NN-based word embeddings • Similarity distribution • Proposing threshold • Experiments

Our Contribution • Analytical exploration of a general threshold on term similarity

• Defined for all the terms in the lexicon • Tested on Ad Hoc retrieval

• Showing the advantage of threshold-based rather than TopN-based approach

Contribution

Page 9: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

Observations

Page 10: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

Observations

W2V

W2V: Word2Vec SkipGram with 200 Dimensions

Page 11: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

Observations

W2V

BookDwarfish

W2V: Word2Vec SkipGram with 200 Dimensions

Page 12: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

Observations

W2V

BookDwarfish

W2V: Word2Vec SkipGram with 200 Dimensions

Page 13: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

Observations

W2V

W2V

BookDwarfish

W2V: Word2Vec SkipGram with 200 Dimensions

Page 14: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

Observations

W2V

W2V

BookDwarfish

BookDwarfish

W2V: Word2Vec SkipGram with 200 Dimensions

Page 15: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

Observations

W2V

W2V

BookDwarfish

BookDwarfish

W2V: Word2Vec SkipGram with 200 Dimensions

Page 16: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

Observations

W2V

W2V

BookDwarfish

BookDwarfish X 4

W2V: Word2Vec SkipGram with 200 Dimensions

Page 17: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

Observations

Similarity values: Base

W2V

W2V

BookDwarfish

Average of similarity values of 5 models:

Avg

BookDwarfish X 4

W2V: Word2Vec SkipGram with 200 Dimensions

Page 18: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

Uncertainty

Page 19: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

Uncertainty

Uncertainty:

Page 20: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

Uncertainty• Arbitrary term: Average of 100 representative terms [Schnabel et al. 2015]

Page 21: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

Uncertainty• Arbitrary term: Average of 100 representative terms [Schnabel et al. 2015]

• Less uncertainty in higher similarity values • Less uncertainty in higher dimensions

Page 22: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

Similarity Probability Distribution• Similarity between terms as probability distribution instead

of concrete values • Assumption: normal distribution based on 5 observed

similarities

Page 23: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

Mixture of Cumulative Similarity Distributions

• Y axes: number of neighbors, located in the space between the X value and the term

Page 24: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

Mixture of Cumulative Similarity Distributions

• Y axes: number of neighbors, located in the space between the X value and the term

Page 25: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

Mixture of Cumulative Similarity Distributions

• Y axes: number of neighbors, located in the space between the X value and the term

Page 26: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

Mixture of Cumulative Similarity Distributions

• Y axes: number of neighbors, located in the space between the X value and the term

Page 27: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

Mixture of Cumulative Similarity Distributions

• Y axes: number of neighbors, located in the space between the X value and the term

Page 28: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

Filtering NeighborsWhat is the best threshold for filtering the related terms?

Page 29: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

Filtering NeighborsWhat is the best threshold for filtering the related terms?

Hypothesis: it can be estimated based on the average number of synonyms over the terms

Page 30: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

Filtering NeighborsWhat is the best threshold for filtering the related terms?

Hypothesis: it can be estimated based on the average number of synonyms over the terms

What is the expected number of synonyms for a word in English?

Page 31: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

Filtering NeighborsWhat is the best threshold for filtering the related terms?

Hypothesis: it can be estimated based on the average number of synonyms over the terms

What is the expected number of synonyms for a word in English?

# of terms: Average # of synonyms per term:

Standard deviation :

147306 1.6 3.1

Page 32: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

Threshold• Proposed Threshold: cumulative frequency equal to 1.6

Page 33: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

Threshold• Proposed Threshold: cumulative frequency equal to 1.6

Page 34: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

Threshold• Proposed Threshold: cumulative frequency equal to 1.6

Page 35: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

Threshold• Proposed Threshold: cumulative frequency equal to 1.6

Page 36: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

Experiments Setup• Translation Language Model with Dirichlet smoothing • Use word embedding similarity value for translation probability

Zuccon et al. [2015] • Apply the proposed threshold to select the similar words • Brute-force search to find the optimal threshold

Page 37: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

Experiments Setup• Translation Language Model with Dirichlet smoothing • Use word embedding similarity value for translation probability

Zuccon et al. [2015] • Apply the proposed threshold to select the similar words • Brute-force search to find the optimal threshold

• Test collections:

Page 38: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

Experiments Setup• Translation Language Model with Dirichlet smoothing • Use word embedding similarity value for translation probability

Zuccon et al. [2015] • Apply the proposed threshold to select the similar words • Brute-force search to find the optimal threshold

• Test collections:

• Baseline: language model (Dirichlet smoothing) • Significance Test : T-Test p<0.05

Page 39: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

Experiments Results• Gain of MAP over baseline, averaged on four collections. • Conclusion1: Optimal threshold is either the same or in the

confidence interval of the proposed threshold.

Page 40: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

Experiments Results• Gain of MAP over baseline, averaged on four collections. • Conclusion1: Optimal threshold is either the same or in the

confidence interval of the proposed threshold.

Page 41: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

Threshold vs. TopN• Conclusion2: Threshold outperforms TopN

Threshold-based TopN

Page 42: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

Take Home Message

• Uncertainty in neural network word embeddings: • depends on similarity value • depends on dimensionality

• Threshold to filter unrelated terms : • Proposed threshold as good as optimal threshold • Threshold approach much better than Top-N approach

Page 43: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

@NRekabsaz [email protected]

Questions?

Ideas!

Follow-up paper in CIKM 2016: Generalizing Translation Models in the Probabilistic Relevance Framework Navid Rekabsaz, Mihai Lupu, Allan Hanbury

Page 44: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

Dimensionality

Page 45: Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

Proposed vs. Optimal Threshold


Improving Phrase-Based SMT Using Cross-Granularity ... · Improving Phrase-Based SMT Using Cross-Granularity Embedding Similarity Peyman PASSBAN, Chris HOKAMP, Andy WAY, Qun LIU ADAPT

Keysight Technologies De-Embedding and Embedding …literature.cdn.keysight.com/litweb/pdf/5980-2784EN.pdf · Keysight Technologies De-Embedding and Embedding S ... Advanced Design

Embedding Environmental Sustainability - Family Repositoryfamilyrepository.lppkn.gov.my/601/1/Embedding Environmental Sustainability.pdf · Embedding Environmental Sustainability

Elastiq: Answering Similarity-threshold Instance Queries in EL · Elastiq: Answering Similarity-threshold Instance Queries in EL Andreas Ecke 1?, Maximilian Pensel??, and Anni-Yasmin

Computational approaches to plagiarism detection and … · 2015. 10. 15. · tic plagiarism, allows researchers to calculate the threshold level of textual similarity which becomes

Scalable Ordinal Embedding to Model Text Similarity · Scalable Ordinal Embedding to Model Text Similarity PhD Thesis Proposal Jesse Anderton March 21, 2017 Abstract Practitioners

Subgraph-augmented Path Embedding for Semantic User Search ...shichuan.org/hin/topic/Similarity Measure/2018.WWW... · neous social network and a semantic relation (e.g., schoolmates),

LSH: LOCALITY SENSITIVE HASHING - uniroma1.ittwiki.di.uniroma1.it/pub/BDC/Schedule/lecture13-localitySensitive.pdfCandidates from min-hash signatures ! Pick a similarity threshold

3 EMAN2 Intro - blake.bcm.edu · testimage testimage.scurve threshold threshold.binary threshold.clampminmax xform xform.centerofmass xform.fourierorigin.tocenter Similarity Metrics

CS230 Deep Learning · Similarity and tf-idf are insufficient for questions but are more effective for bags of words approaches. Word2vec[16] is a word embedding which captures similarity

Lexical and Statistical Semantics in Professional Search · Similarity based on Uncertainty in Word Embedding. In Proceedings of the European Conference on Information Retrieval Research

Chapter 3 Learning embeddings that reflect similaritygregory/thesis/thesisChapter3.pdf · (3.3) 3.1.1 Threshold evaluation procedure The embedding algorithms in this chapter differ

Approximate Similarity Search in Genomic Sequence Databases using Landmark-Guided Embedding

Translation Inference through Multi-lingual Word Embedding ...ceur-ws.org/Vol-2493/system3.pdfTranslation Inference through Multi-lingual Word Embedding Similarity 3 Despite its simplicity,

5A-Sequence comparison-Significance of similarity …elbo.gs.washington.edu/courses/GS_373_12_sp/slides/6A...• Selecting a significance threshold requires evaluating the cost of

Convolutional Embedding for Edit Distance · 2020-05-25 · Edit-distance-based string similarity search has many applications such as spell correction, data de-duplication, and sequence

Embedding and Similarity Search for Point Sets under Translation

Visualization in AIg.legrady/academic/... · tSNE - Multi-dimensional Data Visualization tool t-Stochastic Neighbour Embedding, or t-SNE Visualizing data similarity of high-dimensional

Face Hallucination Based on Neighbor Embedding via ... Files/2015/ssj_apsipa15.pdf · struct the testing face via neighbor embedding, which takes the nonlocal similarity into account

METODE EMBEDDING LSTM BERBASIS SIMILARITY CLUSTER LDA

TST: Threshold Based Similarity Transitivity Method in …security.riit.tsinghua.edu.cn/share/XieFeng-TST.pdf · TST firstly filters out those inaccurate similarities by setting

Disentangling synchrony from serial dependency in complex ...€¦ · Data-adaptive significance testing instead of global threshold to similarity measure (also proposed for correlation-based

A review of word embedding and document similarity ...ad-publications.informatik.uni-freiburg.de/theses/... · Word embeddings, which facilitate working with abstract semantic concepts

STRATEGIES FOR AUTOMATIC DETERMINATION OF SIMILARITY ... · STRATEGIES FOR AUTOMATIC DETERMINATION OF SIMILARITY THRESHOLD FOR GENRE-AWARE FOCUSED CRAWLING PROCESSES Gustavo Oliveira

USE OF MORPHOLOGY IN DISTRIBUTIONAL WORD EMBEDDING … · approaches on the newly created rare and multimorphemic word similarity dataset, which itself is another contribution of

MDS Embedding - math.tau.ac.ildcor/Graphics/cg-slides/sgp-RMDS.pdf · 30. Precision and Recall. 31. Threshold Performance The outlier detection rate as a function of the shrinkage

Multiple Visual-Semantic Embedding for Video Retrieval ... · guitar Similarity Aggregation s N Final Similarity Global Visual Network Textual Embedding Network Sequential Visual

Embedding parameters in ab initio theory to develop well ... · 1 Embedding parameters in ab initio theory to develop well-controlled approximations based on molecular similarity

To determine the effect changing threshold values has on SISCOM image data sets. Threshold changes used did not increase the similarity of the 2-detector

An Efficient Partition Based Method for Exact Set Similarity Joins › pvldb › vol9 › p360-deng.pdf · Given a similarity threshold , two records Xand Yare said to be similar

ClusteringSNN Clustering Algorithm … 3. Construct the shared nearest neighbor graph from the sparsified similarity matrix At this point, we could apply a similarity threshold and

Visualization in AI · tSNE - Multi-dimensional Data Visualization tool t-Stochastic Neighbour Embedding, or t-SNE Visualizing data similarity of high-dimensional data by maintaining

NormFace: L2 Hypersphere Embedding for Face Verification · ing training. But in the testing phase, all methods used a normalized similarity, e.g. cosine, to compare two features

Spherical Text Embedding · 2020-03-11 · Word similarity: Train 100-d word embedding on the latest Wikipedia dump (~13G) Compute embedding cosine similarity between word pairs to

Distance Threshold Similarity Searches: Efficient Trajectory Indexing …jan.ucc.nau.edu/mg2745/publications/Gowanlock_Casanova_TPDS2015.pdf · 1 Distance Threshold Similarity Searches: