analysis of similarity measures between short text for the...

1

Analysis of Similarity Measures between Short Text for the NTCIR-12 Short Text Conversation Task Kozo Chikai and Yuki Arase Graduate school of information science and technology, Osaka University Introduction Goal Given an input post, search appropriate replies from the pool. Approach we compare the state-of-the-art methods for estimating text similarities to investigate their performance in handling short text, specially, under the scenario of short text conversation. System Design We implemented these five methods as Similarity Calculation. ① TF-IDF ② Weighted Text Matrix Factorization ③ Word2vec → TF-IDF ④ Random Forest → TF-IDF ⑤ Random Forest ＋ TF-IDF Results 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 ndcg-1 nerr-5 accuracy2-1 accuracy2-5 accuracy12-1 accuracy12-5 Mean of labels Evaluation Method Result of experiments Oni-J-R3 (①TF-IDF) Oni-J-R5 (②WTMF) Oni-J-R4 (③Word2Vec → TF-IDF) Oni-J-R1 (④RandomForest → TF-IDF) Oni-J-R2 (⑤RandomForest + TF-IDF) 0.3 0.5 0.7 0.9 rank1 rank2 rank3 rank4 rank5 Mean of labels Order Mean of labels for each rank Oni-J-R3 (①TF-IDF) Oni-J-R5 (②WTMF) Oni-J-R4 (③Word2Vec → TF-IDF) Oni-J-R1 (④RandomForest → TF-IDF) Oni-J-R2 (⑤RandomForest + TF-IDF)

Upload: others

Post on 18-Jan-2021

11 views

Category:

Documents

0 download

Report

Download

Embed Size (px):

TRANSCRIPT

Page 1: Analysis of Similarity Measures between Short Text for the ...research.nii.ac.jp/ntcir/workshop/OnlineProceedings12/...Analysis of Similarity Measures between Short Text for the NTCIR-12

AnalysisofSimilarityMeasuresbetweenShortTextforthe NTCIR-12ShortTextConversationTask

KozoChikai andYukiAraseGraduateschoolofinformationscienceandtechnology,OsakaUniversity

IntroductionGoalGivenaninputpost,searchappropriaterepliesfromthepool.

Approachwecomparethestate-of-the-artmethodsforestimatingtextsimilaritiestoinvestigatetheirperformanceinhandlingshorttext,specially,underthescenarioofshorttextconversation.

SystemDesignWeimplementedthesefivemethodsasSimilarityCalculation.

① TF-IDF② WeightedTextMatrixFactorization③ Word2vec→ TF-IDF④ RandomForest→ TF-IDF⑤ RandomForest＋ TF-IDF

Results

00.050.10.150.20.250.30.350.4

ndcg-1 nerr-5 accuracy2-1 accuracy2-5 accuracy12-1 accuracy12-5

Meanoflabels

EvaluationMethod

Resultofexperiments

Oni-J-R3(①TF-IDF) Oni-J-R5(②WTMF)Oni-J-R4(③Word2Vec→TF-IDF) Oni-J-R1(④RandomForest→TF-IDF)Oni-J-R2(⑤RandomForest+TF-IDF)

0.3

0.5

0.7

0.9

rank1 rank2 rank3 rank4 rank5

Meanoflabe

ls

Order

Meanoflabelsforeachrank

Oni-J-R3(①TF-IDF) Oni-J-R5(②WTMF)Oni-J-R4(③Word2Vec→TF-IDF) Oni-J-R1(④RandomForest→TF-IDF)Oni-J-R2(⑤RandomForest+TF-IDF)

Overview of the Seventh NTCIR Workshopresearch.nii.ac.jp/ntcir/workshop/OnlineProceedings7/pdf/... · 2010-06-29 · proceedings of the seventh NTCIR Workshop. Keywords: evaluation,

The MCAT Math Retrieval System for NTCIR-11 Math Trackresearch.nii.ac.jp/ntcir/workshop/OnlineProceedings11/pdf/NTCIR/M… · the same information as in opaths but with ordering infor-mation

NTCIR-12 MathIR Task Overviewresearch.nii.ac.jp/ntcir/workshop/OnlineProceedings12/...NTCIR-12 MathIR Task Overview Richard Zanibbi Rochester Institute of Technology [email protected]

Overview of the Second NTCIR Workshop · Overview of the Second NTCIR Workshop National Institute of Informatics (NII), Japan 1.1 Purpose 3. 1.2 Brief History 1.3 Focus of the NTCIR

Overview of the NTCIR-12 MobileClick-2 Taskresearch.nii.ac.jp/ntcir/workshop/OnlineProceedings12/pdf/ntcir/... · Overview of the NTCIR-12 MobileClick-2 Task Makoto P. Kato Kyoto

Call for Task Participation NTCIR-12 Pilot Task: Short ...research.nii.ac.jp/ntcir/ntcir-12/pdf/NTCIR-12-Kickoff-STC.pdf · NTCIR-12 Pilot Task: Short Text Conversation (STC) Lifeng

Analysis of Similarity Measures between Short Text …research.nii.ac.jp/ntcir/workshop/OnlineProceedings12/...Analysis of Similarity Measures between Short Text for the NTCIR-12 Short

Overview of the NTCIR-14 Lifelog-3 Taskresearch.nii.ac.jp/ntcir/workshop/OnlineProceedings14/...Overview of the NTCIR-14 Lifelog-3 Task 3 2.3 LIT SubTask The LIT subtask was exploratory

Overview of the NTCIR-7 ACLIA tasks - Machine · PDF fileProceedings of NTCIR-7 Workshop Meeting, December 16 19, 2008, Tokyo, Japan Overview of the NTCIR-7 ACLIA Tasks: Advanced Cross-Lingual

QA System Metis Based on Semantic Graph Matching at NTCIR 6research.nii.ac.jp/ntcir/workshop/OnlineProceedings6/... · 2010-07-28 · Proceedings of NTCIR-6 Workshop Meeting, May

What’s happening at NTCIR

A Baseline Interactive Retrieval Engine for the NTCIR-14 ...research.nii.ac.jp/.../ntcir/06-NTCIR14-LIFELOG-NinhV.pdfthe NTCIR-14 Lifelog task [2]. In this paper we introduce this

NTCIR Evaluation Activities: Recent Advances on RITE

Proceedings of the Third NTCIR Workshopresearch.nii.ac.jp/ntcir/workshop/OnlineProceedings3/NTCIR3-OV-WEB... · Overview of the Web Retrieval Task at the Third NTCIR Workshop Koji

Proceedings of the Third NTCIR Workshopresearch.nii.ac.jp/ntcir/workshop/Online... · Proceedings of the Third NTCIR Workshop 0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0,0 0,1

Overview of NTCIR

Overview of NTCIR-9 1CLICKresearch.nii.ac.jp/ntcir/workshop/OnlineProceedings9/NTCIR/01-NTCI… · The PMO is used for normalising our evaluation metric. URL is a “supporting document”

Short Text Conversation@NTCIR-12 Kickoff

IMTKU Question Answering System for World …research.nii.ac.jp/ntcir/workshop/OnlineProceedings12/...IMTKU Question Answering System for World History Exams at NTCIR-12 QA Lab2 Min-Yuh

Information Extraction based Approach for the NTCIR-9 ...research.nii.ac.jp/ntcir/workshop/Online... · Information Extraction based Approach for the NTCIR-9 1CLICK Task ABSTRACT

NTCIR-15 Conference

IMTKU Question Answering System for World History …research.nii.ac.jp/ntcir/workshop/OnlineProceedings12/...World History Exams at NTCIR-12 QA Lab2 [email protected] NTCIR-12

Overview of the NTCIR-12 QA Lab-2 Taskresearch.nii.ac.jp/ntcir/workshop/OnlineProceedings12/pdf/ntcir/... · 2.1 Topics Table 3 shows training set and test set in each phase. Each

It d it t NTCIRIntroduction to NTCIR-7research.nii.ac.jp/ntcir/workshop/Online...NTCIR: NII Test Collection for Information Retrieval Research Infrastructure for Evaluating IA A series

IMTKU Textual Entailment System Demoresearch.nii.ac.jp/ntcir/workshop/OnlineProceedings9/NTCIR/08... · IMTKU Textual Entailment System for Recognizing Inference in Text at NTCIR-9

BCMI NLP Labeled Alignment Based System for NTCIR 10 …research.nii.ac.jp/ntcir/workshop/Online...BCMI‐NLP Labeled‐Alignment‐Based Entailment System for NTCIR‐10 RITE‐2

Overview of the NTCIR-12 Short Text Conversation Taskresearch.nii.ac.jp/ntcir/workshop/OnlineProceedings12/...Overview of the NTCIR-12 Short Text Conversation Task Lifeng Shang Noah’s

HITS’ Graph-based System at the NTCIR-9 Cross …research.nii.ac.jp/.../NTCIR/04-NTCIR9-CROSSLINK-FahrniA.pdfHITS’ Graph-based System at the NTCIR-9 Cross-lingual Link Discovery

Overview of the NTCIR-7 ACLIA Tasks: Advanced Cross ...research.nii.ac.jp/ntcir/workshop/OnlineProceedings7/pdf/...Proceedings of NTCIR-7 Workshop Meeting, December 16 19, 2008, Tokyo,

Overview of the NTCIR-9 INTENT Taskresearch.nii.ac.jp/ntcir/workshop/OnlineProceedings9/...Overview of the NTCIR-9 INTENT Task Ruihua Song† Min Zhang‡ Tetsuya Sakai† Makoto P

KitAi-PI: Summarization System for NTCIR-14 QA Lab-PoliInforesearch.nii.ac.jp/ntcir/workshop/Online... · This paper describes a summarization system for NTCIR-14 QA Lab-PoliInfo

Overview of CLIR Task at the Third NTCIR Workshopresearch.nii.ac.jp/ntcir/workshop/Online... · Overview of CLIR Task at the Third NTCIR Workshop Kuang-hua Chen*, Hsin-Hsi Chen+,

Overview of the Sixth NTCIR Workshopresearch.nii.ac.jp/ntcir/ntcir-ws6/OnlineProceedings/NTCIR/... · ntcir6 2006-05-16 Noriko kando 2 NTCIR Workshop is : A series of evaluation workshops

Overview of the Second NTCIR Workshopresearch.nii.ac.jp/ntcir/workshop/OnlineProceedings2/ovview-kando.pdfOverview of the Second NTCIR Workshop National Institute of Informatics (NII),

Decision Tree Method for the NTCIR-13 ECA Taskresearch.nii.ac.jp/ntcir/workshop/OnlineProceedings13/pdf/ntcir/03-NTCIR13-ECA-LiX.pdfDecision Tree Method for the NTCIR-13 ECA Task Xiangju