structural normalisation methods for improving best answer identification in question answering...
TRANSCRIPT
GRÉGOIRE BUREL, PAUL MULLHOLAND, HARITH ALANI Knowledge Media Institute, The Open University, Milton Keynes, UK. WWW’16, Montréal, Canada. 11th April 2016
Structural Normalisation Methods for Improving Best Answer Identification in Question Answering Communities
Outline�
Structural Normalisation Methods for Improving Best Answer Identification in Question Answering Communities
Publications
2
Identifying Best Answers in Q&A Communities - Question Answering Communities - Automatic Best Answer Identification - Reference Model
Structural Normalisation - No Normalisation Vs. Structural Normalisation - Normalisation Methods - Normalisation Method Selection
Best Answer Identification with Structural Normalisation - Models Comparison - Features Comparison
Conclusions and Future Work
Q&A Communities�
Structural Normalisation Methods for Improving Best Answer Identification in Question Answering Communities
Publications
Question
Answer #1
Answer #2
...
Answer #n
Que
stio
n Th
read
3
Q&A communities are communities composed of askers and answerers looking for solutions to particular issues. When looking for answers, users need to identify if similar questions have already been answered correctly and see if a best answer exists. Unfortunately not all questions have labelled best answers (43.2% of questions do not have labelled best answers). Existing works that have focused on best answer identification have also generally ignored the structure of Q&A websites while designing feature normalisation methods.
?
!
-
Q&A Communities�
Structural Normalisation Methods for Improving Best Answer Identification in Question Answering Communities
Publications
Question
Answer #1
Answer #2
...
Answer #n
Que
stio
n Th
read
4
Q&A communities are communities composed of askers and answerers looking for solutions to particular issues. When looking for answers, users need to identify if similar questions have already been answered correctly and see if a best answer exists. Unfortunately not all questions have labelled best answers (43.2% of questions do not have labelled best answers). Existing works that have focused on best answer identification have also generally ignored the structure of Q&A websites while designing feature normalisation methods.
?
!
-
Q&A communities are communities composed of askers and answerers looking for solutions to particular issues. When looking for answers, users need to identify if similar questions have already been answered correctly and see if a best answer exists. Unfortunately not all questions have labelled best answers (43.2% of questions do not have labelled best answers). Existing works that have focused on best answer identification have also generally ignored the structure of Q&A websites for designing feature normalisation methods.
Q&A Communities�
Structural Normalisation Methods for Improving Best Answer Identification in Question Answering Communities
Publications
Question
Answer #1
Answer #2
...
Answer #n
Que
stio
n Th
read
?
!
-
5
Feature-based Best Answers Identification and Reference Model
Structural Normalisation Methods for Improving Best Answer Identification in Question Answering Communities
6
In order to identify best answers features are extracted and associated with each answers and a binary classifier is trained (Burel et al. 2012 , 2013, 2014): - Such features are divided into User, Content
and Thread features. - 30 features are used for the reference model. - The Alternating Decision Tree algorithm is
used.
Reference Model Results: - The reference model achieve an F1 of 0.817
on average. - Thread features in particular score ratios are
highly related to best answers.
What methods could be used for improving such model? ?
Feature-based Best Answers Identification and Reference Model
Structural Normalisation Methods for Improving Best Answer Identification in Question Answering Communities
7
In order to identify best answers features are extracted and associated with each answers and a binary classifier is trained (Burel et al. 2012 , 2013, 2014): - Such features are divided into User, Content and
Thread features. - 30 features are used for the reference model. - The Alternating Decision Tree algorithm is used.
Reference Model Results: - The reference model achieve an F1 of 0.817 on
average. - Thread features in particular score ratios are
highly related to best answers.
What methods could be used for improving such model? ?
Contributions
Structural Normalisation Methods for Improving Best Answer Identification in Question Answering Communities
8
Does the thread-like structure of Q&A communities can help the automatic identification of best answers ? - Introduce the concept of structural normalisation, a method for
normalising features based on the structure of Q&A communities. - Present 3 different features normalisation methods: 1) Min/Max
normalisation; 2) Order Normalisation, and; 3) Normalised Order Normalisation.
- Identify the best normalisation method for automatically identifying best answers.
- Compare different models and features sets and show that structural normalisation improves best answer identification.
?
Structural Normalisation / No Normalisation �
Structural Normalisation Methods for Improving Best Answer Identification in Question Answering Communities
9
Question
Answer #1
Answer #2
...
Answer #n
Que
stio
n Th
read
Question
Answer #1
Answer #2
...
Answer #nQ
uest
ion
Thre
ad
Question
Answer #1
Answer #2
...
Answer #nQ
uest
ion
Thre
ad
Question
Answer #1
Answer #2
...
Answer #nQ
uest
ion
Thre
ad
Question
Answer #1
Answer #2
...
Answer #nQ
uest
ion
Thre
ad
Standard classifiers such as the reference model compare all the answers even if they are not part of the same thread. Thread-wise normalisation normalise features based on their relation within a thread.
!
Structural Normalisation / No Normalisation �
Structural Normalisation Methods for Improving Best Answer Identification in Question Answering Communities
10
Question
Answer #1
Answer #2
...
Answer #n
Que
stio
n Th
read
Question
Answer #1
Answer #2
...
Answer #nQ
uest
ion
Thre
ad
Question
Answer #1
Answer #2
...
Answer #nQ
uest
ion
Thre
ad
Question
Answer #1
Answer #2
...
Answer #nQ
uest
ion
Thre
ad
Question
Answer #1
Answer #2
...
Answer #nQ
uest
ion
Thre
ad
Standard classifiers such as the reference model compare all the answers even if they are not part of the same thread. Thread-wise normalisation normalise features based on their relation within a thread.
!
Normalisation Methods�
Structural Normalisation Methods for Improving Best Answer Identification in Question Answering Communities
Min/Max Normalisation (Thread-wise min/max) Normalise features within a thread using the minimum and maximum value of a particular feature.
1 2
Normalised Order Normalisation Same as Order Normalisation but the values are normalised between 0 and 1.
3
11
Order Normalisation Generalises Gkotsis et al. (2014) approach. Features within a thread are ranked according to their value. The lowest value is transformed to the length of the thread while the highest value is transformed to 1.
Normalisation Methods�
Structural Normalisation Methods for Improving Best Answer Identification in Question Answering Communities
Min/Max Normalisation (Thread-wise min/max) Normalise features within a thread using the minimum and maximum value of a particular feature.
1
Order Normalisation Generalises Gkotsis et al. (2014) approach. Features within a thread are ranked according to their value. The lowest value is transformed to the length of the thread while the highest value is transformed to 1.
2
Normalised Order Normalisation Same as Order Normalisation but the values are normalised between 0 and 1.
3
12
Adaptive Feature Normalisation Not all features need to be normalised. For instance, number of question views does not change within a question thread but does across questions. If for a given feature the variance is zero and remains constant between all the threads, the feature is not normalised. Otherwise, the feature is normalised with one of the previous functions.
!
Normalisation Method Selection
Structural Normalisation Methods for Improving Best Answer Identification in Question Answering Communities
13
? What normalisation method is the most useful ?
Approach 1. Calculate the average Information Gain (IG) of each feature associated with each
normalisation method. 2. Select the normalisation method that has the highest average IG across all the datasets.
Normalisation Method Selection
Structural Normalisation Methods for Improving Best Answer Identification in Question Answering Communities
14
? What normalisation method is the most useful ?
Approach 1. Calculate the average Information Gain (IG) of each feature associated with each
normalisation method. 2. Select the normalisation method that has the highest average IG across all the datasets.
Best Answers Identification Models Comparisons
Structural Normalisation Methods for Improving Best Answer Identification in Question Answering Communities
15
Best Answers Identification Models Comparisons
Structural Normalisation Methods for Improving Best Answer Identification in Question Answering Communities
16
General Results - Thread normalisation approaches improves best answer identification (p =
2.817e − 05). - On average, an increase in F1 of +5.3% is observed compared to the non
normalised models.
Baseline Models Results - In general, thread normalisation approaches improves best answer
identification: On average F1 = 0.718 (+26.8%) . - The longuest answers within a thread is more likely to be associated with a
best answers.
Core Features Models Results - Unsurprisingly, there is no increase in F1 for thread features but there is a
significant increase for other features sets (+20% F1). - Content features are more important within a thread whereas user features
are better for discriminating best answers globally.
Extended Features Models Results - When score are present, normalisation does not improve as score ratio
already perform very well.
8
0
1
Best Answers Identification Models Comparisons
Structural Normalisation Methods for Improving Best Answer Identification in Question Answering Communities
17
General Results - Thread normalisation approaches improves best answer identification (p =
2.817e − 05). - On average, an increase in F1 of +5.3% is observed compared to the non
normalised models.
Baseline Models Results - In general, thread normalisation approaches improves best answer
identification: On average F1 = 0.718 (+26.8%) . - The longuest answers within a thread is more likely to be associated with a
best answers.
Core Features Models Results - Unsurprisingly, there is no increase in F1 for thread features but there is a
significant increase for other features sets (+20% F1). - Content features are more important within a thread whereas user features
are better for discriminating best answers globally.
Extended Features Models Results - When score are present, normalisation does not improve as score ratio
already perform very well.
8
0
1
Best Answers Identification Models Comparisons
Structural Normalisation Methods for Improving Best Answer Identification in Question Answering Communities
18
General Results - Thread normalisation approaches improves best answer identification (p =
2.817e − 05). - On average, an increase in F1 of +5.3% is observed compared to the non
normalised models.
Baseline Models Results - In general, thread normalisation approaches improves best answer
identification: On average F1 = 0.718 (+26.8%) . - The longuest answers within a thread is more likely to be associated with a
best answers.
Core Features Models Results - Unsurprisingly, there is no increase in F1 for thread features but there is a
significant increase for other features sets (+20% F1). - Content features are more important within a thread whereas user features
are better for discriminating best answers globally.
Extended Features Models Results - When score are present, normalisation does not improve as score ratio
already perform very well.
8
0
1
Best Answers Identification Models Comparisons
Structural Normalisation Methods for Improving Best Answer Identification in Question Answering Communities
19
General Results - Thread normalisation approaches improves best answer identification (p =
2.817e − 05). - On average, an increase in F1 of +5.3% is observed compared to the non
normalised models.
Baseline Models Results - In general, thread normalisation approaches improves best answer
identification: On average F1 = 0.718 (+26.8%) . - The longuest answers within a thread is more likely to be associated with a
best answers.
Core Features Models Results - Unsurprisingly, there is no increase in F1 for thread features but there is a
significant increase for other features sets (+20% F1). - Content features are more important within a thread whereas user features
are better for discriminating best answers globally.
Extended Features Models Results - When score are present, normalisation does not improve as score ratio
already performs very well.
8
0
1
Features Importance
Structural Normalisation Methods for Improving Best Answer Identification in Question Answering Communities
20
?
Approach 1. Calculate the Information Gain (IG) of each feature for each dataset with and without
normalisation. 2. Order the features by IG across all the datasets. The higher the IG the more important the
feature.
What features are the most useful ? How does order normalisation impact feature importance?
Features Importance- Results
Structural Normalisation Methods for Improving Best Answer Identification in Question Answering Communities
21
Features Importance - Results
Structural Normalisation Methods for Improving Best Answer Identification in Question Answering Communities
22
General Results - As previously observed, scores based features are the most important best answer
predictors. - Similarly, user features are more important than content features when features are not
normalised. When features are normalised the difference diminishes.
Features Comparison - Number of comments is associated with best answers when normalised. Therefore, the
relative amount of comments may be a good indicator of best answers. - The relative number of answer for a user is an important indicator for identifying best
answers (i.e. good answers comes from users that post a lot of answers) - Answers with relatively original content are more likely to be linked with best answers. - Relative topic reputation is not really important meaning that this feature only helps when
distinguishing best answers in a global context.
Conclusions and Future Work
Structural Normalisation Methods for Improving Best Answer Identification in Question Answering Communities
Conclusions - Compared to the reference models,
structural approaches improve automatic best answer identification by around +5% F1.
- Thread-normalisation change the importance of features (e.g. content length)
- Score features are overwhelming important.
- Structural methods may be used successfully for other classification tasks where analysed communities are highly structured.
23
1
Conclusions and Future Work
Structural Normalisation Methods for Improving Best Answer Identification in Question Answering Communities
Conclusions - Compared to the baseline models,
structural approaches improve automatic best answer identification by around +5% F1.
- Thread-normalisation change the importance of features (e.g. content length)
- Score features are overwhelming important.
- Structural methods may be used successfully for other classification tasks where analysed communities are highly structured.
24
1 2
Future/Current Work - Evaluate other types of normalisation (e.g.
signed feature ratios) - Consider other structural optimisation
methods (e.g. Learning-to-Rank models) - Consider applicability to other type of
communities
Questions and Discussion
@ Email: [email protected] Twitter: @evhart
Structural Normalisation Methods for Improving Best Answer Identification in Question Answering Communities
25