evaluating citation functions in cito: cognitive issues

STLab University of Bologna#eswc2014Ciancarini

!

Evaluating citation functions in CiTO: cognitive issues

!

!

!!!

28 May 2014 - Heraklion, Crete

Paolo Ciancarini1,2, Angelo Di Iorio1, Andrea Giovanni Nuzzolese1,2, Silvio Peroni1 and Fabio Vitali1

1Department of Computer Science and Engineering, University of Bologna, Italy 2STLab, Institute of Cognitive Science and Technology, National Research Council, Rome, Italy

STLab University of Bologna#eswc2014Ciancarini

Outline

2

• Motivations

• CiTO

• Experiment

• Evaluation

• Lessons learnt and conclusions

STLab University of Bologna#eswc2014Ciancarini

• Bibliographic citations can be seen as tools for:

• linking, disseminating, exploring, evaluating research

• The task of characterising the functional role of citations in scientific literature is very difficult for agents and humans

• Investigating how an existing reference model for classifying citations, i.e., CiTO, is interpreted and used by human annotators

• We want to study human’s behaviour in order to simulate it within CiTalO, a tool that automatically classifies citations with CiTO

Motivations and goals

3

STLab University of Bologna#eswc2014Ciancarini

• An OWL ontology for describing factual as well as rhetorical functions of citations in scholarly articles

• Defines

• a top-level property cites (and its inverse isCitedBy)

• 41 sub-properties of cites that allow users to characterise precisely the semantics of a citation act

• Has been successfully used in large projects, like CiteULike, data.open.ac.uk and the Open Citation Corpus

• Several tools have been developed to annotate citations with CiTO

• e.g., Chrome and WordPress plug-ins

CiTO

4

STLab University of Bologna#eswc2014Ciancarini

• The richness of properties in CiTO (CiTO-Ps) is

• a key feature of CiTO: this aspect has contributed to the adoption of the ontology by the Semantic Publishing

Users’ adoption of CiTO

5

STLab University of Bologna#eswc2014Ciancarini

• The richness of properties in CiTO (CiTO-Ps) is

• a key feature of CiTO: this aspect has contributed to the adoption of the ontology by the Semantic Publishing

• an hindrance: most tools actually employ a sub-set of the CiTO properties,

• e.g., 6 CiTO-Ps enabled for user annotation by Pensoft Publishers and 9 in the Chrome plug-in

Users’ adoption of CiTO

5

STLab University of Bologna#eswc2014Ciancarini6

user

author

“It extends the research outlined in earlier work [3]”

CiTO annotations and mental models

CiTO

STLab University of Bologna#eswc2014Ciancarini6

Interpretation of author’s

text

Understanding of CiTO

user

author

“It extends the research outlined in earlier work [3]”

CiTO annotations and mental models

CiTO

STLab University of Bologna#eswc2014Ciancarini6

Interpretation of author’s

text

Understanding of CiTO

user

mental model

author

“It extends the research outlined in earlier work [3]”

CiTO annotations and mental models

CiTO

STLab University of Bologna#eswc2014Ciancarini6

Interpretation of author’s

text

Understanding of CiTO

user

mental model

Annotation

cito:extends

author

“It extends the research outlined in earlier work [3]”

CiTO annotations and mental models

CiTO

STLab University of Bologna#eswc2014Ciancarini

mental model

mental model

6

Interpretation of author’s

text

Understanding of CiTO

user

mental model

Annotation

cito:extends

author

“It extends the research outlined in earlier work [3]”

mental modelmental

model

CiTO annotations and mental models

CiTO

STLab University of Bologna#eswc2014Ciancarini

mental model

mental model

6

Interpretation of author’s

text

Understanding of CiTO

user

mental model

Annotation

cito:extends

author

“It extends the research outlined in earlier work [3]”

mental modelmental

model

CiTO annotations and mental models

CiTO!cito:citesForInformation

cito:givesSupportTo …

The mental models of different annotators

hardly ever converge to a single shared opinion

STLab University of Bologna#eswc2014Ciancarini7

What we did

We performed an experiment to investigate how humans use CiTO to annotate citations with a type

• 20

• 105 citations chosen among the seventh volume of the proceedings of the Balisage Conference Series

subjects

STLab University of Bologna#eswc2014Ciancarini7

What we did

We performed an experiment to investigate how humans use CiTO to annotate citations with a type

• 20

• 105 citations chosen among the seventh volume of the proceedings of the Balisage Conference Series

(lucky) subjects

STLab University of Bologna#eswc2014Ciancarini

• The experiment had one independent variable, i.e., the number of CiTO-Ps available to subjects for the annotation

• Condition T41: 10 subjects used 41 CiTO-Ps

• Condition T10: 10 subjects used a subset of 10 CiTO-Ps

• i.e., citesAsDataSource, citesAsPotentialSolution, citesAsRecommendedReading, citesAsRelated, citesForInformation, credits, critiques, includesQuotationFrom, obtainsBackgroundFrom, usesMethodIn

• T10 CiTO-Ps were chosen among those that had shown a moderate inter-rater agreement (Fleiss’k>0.33) in a preliminary experiment on the same data sample

The experiment

8

STLab University of Bologna#eswc2014Ciancarini

!

Evaluation framework

9

Citation context CITO-Ps

Explanations and examples of the usage of the CiTO-Ps

Available on line at http://www.cs.unibo.it/~nuzzoles/cito_1/?user=r

STLab University of Bologna#eswc2014Ciancarini10

1. Which properties have been used by subjects during the experiment?

2. Which were the most used properties?

3. What was the global inter-rater agreement of the subjects?

4. Did the number of available choices bias the global inter-rater agreement?

5. Which properties showed an acceptable positive agreement among subjects?

6. Could properties be organized according to their similarity in subjects’ annotations?

7. What was the perceived usability of the CiTO-Ps?

8. Which were the features of CiTO-Ps that subjects perceived as most useful or problematic?

Target questions

STLab University of Bologna#eswc2014Ciancarini11

!

!

• Condition T41

• used 37 different CiTO-Ps over 41(avg: 21.7 CiTO-Ps per subject)

• 4 properties not selected by any subject

• i.e., parodies, plagiarizes, repliesTo and ridicules

!

• Condition T10

• used all the 10 CiTO-Ps

Results“Which properties have been used by subjects during the experiment?”

STLab University of Bologna#eswc2014Ciancarini12

Results“Which were the most used properties?”

STLab University of Bologna#eswc2014Ciancarini

“What was the global inter-rater agreement of the subjects?” “Did the number of available choices bias the global inter-rater agreement?”

“Which properties showed an acceptable positive agreement among subjects?”

13

• Condition T41

• Global Fleiss’k = 0.13 • 5 CiTO-Ps with moderate local positive agreement (k > 0.5)

• i.e., citesAsPotentialSolution (0.66), citesAsRecommendedReading (0.6), agreesWith (0.54), citesAsDataSource (0.52), usesMethodIn (0.54)

• Condition T10

• Global Fleiss’k = 0.15 • 4 CiTO-Ps with moderate local positive agreement

• i.e., citesAsPotentialSolution (0.71), citesAsDataSource (0.63), citesAsRecommendedReading (0.52), includesQuotationFrom (0.69)

Data evaluation

STLab University of Bologna#eswc2014Ciancarini

“What was the global inter-rater agreement of the subjects?” “Did the number of available choices bias the global inter-rater agreement?”

“Which properties showed an acceptable positive agreement among subjects?”

13

• Condition T41

• Global Fleiss’k = 0.13 • 5 CiTO-Ps with moderate local positive agreement (k > 0.5)

• i.e., citesAsPotentialSolution (0.66), citesAsRecommendedReading (0.6), agreesWith (0.54), citesAsDataSource (0.52), usesMethodIn (0.54)

• Condition T10

• Global Fleiss’k = 0.15 • 4 CiTO-Ps with moderate local positive agreement

• i.e., citesAsPotentialSolution (0.71), citesAsDataSource (0.63), citesAsRecommendedReading (0.52), includesQuotationFrom (0.69)

Data evaluation

The global agreement is

very low

STLab University of Bologna#eswc2014Ciancarini

“What was the global inter-rater agreement of the subjects?” “Did the number of available choices bias the global inter-rater agreement?”

“Which properties showed an acceptable positive agreement among subjects?”

13

• Condition T41

• Global Fleiss’k = 0.13 • 5 CiTO-Ps with moderate local positive agreement (k > 0.5)

• i.e., citesAsPotentialSolution (0.66), citesAsRecommendedReading (0.6), agreesWith (0.54), citesAsDataSource (0.52), usesMethodIn (0.54)

• Condition T10

• Global Fleiss’k = 0.15 • 4 CiTO-Ps with moderate local positive agreement

• i.e., citesAsPotentialSolution (0.71), citesAsDataSource (0.63), citesAsRecommendedReading (0.52), includesQuotationFrom (0.69)

Data evaluation

The global agreement is

very low

The # of CiTO-Ps does not affect the

agreement

STLab University of Bologna#eswc2014Ciancarini

“What was the global inter-rater agreement of the subjects?” “Did the number of available choices bias the global inter-rater agreement?”

“Which properties showed an acceptable positive agreement among subjects?”

13

• Condition T41

• Global Fleiss’k = 0.13 • 5 CiTO-Ps with moderate local positive agreement (k > 0.5)

• i.e., citesAsPotentialSolution (0.66), citesAsRecommendedReading (0.6), agreesWith (0.54), citesAsDataSource (0.52), usesMethodIn (0.54)

• Condition T10

• Global Fleiss’k = 0.15 • 4 CiTO-Ps with moderate local positive agreement

• i.e., citesAsPotentialSolution (0.71), citesAsDataSource (0.63), citesAsRecommendedReading (0.52), includesQuotationFrom (0.69)

Data evaluation

The global agreement is

very low

The # of CiTO-Ps does not affect the

agreementThe set of CiTO-PS with moderate local positive

agreement is little affected by the # of

CiTO-Ps

STLab University of Bologna#eswc2014Ciancarini

• We applied the Chinese Whispers clustering algorithm

• Input: 2 graphs built by combining all the pairs of different CiTO-Ps as annotated by subjects for each citation

• Gr - it takes into account repetitions in annotations for a each CiTO property

• e.g., “extends”, “extends”, and “updates” on a citation generate (extends,updates) and (extends,updates)

• Gn - it does not take into account repetitions

• e.g., “extends”, “extends”, and “updates” generate (extends,updates)

14

Clustering CiTO-Ps“Could properties be organized

according to their similarity in subjects’ annotations?”

STLab University of Bologna#eswc2014Ciancarini14

Clustering CiTO-Ps“Could properties be organized

according to their similarity in subjects’ annotations?”

Gr Gn

disputes

critiques

deridesrefutes confirms

credits

obtainsSupportFrom

STLab University of Bologna#eswc2014Ciancarini14

Clustering CiTO-Ps“Could properties be organized

according to their similarity in subjects’ annotations?”

Gr Gn

disputes

critiques

deridesrefutes confirms

credits

obtainsSupportFrom

There exist some sort of relations (e.g., taxonomical,

equivalence) among the CiTO-Ps of each cluster

STLab University of Bologna#eswc2014Ciancarini15

• We computed the System Usability Scale (SUS)

Measuring the usability of CiTO-Ps

0.00!

10.00!

20.00!

30.00!

40.00!

50.00!

60.00!

70.00!

80.00!

90.00!

100.00!

SUS mean! Usability mean! Learnability mean!

T41!

T10!

“What was the perceived usability of the CiTO-Ps? ”

STLab University of Bologna#eswc2014Ciancarini15

• We computed the System Usability Scale (SUS)

Measuring the usability of CiTO-Ps

0.00!

10.00!

20.00!

30.00!

40.00!

50.00!

60.00!

70.00!

80.00!

90.00!

100.00!

SUS mean! Usability mean! Learnability mean!

T41!

T10!

“What was the perceived usability of the CiTO-Ps? ”

Only the usability score approaches the statistical significance

STLab University of Bologna#eswc2014Ciancarini16

Grounded theory analysis

• The subjects filled a final text questionnaire aimed at capturing positive and negative aspects of CiTO-Ps

• We used the text answers for performing a grounded theory analysis, used in Social Science to extract relevant concepts from unstructured text

“Which were the features of CiTO-Ps that subjects perceived as most useful or problematic?”

STLab University of Bologna#eswc2014Ciancarini16

Grounded theory analysis“Which were the features of CiTO-Ps that subjects perceived as most useful or

problematic?”

STLab University of Bologna#eswc2014Ciancarini17

Conclusions• Lessons learnt and suggestions to improve CiTO

• Reduce the number of less-used properties

• Identify the most-used neutral properties

• Investigate motivations for low inter-rater agreement

• Define explicit relations between CiTO properties

• Add support for customised properties

• Extend examples, labels and explanations

• Future work

• Improve CiTalO, a tool for identifying automatically the nature of citations

• e.g., by investigating cognitive architectures in order to simulate humans’ behaviour

STLab University of Bologna#eswc2014Ciancarini18

Thank you!