iterative readability computation for domain-specific resources
DESCRIPTION
Iterative Readability Computation for Domain-Specific Resources. By Jin Zhao and Min-Yen Kan 11/06/2010. Domain-Specific Resources. Domain-specific resources cater for a wide range of audience. Wikipedia page on modular arithmetic. Interactivate page on clocks and modular arithmetic. - PowerPoint PPT PresentationTRANSCRIPT
Iterative Readability Computation for Domain-Specific Resources
• By Jin Zhao and Min-Yen Kan11/06/2010
Jin Zhao and Min-Yen Kan
11/06/2010 / 20
Domain-Specific Resources
2WING, NUS
Wikipedia page on modular arithmetic
Interactivate page on clocks and modular arithmetic
Domain-specific resources cater for a wide range of audience.
Jin Zhao and Min-Yen Kan
11/06/2010 / 20
Challenge for a Domain-Specific Search Engine
3WING, NUS
How to measure readability for domain-specific resources?
Jin Zhao and Min-Yen Kan
11/06/2010 / 20
Literature Review• Heuristic Readability Measures– Weighted sum of textual feature values
– Examples: Flesch Kincaid Reading Ease:
Dale-Chall:
– Quick and indicative but oversimplifying
4WING, NUS
Jin Zhao and Min-Yen Kan
11/06/2010 / 20
Literature Review• Natural Language Processing and Machine Learning
Approaches– Extract deep text features and construct sophisticated models for
prediction
– Text Features N-gram, height of parse tree, Discourse relations
– Models Language Model, Naïve Bayes, Support Vector Machine
– More accurate but annotated corpus required and ignorant of the domain-specific concepts
5WING, NUS
Jin Zhao and Min-Yen Kan
11/06/2010 / 20
Literature Review• Domain-Specific Readability Measures– Derive information of domain-specific concepts from expert
knowledge sources
– Examples: Wordlist Ontology
– Also improves performance but knowledge sources still expensive and not always available
6WING, NUS
Is it possible to measure readability for domain-specific resources without expensive
corpus/knowledge source?
Jin Zhao and Min-Yen Kan
11/06/2010 / 20
Intuitions• A domain-specific resource is less readable than another if the
former contains more difficult concepts
• A domain-specific concept is more difficult than another if the former appears in less readable resources
• Use an iterative computation algorithm to estimate these two scores from each other
• Example:– Pythagorean theorem vs. ring theory
7WING, NUS
Jin Zhao and Min-Yen Kan
11/06/2010 / 20
Algorithm• Required Input– A collection of domain-specific resources (w/o annotation)– A list of domain-specific concepts
• Graph Construction– Construct a graph representing resources, concepts and
occurrence information
• Score Computation– Initialize and iteratively compute the readability score of domain-
specific resources and the difficulty score of domain-specific concepts
8WING, NUS
Jin Zhao and Min-Yen Kan
11/06/2010 / 209WING, NUS
Graph Construction• Preprocessing– Extraction of occurrence information
• Construction steps– Resource node creation– Concept node creation– Edge creation based on occurrence information
Pythagorean Theorem……triangle… …sine……tangent…
trigonometry...sine… …tangent……triangle…
Resource 1 Resource 2 Concept List
Pythagorean Theorem,tangent, triangle trigonometry, sine,
Pythagorean Theorem
triangle
sine
tangent
trigonometry
Resource 1
Resource 2
Jin Zhao and Min-Yen Kan
11/06/2010 / 2010WING, NUS
Score Computation• Initialization– Resource Node (FKRE)– Concept Node (Average score of neighboring nodes)
• Iterative Computation– All nodes (Current score + average score of neighboring nodes)
• Termination Condition– The ranking of the resources stabilizes
w x y z
a b c
Resource Nodes
Concept Nodes
w x y z a b cInitialization 1 3 3 5 2 3 4
Iteration 1 3 5.5 6.5 9 4 6 8
Iteration 2 7 10.5 13.5 17 8.25 12 15.75
Jin Zhao and Min-Yen Kan
11/06/2010 / 20
Evaluation• Goals– Effectiveness
Iterative computation vs. other readability measures in math domain
– Efficiency Iterative computation with domain-specific resources and
concepts selection in math domain– Portability
Iterative computation vs. other readability measures in medical domain
11WING, NUS
Jin Zhao and Min-Yen Kan
11/06/2010 / 20
Effectiveness Experiment• Corpus– Collection
27 math concepts 1st 100 search results from Google
– Annotation 120 randomly chosen webpages
Annotated by first author and 30 undergraduate students using a 7-point readability scale
Kappa: 0.71, Spearman’s rho: 0.9312WING, NUS
Value Education Background
1 Primary
2 Lower Secondary
3 Higher Secondary
4 Junior College (Basic)
5 Junior College (Advanced)
6 University (Basic)
7 University (Advanced)
Jin Zhao and Min-Yen Kan
11/06/2010 / 20
Effectiveness Experiment• Baseline:– Heuristic
FKRE– Supervised learning
Naïve Bayes, Support Vector Machine, Maximum Entropy Binary word features only
• Metrics:– Pairwise accuracy– Spearman’s rho
13WING, NUS
Jin Zhao and Min-Yen Kan
11/06/2010 / 20
Effectiveness Experiment• Results– FKRE and NB show modest
correlation
– SVM and Maxent perform significantly better
– Best performance is achieved by iterative computation
14WING, NUS
Pairwise SpearmanFKRE .72 .48
NB .72 .52
SVM .80 .70
Maxent .82 .67
IC .85 .72
Jin Zhao and Min-Yen Kan
11/06/2010 / 20
Efficiency Experiment• Corpus/Metrics same as before
• Different selection strategies– Resource selection by random– Resource selection by quality– Concept selection by random– Concept selection by TF.IDF
15WING, NUS
Jin Zhao and Min-Yen Kan
11/06/2010 / 20
Efficiency Experiment• Results– If chosen at random, the more
resources/concepts the better
– When chosen by quality, a small set of resources is also sufficient
– Selection by TF.IDF helps to filter out useless concepts
16WING, NUS
20% 40% 60% 80% 100%0.5
0.550.6
0.650.7
0.750.8
0.850.9
0.951
Quality (Pairwise) Random (Pairwise)Quality (Spearman) Random (Spearman)
20% 40% 60% 80% 100%0.5
0.550.6
0.650.7
0.750.8
0.850.9
0.951
TF.IDF (Pairwise) Random (Pairwise)TF.IDF (Spearman) Random (Spearman)
Jin Zhao and Min-Yen Kan
11/06/2010 / 20
Portability Experiment• Corpus– Collection
27 medical concepts 1st 100 search results from Google
– Annotation Readability of 946 randomly chosen webpages annotated by
first author on the same readability scale
• Metric/Baseline same as before
17WING, NUS
Jin Zhao and Min-Yen Kan
11/06/2010 / 20
Portability Experiment• Results– Heuristic is still the weakest
– Supervised approaches benefit greatly from the larger amount of annotation
– Iterative computation remains competitive
– Limited readability spectrum in medical domain
18WING, NUS
Pairwise Spearman
FKRE .63 .28
NB .73 .53
SVM .82 .70Maxent .76 .60
IC .72 .49
ICS .75 .54
Jin Zhao and Min-Yen Kan
11/06/2010 / 20
Future Work• Processing– Noise reduction
• Probabilistic formulation– Distribution of values
e.g. 70% of webpages highly readable and 30% much less readable
– Correlations between multiple pairs of attributes e.g. Genericity and page type
19WING, NUS
Jin Zhao and Min-Yen Kan
11/06/2010 / 20
Conclusion• Iterative Computation– Readability of domain-specific resources and difficulty of
domain-specific concepts can be estimated from each other– Simple yet effective, efficient and portable
• Part of the exploration in Domain-specific Information Retrieval
– Categorization– Readability– Text to domain-specific construct linking
20WING, NUS
Jin Zhao and Min-Yen Kan
11/06/2010 / 20
Any questions?
21WING, NUS
Jin Zhao and Min-Yen Kan
11/06/2010 / 20
Related Graph-based Algorithms • PageRank– Directed links– Backlinks indicate popularity/recommendation
• HITS– Hub and authority score for each node
• SALSA
22WING, NUS