Evaluating Visual and Statistical Exploration of Scientific Literature Networks
Robert Gove1,3, Cody Dunne1,3, Ben Shneiderman1,3, Judith Klavans2, and Bonnie Dorr2,3
1HCIL, 2CLIP, 3Computer ScienceUniversity of Maryland, College Park, USA
How do we help users explore scientific literature domains?
Commonly available capabilities
• Document statistics• Keyword summary• Citation visualization• Corpus statistics• Full-text search
Three types of systems
• Search engines & digital libraries• Reference managers• Summarization & recommender systems
Taxonomy of System Functionalities
Action Science Explorer
Visual and statistical exploration
Evaluation
• Corpus of 147 papers on Dependency Parsing• Exploration-based usability testing– 4 domain experts, two hours each
1. Training
2. Predefined tasks
3. Exploration questions
4. Participant-defined tasks
1. Training
• 15-30 minutes• Showed videos demonstrating Action Science
Explorer• Answered participant questions• Gave think-aloud instructions
2. Predefined tasks
Number of participants
Capability
Task 1: Identifying important authors and
papers
Task 2: Collecting evidence
Attribute Ranking ███ 3 ██ 2
Find Communities █ 1
Plot Nodes
In-Cite Text █ 1
In-Cite Summary
Search █ 1
Sorting █ 1
Groups ██ 2 █ 1
3. Participants’ questions
• How has the field evolved?• What are subtopics within the field?• What are the most accessible papers?• What are the relationships between authors?• What are the methods and applications?
4. Participant-defined tasks
Number of participants
CapabilityAll participant-defined
tasks
Attribute Ranking ██ 2
Find Communities █ 1
Plot Nodes
In-Cite Text █ 1
In-Cite Summary
Search █ 1
Sorting █ 1
Groups █ 1
Answering participants’ questions
• 2/4 questions were answered– 1 unanswered question was uncompletable– 1 unanswered question not well supported
Design recommendations for exploration
• Confirmed design decisions– Provide metrics for ranking documents– Support user-created groups and document
annotation– Give overview of corpus
• Suggested new capabilities– Allow control over corpus refinement– Incorporate undo/redo functionality
Conclusion
• A taxonomy of capabilities in current tools• An evaluation of Action Science Explorer• A list of literature exploration questions• A list of system design recommendations
Robert Gove - [email protected]
Acknowledgements
• Supported by NSF grant IIS 0705832and Booz Allen Hamilton
• Dragomir Radev and Vahed Qazvinian, University of Michigan
• David Zajic and Michael Whidby,University of Maryland