integrating behavior user studies with log analysis
TRANSCRIPT
Integrating Behavior User Studies with Log Analysis
Tao ZhangAssistant ProfessorLibrary SciencePurdue University
Xi NiuAssistant ProfessorSoftware and Information SystemsUniversity of North Carolina at Charlotte
• Sample log record
Log analysis
50.117.41.253 - - [01/Sep/2014:00:08:46 -0400] "GET /primo_library/libweb/action/dlSearch.do?institution=PURDUE&vid=PURDUE&indx=1&bulkSize=20&search_scope=everything&highlight=true&query=any,contains,hard+time+for+soft+balancing HTTP/1.1" 200 45345 2373 03456FAEC7526F199BD42BEAE95030A5 - 50.117.41.253
• Request URL parsed into components:• Session ID• Search field• Query string• Facets
IP Date&Time RequestURL Status BytesSent ReferringURL UserAgent
• Data extraction
Log analysis
Originallogfiles
Sessionedcodedcomponents
Perl
Python
Import SASR
• Search behavior metrics:• Search fields, facets• Number of queries in session• Query length• Query formulation and reformulation
+ Big data (for all user base)+ Unobtrusive+ Established metrics+ Efficient
Log analysis
- Task context- System response- User needs and perceptions- User actions and
preferences
Behavioraluserstudy
• Comparing two discovery tools (Nov.8toDec.7,2012)
Case study
Niu, X., Zhang, T., & Chen, H. L. (2014). Study of user search activities with two discovery tools at an academic library. International Journal of Human-Computer Interaction, 30(5), 422-433.
• Search field
Log analysis results
Percentage of keyword searches: VuFind: 68.4%, Primo: 88.2%
• Percentage of facet operations in all search actions• VuFind: 8.4%• Primo: 9.7%
• Top used facets• VuFind: Format, Access, Topic, Building, Author• Primo: Show Only (Online, Peer-Review, On Shelf), Format,
Subject, Publication Date, Library
• Nested facet selections are rare
Log analysis results
• Query results for Primo
Log analysis results
Non-electronic resources Mean (SD)
Electronic resourcesMean (SD)
Query length 5.1(5.4) 4.1(4.0) Number of query submissions
3.6 (5.4) 2.6(2.3)
Percentages of searches that were reformulated
61.0% 57.8%
• Visualization of query reformulation
Log analysis results
Narrowing:
Parallel:
Mixed:
• Users predominantly use keyword (default) search• Use of facets is relatively low• Most search sessions involve fewer than 4 queries• Average number of words per query is generally less than
3• More than half of search sessions reformulate queries by
adjusting original keywords
Summary of log analysis results
• One-and-one user test• Understand the search context• Designed tasks• Lab observations of user interaction with discovery tools
• Query• Search field• Facet• Search results list• Individual item
Behavioral user study
Usability lab
Type Instruction ObservationClose-ended task Find the book Introduction to
Algorithms by Thomas H. Cormen
QuerySearch field
Determine if the library has the book The Machine that Changed the World: The Story of Lean Production by James Womack
QueryFacet
Find the book and video of Wizard of Oz
Facet
Open-ended task Find a recent journal article on soap operas (as a sociology student)
QueryFacet
Find an e-book on Supply Chain Management
Facet
Locate the book No Impact Man in a library closest to you
Facet
User study tasks
• General behavior pattern:• Start with default search and keyword from instruction• Browse first page of results• Low usage of facets• Reformulate keywords from instruction when target not in top
results, not using facets
• Users’ difficulties with search results:• Scan potentially large number of results• Identify material type (book, article, journal, video)• Identify format (print, online access)
General observations
• Query formulation• Short queries for both open-ended and close-ended searches• Users want more initial search results (just in case …)• Primo may return 0 results for long queries
• Query reformulation• When top results not relevant• Limited effect of number of search results• Users preferred reformulating queries than using facets (adding
keywords like “book”, “article”, “journal”, etc.)• No clear search strategy, but users tended to narrow a search
than to broaden one
Log results & observation
• Reasons for low facet usage:• Interface design • Users’ awareness of available facets• Facet combinations not intuitive• Users’ understanding of the terminology
• Users used facets for:• Refine results (Online, Peer-Review, On Shelf)• Exclude unwanted results (publication date)• Library location
Log results & observation
Facet UI change
Before After
Search results UI change
Before
Search results design change
After
• User study informed by log analysis results• Test tasks targeting certain discovery tool features (facet)• User behavior to observe• Questions about the context
• Behavior observations complement mining of big data• Task context• Potential usability issues• Underlying user needs
• Data-driven design changes• Search results for visual scanning• Simplified facets display for exploration and interaction
What we’ve learned
• Zhang,T.,Niu,X.,Zhu,L.&Chen,H.(2015).SearchinOne’sHand:HowUsersSearchaMobileLibraryCatalog.Paperpresentedatthe17thInternationalConferenceonHuman-ComputerInteraction,LosAngeles,CA.August2-7,2015.
• Zhang,T.,Niu,X.,&Promann,M.(inreview).AssessingUserExperienceofE-BooksinAcademicLibraries.PapersubmittedtoCollegeandResearchLibraries.
Subsequent studies
