discovery study detailed results 20140728
DESCRIPTION
Levine-Clark, Michael, John McDonald, and Jason Price. Discovery or Displacement? A Large-Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage. July 23, 2014.TRANSCRIPT
![Page 1: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/1.jpg)
Discovery or Displacement? A Large-Scale Longitudinal Study of the Effect of Discovery Systems on Online
Journal Usage
July 23, 2014
Michael Levine-Clark, University of DenverJohn McDonald, University of Southern California
Jason Price, SCELC Consortium
![Page 2: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/2.jpg)
Why Do We Use Discovery Services?
• Too many sources of information– Specialized– Confusing– What’s the right specialized database for my
subject?
• Why do you search in one place for an article, another place for a different article, another for a book?
![Page 3: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/3.jpg)
Terminology
Discovery Tool = Discovery Layer = Discovery Service = Discovery System
![Page 4: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/4.jpg)
![Page 5: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/5.jpg)
Web-scale discovery services
• Single source for finding information– Books– Articles– Local content
• Metadata and/or full text
• Content is pre-indexed and/or pre-harvested
• Single fast search
ILS
HathiTrust
MLA Bibliograph
y
Institutional Repository
Publisher Metadata
Discovery Service
![Page 6: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/6.jpg)
Does implementation of a discovery service impact usage of publisher-
hosted journal content?
![Page 7: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/7.jpg)
What did we measure?
• Whether there is an effect
• NOT why that effect exists (that’s a future study!)
![Page 8: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/8.jpg)
Publisher-hosted journals are only part of the picture
eBooks, pBooks, newspaper articles, aggregator journal content, etc.
publisher journal content
The six publishers in this study
![Page 9: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/9.jpg)
52.3
12.41
2.8
32.5
Open web searchLibrary referralsSocial mediaAcademicN/A
Journals Traffic Sources (SAGE, Conrad ALPSP 2013)
![Page 10: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/10.jpg)
An assumption
• At any given institution, given a relatively stable user base, the total search effort will remain roughly the same.
![Page 11: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/11.jpg)
Discovery services Will take up an increasing amount of a finite
time for searching
Will draw users from other (more or less efficient) search tools
Will alter the overall productivity of searches (users will find more or less)
Will alter the overall efficiency of users (users will access more or less full-text)
![Page 12: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/12.jpg)
Dataset• 33 Libraries– 28 US, 2 CA, 1 each from UK, AUS, NZ
– WorldCat book holdings>Average: 1,114,193 ; Range: ~300k to ~2.6mil
– 4 discovery groups, of 6 libraries each
– 1 control group, 9 libraries• Implementation dates (Discovery Libraries):
>2010 (3), 2011 (19), 2012 (2)
• 6 Publishers• 9,206 Journals• 163,545 Usable Observations
![Page 13: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/13.jpg)
Methodology
Compared COUNTER JR1 total full text article views for the
12 months before vs 12 months after implementation date
June
201
0St
art
Impl
emen
tatio
nM
ay 2
011
May
201
2En
d
Year 1 Year 2
Included implementation month in Year 1 to ensure that both periods included an entire academic year
![Page 14: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/14.jpg)
Examine Data for Outliers
![Page 15: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/15.jpg)
Analyzing Usage Change: % vs Total
Use 12 months before
Use 12 months
after% Change Total
Change
Journal A 500 600 20% 100
Journal B 5 15 200% 10
Which is the better measure?
Is it the same for publisher- & journal-level data?
![Page 16: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/16.jpg)
Observations by Publisher
![Page 17: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/17.jpg)
Journals by Library & Service
EDS Primo Summon WorldCat Control
![Page 18: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/18.jpg)
Average Journal Usage by Library
![Page 19: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/19.jpg)
Testable Effects
• Discovery Tool– Implemented by multiple libraries– Used to find content from all publishers
• Publisher– Accessible in all discovery tools– Accessible across multiple libraries
• Library– Uses content from multiple publishers– Uses only one discovery tool (so only within
DT)
![Page 20: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/20.jpg)
Full Model
Including Discovery Service, Publisher, and Library
Including Discovery Service, Publisher, and Library
![Page 21: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/21.jpg)
Nested ANOVA Model Results
![Page 22: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/22.jpg)
How does usage change differ across discovery services?
ABB
C
D
Letters indicate statistically significant differences (Tukey multiple comparisons, p < .05)
![Page 23: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/23.jpg)
How does usage change differ across publishers?
Publisher (sorted by Mean Change)
C
Letters indicate statistically significant differences (Tukey multiple comparisons, p < .01)
D
BBB
A
![Page 24: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/24.jpg)
How does usage change differ across publishers?
![Page 25: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/25.jpg)
Does usage change vary across libraries?
Institution (sorted by Mean Change)
![Page 26: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/26.jpg)
Usage Change Per Institution: All Journals
Control EDS Primo Summon WCL
![Page 27: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/27.jpg)
Usage Change By Institution: Pub 16.32
Primo WCLSummonEDSControl
![Page 28: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/28.jpg)
Usage Change By Institution: Pub 2
Control EDS Primo Summon WCL
![Page 29: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/29.jpg)
Usage Change By Institution: Pub 3
WCLEDS PrimoControl Summon
![Page 30: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/30.jpg)
Usage Change By Institution: Pub 4
WCLEDSControl Primo Summon
![Page 31: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/31.jpg)
Usage Change Per Institution: Pub 5
Control EDS Primo Summon WCL
![Page 32: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/32.jpg)
Usage Change by Institution: Pub 6
WCLSummonPrimoEDSControl
![Page 33: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/33.jpg)
Publisher 1 by Discovery System
![Page 34: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/34.jpg)
![Page 35: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/35.jpg)
![Page 36: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/36.jpg)
![Page 37: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/37.jpg)
![Page 38: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/38.jpg)
![Page 39: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/39.jpg)
Publisher 2 by Discovery System
![Page 40: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/40.jpg)
![Page 41: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/41.jpg)
![Page 42: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/42.jpg)
![Page 43: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/43.jpg)
![Page 44: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/44.jpg)
![Page 45: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/45.jpg)
Publisher 3 by Discovery System
![Page 46: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/46.jpg)
![Page 47: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/47.jpg)
![Page 48: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/48.jpg)
![Page 49: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/49.jpg)
![Page 50: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/50.jpg)
![Page 51: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/51.jpg)
Publisher 4 by Discovery System
![Page 52: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/52.jpg)
![Page 53: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/53.jpg)
![Page 54: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/54.jpg)
![Page 55: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/55.jpg)
![Page 56: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/56.jpg)
![Page 57: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/57.jpg)
Publisher 5 by Discovery System
![Page 58: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/58.jpg)
![Page 59: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/59.jpg)
![Page 60: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/60.jpg)
![Page 61: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/61.jpg)
![Page 62: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/62.jpg)
![Page 63: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/63.jpg)
Publisher 6 by Discovery System
![Page 64: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/64.jpg)
![Page 65: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/65.jpg)
![Page 66: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/66.jpg)
![Page 67: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/67.jpg)
![Page 68: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/68.jpg)
![Page 69: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/69.jpg)
Next Steps• Design & test for effects of:
–Aggregator full text availability–Journal age (archive vs current)–Journal Subject–Overall usage trends–Configuration options in Discovery services
• Expand pool of libraries• Perhaps explore WHY
![Page 70: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/70.jpg)
Sharing Data• With participating libraries
–Customized reports for each library
• With participating publishers–Customized reports for each publisher–Presentations as requested
• With discovery vendors–Presentations as requested
• In publications and presentations–Maintaining anonymity of data
![Page 71: Discovery study detailed results 20140728](https://reader033.vdocuments.us/reader033/viewer/2022061204/546ea83caf795987318b4934/html5/thumbnails/71.jpg)
Presentations• Ithaka Sustainable Scholarship Conference (October 2013)• Charleston Conference (Nov 2013) http://sched.co/17A3Kun
• ER&L/Library Journal Webinar (December 2013)• Shangai Jiao Tong Univ / Beijing Univ Forum (Jan 2014)• SCELC Colloquium (March 2014) http://goo.gl/WmJoIw
• ER&L (Mar 2014) http://bit.ly/discovery-impact-erl2014 • UKSG (April 2014) https://www.youtube.com/watch?v=d_2ycMk_9fA
• Presentations to three publishers (Spring-Summer 2014)–More to come