discovery or displacement?: a large scale longitudinal study of the effect of discovery systems on...

Discovery  or  Displacement?  A  Large  Scale  Longitudinal  Study  of  the  Effect  of  Discovery  Systems  on  Online  

Journal  Usage  

Charleston  Conference  November  7,  2013  

Michael  Levine-­‐Clark,  University  of  Denver  John  McDonald,  University  of  Southern  California  

Jason  Price,  SCELC  ConsorJum  

“…a steep increase in full text downloads and link resolver click‐throughs suggests Summon had a dramatic impact on user behavior and the use of library collections during this time period.” The Impact of Web-scale Discovery on the Use of a Library Collection Doug Way (2010) http://scholarworks.gvsu.edu/library_sp/9/

h"p://www.oclc.org/partnerships/econtent/solu4ons.en.html  

Vendor  marke5ng  

Does implementation of a discovery service impact journal usage?

Web-­‐scale  discovery  services  

•  Single  source  for  finding  informaJon  –  Books  –  ArJcles  –  Local  content  

•  Metadata  and/or  full  text  

•  Content  is  pre-­‐indexed  and/or  pre-­‐harvested  

•  Single  fast  search    

ILS  

HathiTrust  

MLA  Bibliography  

InsJtuJonal  Repository  

Publisher  Metadata  

Discovery  Service  

An  assump5on  

•  At  any  given  insJtuJon,  given  a  relaJvely  stable  user  base,  the  total  search  effort  will  remain  roughly  the  same.  – X  students  will  have  Y  assignments  and  Z  hours  per  day  to  search  

– X  faculty  will  publish  Y  papers  and  have  Z  hours  per  day  to  search  

Discovery  services  § Will  take  up  an  increasing  amount  of  a  finite  Jme  for  searching  

§ Will  draw  users  from  other  (more  or  less  efficient)  search  tools  

§ Will  alter  the  overall  producJvity  of  searches  (users  will  find  more  or  less)  

§ Will  alter  the  overall  efficiency  of  users  (users  will  access  more  or  less  full-­‐text)  

Prior  studies  

•  Some  studies  have  indicated  substanJal  increases  in  usage  a]er  Discovery  implementaJon  – DescripJve  staJsJcs  only  – Single  insJtuJon  studies  only  

•  Some  publishers  report  decreased  usage  of  content  – Anecdotal,  may  affect  some  and  not  others  

Data  collec5on  •  List  of  libraries  with  discovery  services  

> Searched  on  lib-­‐web-­‐cats  •  Surveyed  Libraries  

> Discovery  service  Implemented  > ImplementaJon  Date  (month/year)  > Search  box  locaJon  > MarkeJng  effort  

•  149  Libraries  Gave  Approval  > 24  libraries  selected  for  this  phase  > 6  for  each  of  the  4  major  discovery  services  

Library  demographics  •  20  US,  1  each  from  UK,  AUS,  NZ,  CA  •  10  ARL  Libraries  included  • WorldCat  book  holdings  

> Average:  1,114,193  > Median:  1,044,153  > High:  2,665,796  > Low:  298,365  

•  ImplementaJon  dates:    > 2010  (3),  2011  (19),  2012  (2)  

Dataset  

•  24  Libraries  

•  4  Discovery  services  

•  6  Publishers  

•  9,206  Journals  

•  159,278  ObservaJons  

•  141,048  Usable  ObservaJons    

Methodology  Compared  COUNTER  JR1  total  full  text  arJcle  views  for  the  12  months  before  vs  12  months  a]er  implementaJon  date  

June

 2010  

Start  

Implem

entaJo

n  May  201

1  

May  201

2  End  

Year  1   Year  2  

Included  implementaJon  month  in  Year  1  to  ensure  that  both  periods  included  an  enJre  academic  year  

Collec5ons  notes                

o  Excluded  journals  that  did  not  have  24  months  of  COUNTER  reporJng  

o  Limited  ability  to  control  for  changes  in  aggregator,  backfile  access,  or  expanded  holdings  

o  Outliers  removed  from  analysis  

General  trends  

•  VariaJon  by  insJtuJon  within  each  discovery  service  

•  VariaJon  by  publisher  within  each  discovery  service  

•  Some  publishers  saw  overall  net  increase,  while  some  experienced  a  decrease  in  usage  

Goals of our inferential statistics Determine whether observed differences are significant or resulted from chance effects

Determine which of the three factors (i.e. library, publisher, discovery service) contribute to determining differences in usage change at the journal level

Start with an exploratory analysis and end with a comprehensive model

ANOVA - Analyzing the data

Observation = Fit + Residual Change In = Library x + Publisher y + Disc Svc z + Residual Err usage +17 = (+2) + (-3) + (+10) + (+8)

After Cobb 2003 Introduction to design and analysis of experiments. Fig 3.1

ANOVA – F Ratio Tests whether the means for levels within a factor are distinguishable from each other

Average variability due to the factor F-ratio = --------------------------------------------------- Average variability due to chance error So, when F ≈ 1, means are not distinguishable, when F is > 1, there are real differences among some means

Does usage change vary across libraries?

Institution (sorted by Mean Change)

Overall Average = 8.5

Does usage change vary across libraries?

Institution (sorted by Mean Change)

Overall Average = 8.5

Does usage change vary across publishers?

Publisher (sorted by Mean Change)

Overall Average = 8.9

Does usage change vary across discovery services?

Overall Average = 8.9

Does the affect of discovery service differ across publishers?

Does the affect of discovery service differ across publishers?

Publishers (distinguished by color)

Do the discovery service means differ in the 2 way model?

Publishers (distinguished by color)

4.5

12.3

15.0

3.7

Publishers (distinguished by color)

4.5

12.3

15.0

3.7

Publishers (distinguished by color)

Do publisher means differ significantly in the two way model?

23.8

-3.9

6.8-9.5

Publishers (distinguished by color)

Do publisher means differ significantly in the two way model?

23.8

-3.9

6.8-9.5

Does the affect of discovery service differ across publishers?

Publishers (distinguished by color)

Publishers (distinguished by color)

Full Model – including disc srvc, publisher, and library

ANOVA Model including all three factors

Results - Can we detect differences between Discovery Services, Publishers, and/or Libraries and/or their interactions?

Discovery Service – Yes Publisher – No Library – Yes Differential discovery service effect by publisher – Yes

Differential library effect by publisher -- Yes

Interpreta5ons  &  Conclusions  >  Analyzing  usage  is  a  complex  task  >  No  discovery  service  increased  or  decreased  usage  across  all  libraries  and/or  all  publishers  

>  Discovery  service  and  publisher  as  variables  on  their  own  were  significant  predictors  of  usage  change  

>  InteracJon  of  Discovery  service  &  Publisher  was  significant  

>  Some  control  needed  for  no  discovery  service  and  for  size  of  insJtuJon.  

A  plethora  of  pending  possible  pursuits  •  Design  &  test  for  effects  of:    

–  Aggregator  full  text  availability  –  InsJtuJon  Size  /  Enrollment  Profile  –  Publisher  Size  –  Journal  Subject  –  Overall  usage  trends  (Requires  Disc  Srvc  ‘control’)  –  ConfiguraJon  opJons  in  Discovery  services  

•  Follow-­‐up  presenta5on  at  UKSG  (April  2014)  –  Including  Control  group  &  AddiJonal  libraries  –  Add  AddiJonal  variables  &  further  analysis  

Ques5ons  

michael.levine-­‐clark@du.edu  |  johndmcd@usc.edu  |  jason@scelc.org