Evaluation of the impact of library discovery technologies on usage of academic content

Presenters: Angela Conyers, Evidence Base (BCU)

Valérie Spezi, LISU (Loughborough University)

Why this study?

• Commissioned by UKSG/Jisc in July 2013Lots of interest in library discovery technologiesQuestions about whether libraries, publishers and other

stakeholders should be engaging with those technologiesFollows on from the UKSG research report on Link resolvers

and the serials supply chain and the work of KBART

• Small-scale study A UK perspective

• No previous usage data analyses – fill in the gap 2 studies about to report shortly

• Work started in July 2013

Objectives of the research

Evaluation of the impact of library discovery technologies on

usage of academic resources Provide evidence to determine if there is a case for

Investment in library discovery technologies by librariesEngagement with library discovery technologies by

publishers and other stakeholders in the information supply chain

Provide recommendations for stakeholders to best support the discovery of academic resources

Identify additional research, data, discussion and initiatives that will support the findings of the study

Methodology

• Phase 1: survey of UK HE libraries Objective: determine the current RDS landscape

• Phase 2: case studies of libraries and publishers Objective: collect usage data + views and perceptions on the

impact of library discovery technologies

• Phase 3: interviews with stakeholders Objective: obtain a bigger picture on the perceived impact of

library discovery technologies and an insight of where the sector is going

Methodology (cont’d)

• Survey: 62 respondentsOnline questionnaire distributed to UK HE library

directors

• 8 publishers and stakeholders

• 6 case study libraries

• Data received from 6 libraries & 4 publishers/content providersCOUNTER JR1, BR2 and DB1 or close equivalent

Phase 1: UK RDS landscape

RDS use in academic libraries

2007-08 2008-09 2009-10 2010-11 2011-12 2012-13 2013-140

5

10

15

20

25

2 1

4

7

10

21

1

2

5

In process Implemented

No

of re

spon

dent

s

The UK HE landscape:

• 77% of UK HE libraries are already using an RDS at their institution

• A further 11% are in the process of implementing an RDS

• RDS implementation in HE libraries had probably reached its peak in the last 12 months

Phase 1: UK RDS landscape

RDS products used by UK HE libraries

• 3 products dominate the library discovery market in our sample

• Half of the survey respondents considered the RDS to be a replacement for their previous OPAC, although there was an indication that online catalogues are still needed for some specific transactions

2% 2%

24%

3%2%

26%

36%

2%2%

2%

AquaBrowser

Blacklight

Ebsco Discovery Service

Encore

Endeca

Primo

Summon

VuFind

WorldCat Local

Other RDS

Phase 2: usage study

What are we going to present?

1. General findings – general picture• Data received from the 6 participating libraries and 4 of the

participating publishers• Libraries’ usage data – journal, e-book and database• Publishers’ usage data – journal

2. Findings for a case study library – more detailed picture

Overall journal usage trends

RDS - 12

RDS - 10

RDS - 8

RDS - 6

RDS - 4

RDS - 2 RDS

RDS + 2

RDS + 4

RDS + 6

RDS + 8

RDS + 10

RDS + 12

RDS + 14

RDS + 16

RDS + 18

RDS + 20

RDS + 22

RDS + 24-2.00

-1.00

0.00

1.00

2.00

3.00

4.00

5.00

6.00

A

B

C

D

E

F

Rela

tive

usa

ge p

er F

TE

RDS implementation

1st year 2nd year

Overall journal usage trends – constant titles

RDS - 12

RDS - 10

RDS - 8

RDS - 6

RDS - 4

RDS - 2

RDS

RDS + 2

RDS + 4

RDS + 6

RDS + 8

RDS + 10

RDS + 12

RDS + 14

RDS + 16

RDS + 18

RDS + 20

RDS + 22

RDS + 24

-2.00

-1.00

0.00

1.00

2.00

3.00

4.00

5.00

AB

C

D

E

F

Rela

tive

usa

ge p

er F

TE

RDS implementation

Overall e-book usage trends

RDS - 12

RDS - 11

RDS - 10

RDS - 9

RDS - 8

RDS - 7

RDS - 6

RDS - 5

RDS - 4

RDS - 3

RDS - 2

RDS - 1

RDS

RDS + 1

RDS + 2

RDS + 3

RDS + 4

RDS + 5

RDS + 6

RDS + 7

RDS + 8

RDS + 9

RDS + 10

RDS + 11

RDS + 12

RDS + 13

RDS + 14

RDS + 15

RDS + 16

RDS + 17

RDS + 18

RDS + 19

RDS + 20

RDS + 21

RDS + 22

RDS + 23

RDS + 24

-3.00

-1.00

1.00

3.00

5.00

7.00

9.00

11.00

13.00

15.00

A

B

D

E

F

Rela

tive

usa

ge p

er F

TE

RDS implementation

Overall e-book usage trends – constant titles

RDS - 12

RDS - 10

RDS - 8

RDS - 6

RDS - 4

RDS - 2

RDS

RDS + 2

RDS + 4

RDS + 6

RDS + 8

RDS + 10

RDS + 12

RDS + 14

RDS + 16

RDS + 18

RDS + 20

RDS + 22

RDS + 24

-3.00

-1.00

1.00

3.00

5.00

7.00

9.00

11.00

13.00

A

B

D

E

F

Rela

tive

usa

ge p

er F

TE

RDS implementation

Overall database usage trends

RDS - 12

RDS - 10

RDS - 8

RDS - 6

RDS - 4

RDS - 2

RDS

RDS + 2

RDS + 4

RDS + 6

RDS + 8

RDS + 10

RDS + 12

RDS + 14

RDS + 16

RDS + 18

RDS + 20

RDS + 22

RDS + 24

-5.50

-4.50

-3.50

-2.50

-1.50

-0.50

0.50

1.50

2.50

3.50

B

C

D

F

Rela

tive

usa

ge p

er F

TE

RDS implementation

Publishers - overall journal trends

RDS - 12

RDS - 10

RDS - 8

RDS - 6

RDS - 4

RDS - 2

RDS

RDS + 2

RDS + 4

RDS + 6

RDS + 8

RDS + 10

RDS + 12

RDS + 14

RDS + 16

RDS + 18

RDS + 20

RDS + 22

RDS + 24

-0.30

-0.20

-0.10

0.00

0.10

0.20

0.30

0.40

W

XY

Z

Rela

tive

usa

ge p

er F

TE

RDS implementation

Broad picture – key findings

• What does the usage analysis suggest?• Journals – some impact• E-books – definitely a positive impact• Databases – cannot tell

• Interpretation of the aggregated data – challenge - why?• Multi-dimensional environment – lots of noise in the results

• other factors may affect usage, notably the volume of content available

• We’ve tried to control for this with the ‘constant titles’ but there may be other factors

• No common pattern by type of resources for each library• Except for e-books maybe?

• Next: look at a case study library in greater detail

Overall time series for journals, eBooks and databases – data for a single case study library

Sep-08

Nov-08

Jan-09

Mar-09

May-09Jul-0

9

Sep-09

Nov-09

Jan-10

Mar-10

May-10Jul-1

0

Sep-10

Nov-10

Jan-11

Mar-11

May-11Jul-1

1

Sep-11

Nov-11

Jan-12

Mar-12

May-12Jul-1

2

Sep-1210,000

100,000

1,000,000

10,000,000

100,000,000

JR1 BR2 DB1JR1 trend BR2 trend DB1 trend

Mon

thly

usa

ge to

tal

RDS implemented

Logarithmic scale

Journal usage per FTE for library E

• Journal usage levels increasing before RDS

• RDS effect on constant titles shows a decrease in usage level immediately after implementation and then a sustained increase at a higher pace-2

-1

0

1

2

3

4

5

6

AllConstant titlesWXYZ

Rela

tive

usa

ge p

er F

TE

Increased journal title availability

E-book usage per FTE for library E

• Overall usage going up

• Constant titles going up Increase inflated by one

• Subscription making up for 70% of the usage of the constant title set throughout

• E-book usage for publisher W increased by a factor of 8

-2.00

0.00

2.00

4.00

6.00

8.00

10.00

12.00

14.00

16.00

All

W

Constant titles

Relati

ve u

sage

per

FTE

New e-book collections

Database usage per FTE for library E

• database usage increased significantly immediately after RDS

• Issues with database counting?

-50

0

50

100

150

200

250

Constant titles

excl. platform M

excl. resource N

Rela

tive

usa

ge p

er F

TE

Suspected multiple counting issue

Usage analysis - summary

What can we say from the usage analysis?

• No straightforward answer – but it seems that overall library discovery technologies influence positively the use of academic resources• The effect may vary according to the type of resources – e-books seem to

benefit greatly from RDS

• Isolating the sole impact of RDS is a challenge – multi-dimensional environment• Many other factors may affect usage – we have tried to control for content

growth by providing an analysis including constant titles

• More data needed for a meaningful analysis• 2-year post-implementation data is not enough to pick up a trend and isolate

other variables influencing usage

Additional findings from libraries

Perceived advantages for libraries:• Generally, high levels of library satisfaction with RDS• Increased usage – borne out by usage data• Enhanced user experience (primary motivation – not increased usage)

• One stop shop = single interface linked to full text

• Better use of subscriptions – no silos

Perceived challenges:• Usage data analysis – not done routinely• RDS searching aimed at undergrads? Starting point?

• Can researchers benefit from RDS too?

• Lack of clarity in coverage from RDS suppliers• Interoperability between systems• Lack of co-operation between some publishers and some RDS suppliers

Additional findings from publishers and content providers

Perceptions when engaging with RDS: • Improving discoverability and visibility of content

still very low traffic from RDS compared to search engines particularly relevant for smaller publishers? better service for their authors and readers can publishers afford to wait and see where it is going?

Perceived challenges:• Metadata RDS optimisation for improved discoverability• Dilution of the publisher’s brand within the RDS• Lack of feedback/communication from RDS suppliers• Lack of visibility and understanding of how data are used

Relevancy ranking

Other stakeholders in the information supply chain – some key findings

RDS – great tool – probably changing the library landscape in the short to medium term

But the use of RDS raises some questions: Are RDS a long term valid solution to Web-scale information

searching? Costs & benefits to libraries? What is the effect on A&I databases for specialist

information?

THANK YOU!

• Recommendations soon available in the full report that will be presented to UKSG

• For a copy of the report:• Check out the UKSG website - http://www.uksg.org/ • Contact us for a copy of the report - lisu@lboro.ac.uk