PDA MetricsHow I Learned to Stop Worrying and Trust My Users

Andrew See Teresa Hazen Jason Dewland

Who is this guy?

Andrew SeeHead of User Services

Northern Arizona University

Cline Library

• Formerly at the University of Arizona Libraries. • Member of UAL’s PDA Metrics group• Former UAL Acquisitions Manager• Co-author: Patron driven acquisitions -

Determining the metrics for success. Library Resources and Technical Services

Forces for Change

ODID (on demand

information delivery

Declining Budget

Return on Investment

User Focused

Collection

Changing Role of

Librarians

PDA at the UA

• ODID Project began in the Summer of 2011

• ODID acts as the main driver of our collection.

• More than 142,000 discoverable titles have been added to our collection.

• Print: direct order links to our vendor are embedded in our Web OPAC and discovery layer. Purchases are delivered to our library shelf-ready and on-hold for the customer.

• Electronic: Seamless to the user. Purchases are triggered after a set number of uses

Attack of the what ifs

Losing control of your

collection?

Sky rocketing budget?

Unbalanced purchases?

Set yourself up for success

Pre-PDA

2010

(1 year)

Post-PDA

2011-present

(4 years)

Books purchased 13,723 18,121

Cost of Purchase $1,116,803 $1,743,146

Cost per Year $1,116,803 $435,787

Unpurchased

Records exposed 0 124,000

% purchased from

exposed titles 0% 12.75%

Create your profile

• All PDA titles are current

scholarly materials

• No textbooks

• No popular fiction

• No manuals

Data supplied by Stephen Bosch

Materials Budget, Procurement & Licensing Librarian

University of Arizona Libraries

Research Services Librarian

Library Analyst

Library Information Associate:

Acquisitions

Library Information

Associate: Library Infrastructure

Materials Budget, Procurement, &

Licensing Librarian

ODID Metrics Project Team

Charge:To coordinate the design and implementation of the data gathering processes to evaluate the ODID programs' effectiveness

Deliverables:An efficient set of metrics to help determine:• The quality of resources exposed to

library customers• Amount of use of those resources• Cost Effectiveness of the program• Customer satisfaction of the program

MetricsFinancial

• Cost Per Use (CPU)

• CPU by LC Class

• Total expenditures of program

Patron

• CustomerSatisfaction

• Customer Behavior

• User Experience

Performance

• Average Delivery Time of print resources

• eBooks allowing adequate download options

Usage

• Circulation of Print books

• Use of eBooks

• How delivery time of print resources affects usage

Resource

• Understanding of resource pool and how that drives user behavior

• Percent of resources purchased from available options

• How currency of resources affects usage

For a complete list and definition of metrics:

Dewland, J. C., & See, A. (2014). Notes on operations: Patron driven

acquisitions - Determining the metrics for success. Library Resources and

Technical Services, 59, 1, 13-23.

UA’s Print PDA ProgramUser and staff perspectives

Who is this gal?

Teresa HazenInterim Head of Delivery, Description, &

Acquisitions

University of Arizona Libraries

• Technical services librarian• Currently manages UA Libraries PDA programs

Print PDA Record in ILS Public Catalog

Authenticating

Coutts API captures information about the user based on the NetID so that a hold can be

eventually placed on the item for the customer. Information gathered via the API is imbedded in a

961 of the MARC record-

Order Placed by User

Status Changes in ILS

Success!

• User gets an email when book has

been processed and is on the hold

shelf ready to be picked up.

• The average is 10 business days

Some Steps in the Process for Library Staff

• Link in catalog is changed (daily report)

• MARC record file arrives with patron information in local 9xx MARC fields

• Staff overlays a bib. record, creates item, and edits order record

• Staff manually places a hold in ILS for user

Actual workflow somewhat more complicated…

Print PDA: Mapping the Workflow

Gathering the DataPulling data from your ILS for analysis and assessment

Used our ILS to Gather and Export the Data

• We are on Millennium 2014 2.0.1_2

• Used Create Lists functionality to gather records for exporting. Created

multiple lists and then exported certain fields in tab delimited format for

data normalization and analysis.

Limiters

�We only wanted to look at monographs regardless of format since we have a legacy of print but a recent history of almost entirely electronic.

�Made the decision to include only three library branches- our Main Library, Science and Engineering Library, and Fine Arts Library.

�Attempted to the best of our abilities to exclude government document loads (ex. MARCIVE).

Limiters, cont.

• We limited to monographs or eBooks—

• Material type= a (book) AND w (local code for ebook)

• Though we included only three libraries, this meant wading through dozens of location codes to include/exclude

• Ex.: Include Oversized

• Ex.: Exclude Information Commons Reference Desk

Yet More Limiters• Created multiple files each based on limiting to specific date ranges based on when

we started our patron driven acquisitions print and electronic programs:

• Pre-PDA: 01-01-2006 – 12-31-2009

• Post-PDA 01-01-2010 – 12-31-2014

• Post-PDA, no print PDA 01-01-2010 – 12-31-2014

• Weeded out as many government documents as possible by identifying a code in the ILS for all MARCIVE loaded documents (a bcode3, which is a locally defined code)

Creating the Print PDA File

• After creating the bigger monograph file from 2010-2014, we pulled out the

purchased print PDA:

• Ran a search on the 2010-2014 monograph list and included—

• The presence of an order record linked to the bibliographic record

• A 500 MARC note field indicating the record used to be a selection PDA

Creating the Export Files

We wanted to export out call number for analysis, knowing that some of our ebook collections don’t have a call number. We wanted location and material type as well as total number of checkouts as possible data points.

Fields to export included:

• Call number

• Location

• Material type

• For print PDA titles:

• Total checkouts (from item record)

• Estimated price (from order record)

Output the Export File & Future Issues

• Using Data Exchange functionality in Millennium, we output the file as a tab delimited for normalization and

analysis.

___________________________________

• With the introduction of discovery tools and next generation library management systems, there will be big

changes in the way we mine data from our curated content.

• As we move away from being in control of the creation of metadata to describe our content, we must push for

better metadata, such as LC call numbers, better identifiers of packages and collections.

• It’s really important to think in advance of actions you can do that help you work better with the data for future

analysis.

• Don’t ever get rid of call numbers

• Make use of local fields (9XX) to identify collections

• When using overlay functionality in your ILS, be sure to preserve metadata such as vendor control numbers

What the Data Tells UsPDA program’s real influence on the collection

Who is this guy?

Jason DewlandAssistant Librarian, Research & Learning

Department

University of Arizona Libraries

• Business & Tech Transfer Librarian• Member of UAL’s PDA Metrics group• Former UAL Acquisitions Manager• Co-author: Patron driven acquisitions -

Determining the metrics for success. Library Resources and Technical Services

Summary of Purchases 2006 -2009

• Added over 440,000 monographs during the time period.

• Approximately 110,000 were hard copy books.

• The rest were eBooks, government documents, pamphlets, and other

materials.

• We only have LC classification data for hard copy books, eBooks were

stripped of this data.

Summary of Purchases 2010 to 2014

• PDA made up only a small percentage (16.7%) of our purchases over this time period.

• PDA resulted in 7,128 monographs and 64,335 eBooks purchased.

• As a percent it was a similar pattern at the Sub - LC Classification pre & post PDA program.

• One hole is that eBooks were stripped of their LC number until 2011. Since the beginning of PDA approximately 16,965 (26.0%) eBooks purchased with no LC number.

• One difference in the shape of the collection – P – Language and Literature – 10% drop in percentage of purchases after implementation of PDA

Dewland, J., Elliott, C., Hazent, T., & See, A. “Patron Driven Acquisition:

Has it Changed Our Collections?” [Poster]. American Library Association Annual Conference,

San Francisco, CA., June 30th, 2015.

P – Language & Literature2010-2014 Comparison by Format

Hard Copy eBook

2010-2014 Count of P

Books Purchased

Percent of P Books

Purchased

2010-2014 Count of P

Books Purchased

Percent of P Books

Purchased

PE 107 8.4% PE 203 2.5%

PN 337 26.5% PN 2290 27.8%

PQ 54 4.2% PQ 394 4.8%

PR 190 14.9% PR 2007 24.4%

PS 265 20.8% PS 1388 16.9%

Total 1271 Total 8236

Red = Larger percentage of eBooks purchased than hardcopies.

Black = Larger percentage of hardcopies purchased than eBooks.

P – Language & Literature2006-2009 v. 2010-2014

2006-2009PDA

2010-2014

P 1167 3.6% P 1094 11.5%

PA 688 2.1% PA 287 3.0%

PB 52 0.2% PB 23 0.2%

PC 327 1.0% PC 58 0.6%

PD 16 0.0% PD 14 0.1%

PE 383 1.2% PE 310 3.3%

PF 125 0.4% PF 17 0.2%

PG 750 2.3% PG 115 1.2%

PH 25 0.1% PH 9 0.1%

PJ 2243 6.9% PJ 146 1.5%

PK 336 1.0% PK 54 0.6%

PL 2247 6.9% PL 247 2.6%

PM 148 0.5% PM 48 0.5%

PN 4076 12.5% PN 2627 27.6%

PQ 7614 23.3% PQ 448 4.7%

PR 4924 15.1% PR 2197 23.1%

PS 5620 17.2% PS 1653 17.4%

PT 1907 5.8% PT 156 1.6%

PZ 22 0.1% PZ 4 0.0%

Total 32670 Total 9507

P – Language & Literature2006-2009 v. 2010-2014

2006-2009 2010-2014 Change

PG 750 2.3% PG 115 1.2% 635 -1.1%

PJ 2243 6.9% PJ 146 1.5% 2097 -5.3%

PL 2247 6.9% PL 247 2.6% 2000 -4.3%

PQ 7614 23.3% PQ 448 4.7% 7166 -18.6%

PT 1907 5.8% PT 156 1.6% 1751 -4.2%

P 1167 3.6% P 1094 11.5% 73 7.9%

PN 4076 12.5% PN 2627 27.6% 1449 15.2%

PR 4924 15.1% PR 2197 23.1% 2727 8.0%

P – Language & Literature

Non-PDA and PDA2010-2014 Non-PDA 2010-2014 Combined 2010-2014

P 1094 11.5% P 4817 9.2% P 5911 9.6%

PA 287 3.0% PA 1171 2.2% PA 1458 2.4%

PB 23 0.2% PB 80 0.2% PB 103 0.2%

PC 58 0.6% PC 385 0.7% PC 443 0.7%

PD 14 0.1% PD 73 0.1% PD 87 0.1%

PE 310 3.3% PE 1065 2.0% PE 1375 2.2%

PF 17 0.2% PF 113 0.2% PF 130 0.2%

PG 115 1.2% PG 824 1.6% PG 939 1.5%

PH 9 0.1% PH 41 0.1% PH 50 0.1%

PI 0 0.0% PI 1570 3.0% PI 1570 2.5%

PJ 146 1.5% PJ 596 1.1% PJ 742 1.2%

PK 54 0.6% PK 1190 2.3% PK 1244 2.0%

PL 247 2.6% PL 12243 23.5% PL 12490 20.2%

PM 48 0.5% PM 2268 4.3% PM 2316 3.8%

PN 2627 27.6% PN 5405 10.4% PN 8032 13.0%

PQ 448 4.7% PQ 7263 13.9% PQ 7711 12.5%

PR 2197 23.1% PR 5624 10.8% PR 7821 12.7%

PS 1653 17.4% PS 6065 11.6% PS 7718 12.5%

PT 156 1.6% PT 1327 2.5% PT 1483 2.4%

PZ 4 0.0% PZ 59 0.1% PZ 63 0.1%

Total 9507 Total 52179 Total 61686

P – Language & Literature

Non-PDA and PDA2010-2014 Non-PDA 2010-2014 Combined 2010-2014

P 1094 11.5% P 4817 9.2% P 5911 9.6%

PA 287 3.0% PA 1171 2.2% PA 1458 2.4%

PB 23 0.2% PB 80 0.2% PB 103 0.2%

PC 58 0.6% PC 385 0.7% PC 443 0.7%

PD 14 0.1% PD 73 0.1% PD 87 0.1%

PE 310 3.3% PE 1065 2.0% PE 1375 2.2%

PF 17 0.2% PF 113 0.2% PF 130 0.2%

PG 115 1.2% PG 824 1.6% PG 939 1.5%

PH 9 0.1% PH 41 0.1% PH 50 0.1%

PI 0 0.0% PI 1570 3.0% PI 1570 2.5%

PJ 146 1.5% PJ 596 1.1% PJ 742 1.2%

PK 54 0.6% PK 1190 2.3% PK 1244 2.0%

PL 247 2.6% PL 12243 23.5% PL 12490 20.2%

PM 48 0.5% PM 2268 4.3% PM 2316 3.8%

PN 2627 27.6% PN 5405 10.4% PN 8032 13.0%

PQ 448 4.7% PQ 7263 13.9% PQ 7711 12.5%

PR 2197 23.1% PR 5624 10.8% PR 7821 12.7%

PS 1653 17.4% PS 6065 11.6% PS 7718 12.5%

PT 156 1.6% PT 1327 2.5% PT 1483 2.4%

PZ 4 0.0% PZ 59 0.1% PZ 63 0.1%

Total 9507 Total 52179 Total 61686

P – Language & LiteraturePackage Purchases

2006-2009Combined

2010-2014

Difference in Titles

Purchased

P 1167 3.6% P 5911 9.6% 4744

PA 688 2.1% PA 1458 2.4% 770

PB 52 0.2% PB 103 0.2% 51

PC 327 1.0% PC 443 0.7% 116

PD 16 0.0% PD 87 0.1% 71

PE 383 1.2% PE 1375 2.2% 992

PF 125 0.4% PF 130 0.2% 5

PG 750 2.3% PG 939 1.5% 189

PH 25 0.1% PH 50 0.1% 25

PI 0 0.0% PI 1570 2.5% 1570

PJ 2243 6.9% PJ 742 1.2% (1501)

PK 336 1.0% PK 1244 2.0% 908

PL 2247 6.9% PL 12490 20.2% 10243

PM 148 0.5% PM 2316 3.8% 2168

PN 4076 12.5% PN 8032 13.0% 3956

PQ 7614 23.3% PQ 7711 12.5% 97

PR 4924 15.1% PR 7821 12.7% 2897

PS 5620 17.2% PS 7718 12.5% 2098

PT 1907 5.8% PT 1483 2.4% (424)

PZ 22 0.1% PZ 63 0.1% 41

Total 32670 Total 61686 29016

P – Language & Literature

Winners/Losers

2006-2009Combined

2010-2014

Difference in

Titles

Purchased

Difference in %

of Collection in

Class P

P 1167 3.6% P 5911 9.6% 4744 6.0%

PE 383 1.2% PE 1375 2.2% 992 1.1%

PL 2247 6.9% PL 12490 20.2% 10243 13.4%

PJ 2243 6.9% PJ 742 1.2% (1501) -5.7%

PQ 7614 23.3% PQ 7711 12.5% 97 -10.8%

PR 4924 15.1% PR 7821 12.7% 2897 -2.4%

PS 5620 17.2% PS 7718 12.5% 2098 -4.7%

PT 1907 5.8% PT 1483 2.4% (424) -3.4%

The PDA Haboob?

• Since 2010 the selection pool has grown by 92% for PDA

• The expenditures have grown 81%

• Continue to expand formats available for PDA

Gust-front dust cloud, called a Haboob, moving across the Llano Estacado toward Yellow House Canyon near the residential community of Ransom Canyon, Texas

Questions?