Collection Analysis:Collection Analysis:OverviewOverview

Sponsored by ALCTS CMDS Measures & Education Sponsored by ALCTS CMDS Measures & Education CommitteesCommittees

Peggy Johnson,Peggy Johnson,Associate University LibrarianAssociate University Librarian

University of MinnesotaUniversity of Minnesotam-john@umn.edum-john@umn.edu

““Culture of Assessment/Evaluation”Culture of Assessment/Evaluation”

Way to demonstrate Way to demonstrate • RelevanceRelevance• ValueValue• ImpactImpact

Considered from the view ofConsidered from the view of• Users Users • Stakeholders Stakeholders

Amos Lakos & Shelley Phipps – “Culture of Assessment”Amos Lakos & Shelley Phipps – “Culture of Assessment”John Crawford – “Culture of Evaluation”John Crawford – “Culture of Evaluation”

““Those who fail to move in Those who fail to move in the direction of the direction of systematic systematic assessmentassessment will be unable will be unable to cope with the to cope with the increasingly difficult increasingly difficult questions that promise to questions that promise to confront collection officers confront collection officers in years to come.”in years to come.”

Mark Sandler, Univ. of MIMark Sandler, Univ. of MI

Why do we do it?Why do we do it?

As part of good management. As part of good management. Accountability: To demonstrate to funders Accountability: To demonstrate to funders

and clients that the service is delivering and clients that the service is delivering the benefits expected when the the benefits expected when the investment was madeinvestment was made

Decision-making: To ensure that resources Decision-making: To ensure that resources are being used efficiently and effectively are being used efficiently and effectively (an internal control mechanism)(an internal control mechanism)

Marketing: To report success and Marketing: To report success and accomplishments (public relations)accomplishments (public relations)

Collection assessment assumes Collection assessment assumes that the criteria for success are that the criteria for success are defined and understood by those defined and understood by those doing the assessment and those doing the assessment and those to whom it is being reports.to whom it is being reports.

What is it?What is it?

A mechanism to determine:A mechanism to determine: If the collection is meeting its objectivesIf the collection is meeting its objectives How well it is serving its usersHow well it is serving its users In which ways or areas it is deficient, and In which ways or areas it is deficient, and

what remains to done to develop the what remains to done to develop the collectioncollection

If selectors are performing their If selectors are performing their responsibilities effectivelyresponsibilities effectively

How to allocate collections/access fundsHow to allocate collections/access funds

How is assessment different How is assessment different from evaluation?from evaluation?

EvaluationEvaluation determines how well the determines how well the collection supports the goals, needs, collection supports the goals, needs, and curriculum of the parent and curriculum of the parent organization. organization.

Assessment Assessment examines or describes examines or describes collections either in their own terms collections either in their own terms or relative to other collections and or relative to other collections and checklists. checklists.

Who is the audience ?Who is the audience ?

Accreditation agenciesAccreditation agencies Parent organization (administration, Parent organization (administration,

board, senior management)board, senior management) Library administrationLibrary administration CDM supervisorCDM supervisor SelectorSelector User community or communities User community or communities Consortial partnersConsortial partners

How can we do it well?How can we do it well?

SimpleSimple PracticalPractical RepeatableRepeatable Clear focusClear focus Understandable resultsUnderstandable results Meaningful resultsMeaningful results Results lead to actionResults lead to action

Collection-based Measures Look at:Collection-based Measures Look at:

SizeSize

GrowthGrowth

Coverage (depth, breadth, balance)Coverage (depth, breadth, balance)

Collection-based MeasuresCollection-based Measures

Checking lists, catalogs, Checking lists, catalogs,

bibliographiesbibliographies

Evaluating the collection directlyEvaluating the collection directly

Compiling comparative statisticsCompiling comparative statistics

Application of collection standardsApplication of collection standards

Use- and User-based Measures Look at:Use- and User-based Measures Look at:

Who is using the collection?Who is using the collection?

How often?How often?

What are users’ expectations?What are users’ expectations?

What are user’s needs?What are user’s needs?

What are their perceptions?What are their perceptions?

Use- & User-based MeasuresUse- & User-based Measures

Circulation studiesCirculation studies

In-house use studiesIn-house use studies

Survey of usersSurvey of users

Shelf availability studiesShelf availability studies

Analysis of online usage of electronic resourcesAnalysis of online usage of electronic resources

Analysis of ILL statisticsAnalysis of ILL statistics

Citation studiesCitation studies

Document delivery testsDocument delivery tests

Cost-per-use Cost-per-use

Quantitative MeasuresQuantitative Measures

Count thingsCount things• UseUse• ExpendituresExpenditures• TitlesTitles• Physical itemsPhysical items

Quantitative MeasuresQuantitative Measures

TitlesTitles

Circulation transactionsCirculation transactions

ExpendituresExpenditures

E-metricsE-metrics

ILL transactionsILL transactions

Ratios (monographs/serials; volumes/students; Ratios (monographs/serials; volumes/students;

expenditures/degree programs; electronic/print)expenditures/degree programs; electronic/print)

E-MetricsE-Metrics

Online sessionsOnline sessions Documents downloadedDocuments downloaded Records downloadedRecords downloaded Virtual visitsVirtual visits Turn-awaysTurn-aways Alert usageAlert usage Personal profile usersPersonal profile users Remote versus onsite usageRemote versus onsite usage

Qualitative ResearchQualitative Research

““A process of inquiry that draws data A process of inquiry that draws data from the context in which events from the context in which events occurs . . . using induction to derive occurs . . . using induction to derive possible explanations based on possible explanations based on observed phenomena”observed phenomena”

Gorman and Clayton, Gorman and Clayton, Qualitative Research for the Qualitative Research for the Information Professional: A Practical HandbookInformation Professional: A Practical Handbook, , 22ndnd ed. (London: Facet, 2005) ed. (London: Facet, 2005)

Qualitative Measures Look at:Qualitative Measures Look at:

StrengthsStrengths

WeaknessesWeaknesses

Non-strengthsNon-strengths

Qualitative MeasuresQualitative Measures

Provide the contextProvide the context Offer a way to understand the Offer a way to understand the

attitudes that inform the statisticsattitudes that inform the statistics

Quantitative MeasuresQuantitative Measures

Focus groupsFocus groups Online or printed surveysOnline or printed surveys Interviews (structured or Interviews (structured or

unstructured) unstructured) ObservationObservation

Collection Analysis MethodsCollection Analysis MethodsUse- or User-basedUse- or User-based Collection-basedCollection-based

QuantitativeQuantitative •ILL statisticsILL statistics•Circulation statisticsCirculation statistics•In-house use statisticsIn-house use statistics•Document delivery Document delivery statisticsstatistics•Shelf availability Shelf availability statisticsstatistics•E-metricsE-metrics

•Collection size and Collection size and growthgrowth•Materials budget size and Materials budget size and growthgrowth•Collection size standards Collection size standards and formulasand formulas•Expenditures by subjectExpenditures by subject•RatiosRatios

QualitativeQualitative User opinion surveysUser opinion surveysUser observationUser observationFocus groupsFocus groupsInterviewsInterviews

List checkingList checkingVerification studiesVerification studiesCitation analysisCitation analysisDirect collection checkingDirect collection checkingCollection mapping Collection mapping (assigning conspectus (assigning conspectus levels)levels)

Steps in a Collection Analysis Steps in a Collection Analysis ProjectProject

Do

StudyAct

Plan

Where to start?Where to start?

Define the question or problemDefine the question or problem Determine metrics to useDetermine metrics to use Decide:Decide:

• Where to locate the informationWhere to locate the information• Who will collect the informationWho will collect the information• Who will analyze and report the Who will analyze and report the

informationinformation• Who will act on the informationWho will act on the information

RememberRemember

Chose measures that matterChose measures that matter Chose an approach that is simpleChose an approach that is simple Don’t aim for perfection—good ‘nuff Don’t aim for perfection—good ‘nuff

is OKis OK Don’t do it once and never againDon’t do it once and never again Know your audienceKnow your audience Present data in a context—explain Present data in a context—explain

what it meanswhat it means