1 © tefko saracevic, rutgers university evaluation of library and information services (lis): an...

© Tefko Saracevic, Rutgers University 1

Evaluation of library and information

services (LIS): an overview

Contexts

Approaches

Levels

Requirements

Measures

Tefko Saracevic, Rutgers University


Why evaluate ?

Importance of evaluation of LIS increasing, because:• Social importance of information

changing• Transition from “just-in-case” to “just-

in-time” model of service - stress on access

• Increased competition - many new players competing for resources

• Growth of electronic inf. resources & networks

Demands for justification growing by funders in practice & research


Broad context

Role that LIS play related to: SOCIETY - community, culture,

discipline ... INSTITUTIONS- universities,

organizations, companies ... INDIVIDUALS - users & potential

users (nonusers)

Roles lead to broad, but hard questions as to what context to choose for evaluation

Each context demands different criteria, measures, methodologies


Context questions

Social:• how well do LIS support inf.

demands, needs & roles of society, community?

– hardest to evaluate

Institutional:• how well do LIS support

institutional/organizational mission & objectives?

– tied to objectives of institution– also hard to evaluate

Individual:• how well do LIS support inf. needs &

activities of people?– most evaluations in this context


Approaches to evaluation

Many approaches exist• quantitative, qualitative …• effectiveness, efficiency ...• each has strong & weak points

Systems approach prevalent• Effectiveness: How well does a

system perform that for which it was designed?

• Evaluation related to objective(s)• Requires choices:

– Which objective, function to evaluate?


Approaches (cont)

Economics approach: • Efficiency: at what costs?• Cost-effectiveness: cost for a

given level of effectiveness Ethnographic approach

• practices, effects within an organization, community

• learning & using practices & comparisons


Approaches ...

Distinction between: Effectiveness:

• how well does a LIS achieve that for which it was designed?

– relates to objectives

Efficiency:• what are the costs in performing

a LIS?– relates to $$$, time, effort …

Cost effectiveness:• what are the costs for a given

level of effectiveness– relates both effectiveness &

efficiency


Levels of evaluation

System- centered:

1. Engineering: hardware & software; reliability, errors

2. Input: contents, coverage

3. Processing: procedures, techniques, algorithms

User- centered:

4. Output: search, interaction

5. Use & user: application to tasks; market; fitness-of-use

6. Social: effect on research, productivity, organization...

Danger: isolation of levels


Requirements for evaluation

Once a context is selected need to specify all five:

1. Construct• A system, process, source

– e.g. a given IR function or system; a Web site, a Dlib source

2. Criteria - to reflect objective(s) • e.g. relevance, utility, satisfaction,

accuracy, completeness, time, costs

3. Measure(s) - to reflect criteria• precision, recall, various Likert

scales, $$$, ...


Requirements …(cont.)

4. Measuring instrument - judgments by users on relevance or on a scale; cost/function

5. Methodology - procedures for collecting & analyzing data

No evaluation can proceed if not ALL of these are specified!

Sometimes specification on some are informal & implied, but they are always there.


LIS functions

When evaluating we have to consider processes/functions• Each function: different evaluation

approaches Major LIS functions:

• AVAILABILITY --acquisition of inf. materials & resources; holdings

• ORGANIZATION -- intellectual, physical

• ACCESS -- physical & intellectual

– searching, retrieval• OUTPUTS

-- dissemination, use


Availability

Social: how good coverage?• field; problem area; community

Criteria: representative, depth, breadth, up-to-date ...

Measures: degree, duplication Method: compare, survey Institutional: how well inf. resources

satisfy mission, needs, plans ... ?• education, research, work ...

Criteria: matching, attributes Method: survey, functional

comparison, e.g. curriculum


Availability (cont.)

Individual: how well users served, satisfied ?

Criteria: awareness, expectations, satisfaction, success & failure rate

Measures: scales, branching diagrams (success or failure at each point of user action)

Methods: surveys, counting & statistical analyses, probability of success• e.g. requests made/fulfilled


Organization

Processing level: How well is a collection/data base represented, organized?

Criteria: depth, breadth, type, relevance, quality, errors, time, effort, costs ...

Measures: degree, precision, recall, quality benchmarks (standards), error rate, time/process, $$$...

Methods: comparative processing, user or expert evaluation, quality analyses, economic analyses


Access Individual: How well did users

interact with a service? About users’ reactions to interaction

with system Criteria: accessibility, effort,

convenience, facilities (ease, adequacy), staff (helpfulness efficiency), frustration, errors, difficulties ...

Measures: scales, indicators Methods: surveys, interviews,

observations, experiments, transaction log analysis


Access: searching, retrieval

Individual: how well did users retrieve relevant answers?

Related to user needs, tasks• But often concentrated on system

algorithms, H-C interactions etc Criterion: relevance

• A few others proposed, e.g. satisfaction

Measures: recall, precision• Other: overlap, consistency, Likert

scales Methods: labs (TREC),

observation,


Dissemination & use

Individual: How did users perceive results of use?

Related to users’ tasks Criteria: cognitive (learning ...),

affective (satisfaction...), accomplishment (task), expectations (getting ...), time (saving, worth ...), money (cost value ...)

Measures: scales, numbers Methods: survey, interviews,

critical incidence, impact estimate


Operational & quality criteria

(Say, Seaman & Cohen)

Reliability - delivery of a LIS accurately & dependably• correct answers, relevant• consistency

Responsiveness - readiness to provide service• minimizing turnaround, time• callbacks

Assurance - knowledge, ability, courtesy of staff• understanding of collection,

technology• providing individual attention


Quality criteria (cont.)

Access - sufficiency in staff, equipment, hours of operation• waiting time• access policies; location

Communication - informing & listening; language adjustment• question negotiation• teaching users; instructing

Security - freedom from danger, risk or doubt• safety; confidentiality

Tangibles - physical facilities• building etc. condition; layouts• equipment condition


Branching method

Total requests (T)

Circulation (C)

Library function (L)

User function (U)

Satisfied requests (S)

Not acquired

In circulation

Library malfunction

User malfunction

Reasons for satisfying (or not satisfying) a known item request : success & failure analysis

Satisfaction rate (percentage) = S/T


Branching ...

T = 437

C = 399

L = 347

U = 299

S = 245

Not acq.=38

In circul.= 52

Libr. malf. = 48

User malf. = 54

Example from a study of requests for specific books from an academic library


Branching ...Calculation of perf. rates:

Satisfaction rate = 245/437 = .56 = 56%

Acquisition performance =399/437=91%i.e. library had 91 % of requested books

Circulation perf. = 347/399 = 87%13% of acquired books were in circulation

Library perf. = 299/347 = 86%

14% of books not in circulation were not found because some library malfunction

User performance = 245/299 = 82%

18% of books that were on the shelf were not found by users because of their error

Satisfaction rate (by probabilities)=

.91 (A) x .87 (C) x .86 (L) x .82 (U) = .56 or 56%


Conclusions

In practice need & importance of evaluation increasing

In research an ever present need • new systems, approaches

Essential for improvements, decisions, resource allocation

But evaluation requires:• commitment by management & staff;

hard work• financial & human resources• knowledge how to do it• continuous, not one-shot effort

If we do not evaluate others will

1 © tefko saracevic, rutgers university evaluation of library and information services (lis): an...

Documents