błażej feret the main libray of thetechnical university of Łódź, poland...

Błażej FeretThe Main Libray of theTechnical University of Łódź, [email protected]

Marzena MarcinekCracow University of Technology Library, [email protected]

Library performance indicators Library performance indicators

does it really make sense to measure them?does it really make sense to measure them?

CASLIN Seminar, June 2006

Data collection and analysis Data collection and analysis of library performance of library performance

a case study for Polish research libraries

22

Ackowledgements

Special thanks go for Mrs Lidia Derfert-Wolf,a member of the Task Group for Standardisation for Polish Research Libraries, for her kind assistance and advice.

33

Plan of the workshopPlan of the workshop1. Introduction 15 minutes

2. A few general remarks on library quality and performance measurement

Quality criteria from different perspectives 10 minutes

3. A Common Project of Polish Research Libraries on Comparable Measures (incl. examples) 40 minutes

Czech realities 20 minutes

4. Standardised terminology and descriptions of library performance indicators based on the ISO 11620 Standard Information and documentation – Library performance indicators

20 minutes

1. Conclusions 15 minutes

44

A few general remarks A few general remarks

on library quality and performance on library quality and performance indicatorsindicators

55

Quality:Quality: fitness for purposefitness for purpose fitness for usefitness for use conformity to requirementsconformity to requirements absence of defectsabsence of defects …………

Quality of librariesQuality of libraries (ISO 11620): (ISO 11620): totality of features and characteristics of a product or totality of features and characteristics of a product or

services that bear on the library's ability to satisfy stated services that bear on the library's ability to satisfy stated or implied needsor implied needs

66

Quality of Quality of researchresearch libraries libraries from the perspective of:from the perspective of:

UsersFunding bodies

LibrariansPlease consider all these perspectives

77

Model of measuring quality of librariesModel of measuring quality of libraries

inputsinputs - - the raw data such as the raw data such as finance, collection, finance, collection, equipment, users and staff, equipment, users and staff, space, seatsspace, seats

outputs outputs - - the work done, i.e. the work done, i.e. circulation, cataloguing, circulation, cataloguing, reference services, reference services, preservation, interlibrary preservation, interlibrary lending, facilities usage and lending, facilities usage and e-sources searchese-sources searches

outcomes outcomes – user satisfaction – user satisfaction and the and the impact of library impact of library services on users at the local services on users at the local institution and society institution and society

quantitative data

qualitative data

from the perspective of:

users

funding bodies

librarians

group work

88

Library quality assessmentLibrary quality assessment- w- what is required?hat is required?

set of quality criteria set of quality criteria librarlibrary’y’s goals and s goals and objectivesobjectives

set of performance indicatorsset of performance indicators national library statistics systemnational library statistics system standardisation on local, national and standardisation on local, national and

interninternaational leveltional level

but ...but ...

99

““The level of quality cannot be defined once and for all, since both The level of quality cannot be defined once and for all, since both the the criteriacriteria and and evaluation methodsevaluation methods, as well as the , as well as the assessment of the results achievedassessment of the results achieved, may , may changechange. .

This is caused by various factors, for example technological, This is caused by various factors, for example technological, political, economic, as well as the ones connected with the political, economic, as well as the ones connected with the community in which and for whom information services work. It is community in which and for whom information services work. It is crucial that these varying criteria and methods, as well as the crucial that these varying criteria and methods, as well as the dissimilarity in the level of the quality achieved by a given dissimilarity in the level of the quality achieved by a given country, are taken into account when aiming at adopting the country, are taken into account when aiming at adopting the international standards on the quality of information work. international standards on the quality of information work.

Countries differ in living standards they achieve, in the level of Countries differ in living standards they achieve, in the level of education, and the way they introduce innovations. education, and the way they introduce innovations. Setting the Setting the goal is of fundamental importance, since its goal is of fundamental importance, since its accomplishment or failure to reach it may be treated as a accomplishment or failure to reach it may be treated as a basis for quality assessmentbasis for quality assessment”.”.

W. PindlowaW. Pindlowa

ActivitiesActivities conducted by conducted by the the Group for Standardisation Group for Standardisation for Polish Research Librariesfor Polish Research Libraries,,

results results and and plansplans for the future for the future

CASLIN Seminar, June 2006

1111

The Group for Standardisation for Polish Research Libraries coordinated by the Adam Mickiewicz University in Poznań

Lidia Derfert-Wolf (ATR, Bydgoszcz)

Ewa Dobrzyńska-Lankosz (AGH, Krakow)

Mirosław Górny (Adam Mickiewicz University, Poznan)

Elżbieta Górska (Warsaw Public Library)

Marek Górski (Cracow University of Technology)

Artur Jazdon (Adam Mickiewicz University, Poznan)

Dariusz Pawelec (Silesian University, Katowice)

Anna Sokołowska-Gogut (Academy of Economics, Krakow)

Teresa Wildhardt (Pedagogical University, Krakow)

1212

Objectives

to gather libraries' statistical data for a computer database

to select a set of performance indicators and standards for library performance (quantity, quality and effectiveness)

to conduct a comparative research

to prepare and publish yearly reports

to define methods for the assessment of Polish research libraries

1313

Project for the Analysis of Polish Research Libraries

1414

Some facts as of May 2006

• Libraries registered – 5954 academic libraries (42 state-owned and 11 non state-owned)

3 public libraries 2 special libraries

• Questionnaires for 2004 completed 31 libraries • Questionnaires for 2003 completed 29 libraries

• Questionnaires for 2002 completed 17 libraries

1515

Software for the collection and analysis of data - general requirements

on-line access to the questionnaire (submission, modification)

selected performance indicators automatically calculated and presented

automatic control and verification of the accuracy of submitted data

access to analysing functions for individual libraries

possibility to conduct a multi-aspect comparative analysis of selected data and performance indicators

1616

Web-based Application

for:

• General users• Library directors• Analysts• Administrators

1717

General userGeneral user

General information about the Project Instructions

• Registration• Rules for filling up questionnaires• Data analyses

Questionnaires (patterns in doc. format) Example results Useful linksUseful links

1818

http://ssk2.bu.amu.edu.pl/standaryzacja/index.htm

General information

Instructions

Questionnaires

Results

Useful links

handouts 1

2020

Library DirectorLibrary Director

Filling up questionnaires; Analysis of the own library data sorted by

years, question categories or indicators; Access to calculated automatically indicators Comparison of the own library data with

average results of other libraries in the same category or in the country (data published in the form of tables).

Establishment and modification of access rights for the library staff

2121

Login

2222

For directors of registered libraries

UsersQuestionnaires

Analyses

Send e-mail

Change password

Logout

2323

Questionnaire

48 questions of various types refer to easily accessible or computable data (e.g. size of collection, number of users etc.) closed questions about the services offered (e.g. on-line reservation: Yes/No)

88 performance indicators 19 calculated by librarians 69 calculated automatically

handouts 2

2424

Questionnaire

• addresses all the elements of a library system, its environment, library processes and services• divided into chapters:

•Staff•Collection•Budget•Infrastructure•Circulation•Information services •Didactics•Publications and data bases created by the library•Library cooperation•Organisation of library events•Professional activity of library staff

,

2525

Examples of performance indicators required to complete the questionnaire

library’s total expenditures

expenditures for library materials/books

ratio of library budget to the budget of its parent university

time required for the technical processing of a document

collection on the computer system as a % of the whole collection of the library

percent of catalogue descriptions acquired from outside resources

2626

Examples of performance indicators

calculated automatically

library expenditure per student/user

expenditures for library materials/books per student/user

library registered users as a percent of potental users

space of the library per user

collection on the computer system as a % of the whole collection of the library

number of user training hours per one staff member

2727

Survey formNo of libraries in the university library and information system (incl. the main library)

For university libraries

data concern main library

university library and information systemStaff

No of library staffcommentary

categories

education

age

2828

Collection

Acquisition

2929

Budget

3030

Infrastructure

Circulation

3232

Standard lending period

Information services

Didactics

Library’s own publications and databases

Electronic sources usage

3333

Library’s own publications and databases

Interlibrary cooperation, staff professional activities

3434

Selected performance indicators

3535

Once the questionnaire is validated the indicators are calcullated automaticly

Indicators calculated automatically

Admin can add other indicators based on the data gathered - no action by director required

3636

Adding and modifying users

3737

values

statistical data

Director can analyse performance of his/her own library for selected years

3838

Tyle kategorii ile pytan i wskaznikow wpisanych i automatycznych

categories for the analysis

Number of categories = number of data and indicators (entered and calculated automatically)

3939

Total library space per student

No of loans per staff membber

No of loans per registered user

4040

No of years analysed: 3

Total library space per student

No of loans per one registered user

No of loans per staff member

4141

Analysis

Questionnaires

•Access to all the questionnairesAccess to all the questionnaires•No modification rightsNo modification rights

AnalysAnalystt

4242

Selection of the library categories and years

Type: academic

Other types

Source of funding

Analyse the years

Sort by (years, types)

4343

180 categories for the analysis found, please select

Staff

Collection

4444

Indicators calculated automaticaly

Total library space per one user


4545

Total library space per user


Research libraries performance in 2003

4646


No of loans per staff member

Research libraries performance in 2004

4747

University libraries University of technologylibrariesPerformance indicators

2002 2003 2004 2002 2003 2004Cost per user in PLN 191,2 182,1 188,6 157,8 171,3 181,1

Acquisition cost per user in PLN 45,6 53,8 52,1 56,6 56,8 54,9

Library budget as % of institutional budget 4,81 4,75 4,23 2,57 2,59 2,55

Registered users as % of potential users 73,8 70,6 70,9 72,2 65,9 68,1

Total books per user 21,8 20 19,9 11,4 12,4 12,8

Books added per user 0,2 0,26 0,27 0,2 0,18 0,19

Loans per registered user 7,7 8,9 8,7 7,9 7,9 8,4

Loans per library staff member 2038 2447 2550 2905 2215 2421

Users per library staff member 341 364 367 451 432 424

Total library space per user 0,34 0,26 0,25 0,18 0,18 0,18

Users per seat 116 109 113 86 95 92

Open access printed books as % of totalprinted books

8,7 10,7 11,3 11,1 9,6 9,5

User services staff as % of total staff 48,9 54 54,2 57,9 58,6 58,3

Staff with higher LIS education as % oftotal staff

34,8 43,6 44 50 51,5 52,9

Time of document acquisition andprocessing in days

16 37,6 32 14,4 16 17

4848

Administartor’s Module

Libraries

logoutDatabase statistics

Change password

Send e-mail

Outstanding questionnaires

Forms

Questionnaires

Users

4949

Registration form

Send via snail mail to….

5050

Sortowanie

Add / modify library

Sort by

Source of funding

Type

Name

5151

Select the library

5252

List of Libraries with outstanding questionnaires

missingnot validated

Questionnaires

validated by directorValidated by administrator

5353

The analysis of data – examples(no complete data for 2005 yet)

5454

University libraries University of technologylibrariesPerformance indicators

2002 2003 2004 2002 2003 2004Cost per user in PLN 191,2 182,1 188,6 157,8 171,3 181,1

Acquisition cost per user in PLN 45,6 53,8 52,1 56,6 56,8 54,9

Library budget as % of institutional budget 4,81 4,75 4,23 2,57 2,59 2,55

Registered users as % of potential users 73,8 70,6 70,9 72,2 65,9 68,1

Total books per user 21,8 20 19,9 11,4 12,4 12,8

Books added per user 0,2 0,26 0,27 0,2 0,18 0,19

Loans per registered user 7,7 8,9 8,7 7,9 7,9 8,4

Loans per library staff member 2038 2447 2550 2905 2215 2421

Users per library staff member 341 364 367 451 432 424

Total library space per user 0,34 0,26 0,25 0,18 0,18 0,18

Users per seat 116 109 113 86 95 92

Open access printed books as % of totalprinted books

8,7 10,7 11,3 11,1 9,6 9,5

User services staff as % of total staff 48,9 54 54,2 57,9 58,6 58,3

Staff with higher LIS education as % oftotal staff

34,8 43,6 44 50 51,5 52,9

Time of document acquisition andprocessing in days

16 37,6 32 14,4 16 17

5555

University Libraries - 2002

28,05

44,72

3,49

12,09

11,65

Collection

Staf f expenditures

Automation

Premises

Others expenditure

University Libraries - 2003

28,41

48,56

2,49

8,14

12,4

Collection

Staf f expenditures

Automation

Premises

Others expenditure

All examined academic libraries - 2003

29,15

50,7

2,33

6,11

11,73

Collection

Staff expenditures

Automation

Premises

Others expenditure

All examined academic libraries - 2002

30,42

45,47

2,85

7,72

13,54

Collection

Staf f expenditures

Automation

Premises

Others expenditure

Technical Univ. Libraries - 2002

34,45

45,43

2,73

5,38

12,02

Collection

Staf f expenditures

Automation

Premises

Others expenditure

Technical Univ. Libraries - 2003

31,63

48,58

2,37

6,35

11,09

Collection

Staf f expenditures

Automation

Premises

Others expenditure

5656

28,05 28,4

44,7248,56

10,36,96

3,49 2,49 3,44 3,27

10 10,32

0

5

10

15

20

25

30

35

40

45

50

Structure of budget in state university libraries in the years 2002 and 2003

collectionsalaries

spacehardware &software

administration

other

5757

34,4431,63

45,4348,58

4,845,68

2,73 2,37 2,44 2,65

10,129,09

0

5

10

15

20

25

30

35

40

45

50

Structure of budget in state technical universities libraries in the years 2002 and 2003

collection

spacehardware &software

administration

othersalaries

5858

17,47

31,45

11,44 12,45

21,8420,03

0

5

10

15

20

25

30

35

uniwersytetypolitechnikiwszystkieakademickie

Liczba vol. książek na 1 użytkownika 2003

2002

Selected indicators for state academic libraries

No of books per one user [vols]

all academiclibraries

technical universities universities

5959

0,250,19 0,2 0,18 0,2

0,26

0

0,1

0,2

0,3

0,4

0,5

0,6

0,7

0,8

0,9

1


Liczba vol. nabytków (książki) na 1 użytkownika

2003

2002


No of new documents (books) per user [vols.]

6060

182,1

191,2171,3

157,8

180,3

195,5

0

20

40

60

80

100

120

140

160

180

200


Wydatki na 1 użytkownika (w PLN) 2003

2002

53,8

45,6

56,8

56,6

55,7

57,4

0

20

40

60

80

100

120

140

160

180

200


Wydatki na zakup zbiorów na 1 użytkownika (w PLN)


Expenditures per user Expenditures for new collection per user

All academic libraries

Technical universities

universities

All academic libraries

Technical universities

Universities

6161

Politechniki

2,57

Uniwersytety

4,81

Wszystkie akademickie

3,87

Wszystkie akademickie

4,81

Politechniki

2,59

Uniwersytety

4,75

rok 2003rok 2003


Budżet biblioteki jako procent budżetu uczelni - rok 2002rok 2002

Library budget as percent of parent university budget

6262

30,87

40,16

48,149,43

39,9643,32

0

5

10

15

20

25

30

35

40

45

50

Uniwersytety Politechniki Wszystkieakademickie

2002

2003


Pracownicy z wyższym wykształceniem bibliotekarskim jako odsetek pracowników działalności podstawowej

Staff with MA degree in librarianship as a percent of the whole library staff

6363

Feedback loopFeedback loop

6464

• lack of statistical data required to complete the questionnaire

• lack of comparable data on the use of electronic resources (incl. differences in usage statistics generated by various providers)

• differences in library structure and budgeting within university

• difficulties with validation – mistakes (e.g. wrong ratio) need correction, misunderstanding of data requirements, wrong interpretation of questions

Problems arisen when receiving data

6565

product improvementstestingproblems noticed by librarians and analysts

• Verivfication of data required• Verification of indicators• Software tools to control data in the fields • Detailed commentaries

...and solving them

6666

Selection of performance indicators –

the reasons for measurement

What? Why? At what cost?

• Part of reporting mechanism (statistics)• Part of internal assessment• Support for decision making (locally, within parent organisation, at the national level)• External requirements

6767

Library performance indicators

in the ISO 11620:1998 Standard

6868

The purpose of library performance indicatorsThe purpose of library performance indicators

• To function as tools to assess the quality and effectiveness of activities and services provided by a library

• To assess the efficiency of resources allocated by the library to its activities and services

6969

Required features of performance indicatorsRequired features of performance indicators

• Informative content

• Reliability

• Validity

• Appropriateness

• Practicality

• Comparability

7070

Informative content

The indicator has to be informative as a tool for The indicator has to be informative as a tool for measuring activity, for identyfying achievements measuring activity, for identyfying achievements and problems in the performance of the library so and problems in the performance of the library so that action can be taken to remedy this. that action can be taken to remedy this.

It should provide information for decision-making, It should provide information for decision-making, e.g. goalsetting, budget allocation, prioritizing e.g. goalsetting, budget allocation, prioritizing services and activities.services and activities.

7171

Reliability

A performance indicator should be A performance indicator should be reliable in the sense that it reliable in the sense that it consistently produces the same consistently produces the same results when used repeatedly under results when used repeatedly under the same circumstances.the same circumstances.

7272

Validity

A performance indicator must be valid, A performance indicator must be valid, that is, it must measure what it is that is, it must measure what it is intended to measure.intended to measure.

7373

Appropriateness

A performance indicator must be A performance indicator must be appropriate for the purpose to which appropriate for the purpose to which it is to be put. That is, the units and it is to be put. That is, the units and scale must be suitable and the scale must be suitable and the operations necessary to implement operations necessary to implement the process of measurement should the process of measurement should be compatible with the library’s be compatible with the library’s procedures, physical layout, etc.procedures, physical layout, etc.

7474

Practicality

A performance indicator has to be A performance indicator has to be practical in the sense that it uses practical in the sense that it uses data that can be made available by data that can be made available by the library with a resonable amount the library with a resonable amount of effort in terms of staff time, staff of effort in terms of staff time, staff qualifications, operational costs and qualifications, operational costs and user’s time and patience.user’s time and patience.

7575

Comparability

A performance indicator allows A performance indicator allows comparison between libraries if the comparison between libraries if the same score, making allowance for same score, making allowance for the accuracy of the score, means the the accuracy of the score, means the same level of quality of services or same level of quality of services or the same level of efficiency in the the same level of efficiency in the libraries to be compared.libraries to be compared.

7676

Some definitions – group work

•Performance•Indicator•Evaluation•Accessability

•Effectiveness•Efficiency•Reliability•Validity

•Loan•Recurrent expenditure•Resources•Target population

Handouts 3 and 4 afterwards

7777

Performance indicators – a descriptive frameworkPerformance indicators – a descriptive framework

Name Objective Scope Definition Method Interpretation Factors affecting the indicator Related indicators

Handout 5

7878

Uses of performance indicatorsUses of performance indicators

The quality and effectiveness of the services of the library The quality and effectiveness of the services of the library as well as the efficiency of the uses of the resources are as well as the efficiency of the uses of the resources are evaluated against the mission, goal and objectives of the evaluated against the mission, goal and objectives of the library itself.library itself.

Performance indicators should be linked to systematic Performance indicators should be linked to systematic library planning and evaluation.library planning and evaluation.

Indicators are useful for comparison over time within the same library.

Comparison between libraries is possible but careful interpretation is required.

7979

Comparability of performance indicator data

Purposes of using library performance indicators:

• self-diagnosis (within the same library) e.g. comparisons of one year’s performance with another;

•comparison accross different libraries in full recognition of the limitations of such comparisons and with respect for each library’s:

•mission, goals and objectives•resources•user groups•governance / funding structure•procedures

8080

LimitationsLimitations

It is impossible to achieve optimum scores simultaneously on all It is impossible to achieve optimum scores simultaneously on all performance indicators (eg. user satisfaction vs. expenditure performance indicators (eg. user satisfaction vs. expenditure per user)per user)

„„The scores on performance indicators must be interpreted in the The scores on performance indicators must be interpreted in the light of what the library intends to accomplish, not simply in light of what the library intends to accomplish, not simply in terms of optimizing scores on particular indicators.”terms of optimizing scores on particular indicators.”

Degree of accuracy Degree of accuracy • Sampling errorsSampling errors• Subjective aspects of the measuring processSubjective aspects of the measuring process• Inadequate time or resource for measuring processInadequate time or resource for measuring process

Users skills vs. libraUsers skills vs. librarry performance perceptiony performance perception Linking resources to services, management approaches, staff Linking resources to services, management approaches, staff

skills etc.skills etc.

8181

What factors would you consider when making decision for registering your

library to the library project for comparable measures?

- group work

8282

• Relevancy to to the services being analysed• Strategic purpose of the measurement activity• Staff involved in the measurement process and their awarness of the fact that measurment process is part of their normal flow of work

• Accuracy• Reliability • Consistency over time• Practical aspect

Consider

8383

SuccessSuccess conditions conditions

Simplicity versus complexity (goals!) Simplicity versus complexity (goals!) The effort involved in data collection versus the The effort involved in data collection versus the

expected outcomesexpected outcomes Possibility to integrate data collection with the Possibility to integrate data collection with the

already carried workalready carried work Trends are (usually) more important than figuresTrends are (usually) more important than figures For some measures (e.g. time taken to handle For some measures (e.g. time taken to handle

information queries) setting aside certain periods information queries) setting aside certain periods throughout the year and extrapolatthroughout the year and extrapolatiing findings to ng findings to a full yeara full year

Representative samplesRepresentative samples Do not collect data for the sake of itDo not collect data for the sake of it

8484

Thank you!

błażej feret the main libray of thetechnical university of Łódź, poland...

Documents