Download - PM7Workshop_de Jager en Self
-
7/29/2019 PM7Workshop_de Jager en Self
1/42
Measures that matter:
Designing and developing yourown Balanced Scorecard
Karin de J ager & J im Self
7th Northumbria International Conference
on Performance Measurement
in Libraries and Information Services
An IFLA Satellite Conference
-
7/29/2019 PM7Workshop_de Jager en Self
2/42
August 2007 7th Northumbria Conference, Spier,South Africa 2
Programme:
14h00 14h40:
Introduction to the Balanced Scorecard. The South Africanenvironment. Implementing a Scorecard with measures thatmatter. Karin de J ager
14h40 - 15h20:The Balanced Scorecard in practice: how it works and what it cantell us. Problems, pitfalls & solutions.J im Self
15h20 16h40: Small group work:Develop the outline of a Balanced Scorecard for a particularinstitution. Specify three metrics (with targets) for the four
quadrants. 16h40 17h30:
Reportback session with comment & feedback from speakers &participants.
-
7/29/2019 PM7Workshop_de Jager en Self
3/42
August 2007 7th Northumbria Conference, Spier,South Africa 3
Measures that matter in SA: an
Introduction Higher Education Evaluation in SA
Dealing with too much data
Assessing Quality Different Perspectives
The Balanced Scorecard
Measures & Targets Components of a Scorecard
-
7/29/2019 PM7Workshop_de Jager en Self
4/42
August 2007 7th Northumbria Conference, Spier,South Africa 4
Higher Education Evaluation
2004 HEQC responsible for Quality Assurance
Quality Audits mandated
CHELSA established own QA Subcommittee
Two documents: Measures for Quality; and a Guideto the Self Review of University Libraries, to assist
Large number of possible indicators. but not
prescriptive Individual libraries to select most appropriate for
own circumstances
-
7/29/2019 PM7Workshop_de Jager en Self
5/42
August 2007 7th Northumbria Conference, Spier,South Africa 5
Data, data everywhere
Over-complexity often a cause for failure
Useful information can drown in too muchdetail
Collecting and sorting through too muchinformation can become more trouble than itis worth
One way of attempting to manage this delugeof data, is to focus on only a small number ofmeaningful indicators
-
7/29/2019 PM7Workshop_de Jager en Self
6/42
August 2007 7th Northumbria Conference, Spier,South Africa 6
What is Quality?
Fitness for purpose
Never static or finally achieved
Accurate measurement essential If something cant be measured, it cant be
understood. If it cant be understood, it cant
be controlled; if it cant be controlled, it cant
be improved (Harrington)
-
7/29/2019 PM7Workshop_de Jager en Self
7/42
August 2007 7th Northumbria Conference, Spier,South Africa 7
Quality in libraries seen from
different points of view
The consumers of library services: Diverse in nature: from first year to advanced
research
Library workers: Have to be competent in wide range of services
Institutional funding authorities:
Objectives addressed & value for money? Overall strategic perspective: Ensure fitness for purpose into the future
-
7/29/2019 PM7Workshop_de Jager en Self
8/42
August 2007 7th Northumbria Conference, Spier,South Africa 8
Balanced Scorecard:
One approach
Commercial management tool
Defines organization's long-term strategy
Specific, measurable goals
From different perspectives
Attempts to integrate large amounts of diversedata into a single system
-
7/29/2019 PM7Workshop_de Jager en Self
9/42
August 2007 7th Northumbria Conference, Spier,South Africa 9
Scorecard perspectives
User perspective: "how well is the library meetingthe needs or expectations of its users?
Finance perspective: "how well are the library'sfinances managed to achieve its mission?"
Internal process perspective: "how do the library'sinternal processes function to deliver librarycollections and services efficiently?"
Future, learning or growth perspective: "how wellis the library positioned to ensure that goals are metin the future?"
-
7/29/2019 PM7Workshop_de Jager en Self
10/42
August 2007 7th Northumbria Conference, Spier,South Africa 10
Balanced Scorecard illustrated
Financeperspective
User
perspective
Processperspective
Future/Learning
perspective
-
7/29/2019 PM7Workshop_de Jager en Self
11/42
August 2007 7th Northumbria Conference, Spier,South Africa 11
Vision at the centre
E.g. a First Class African research andteaching university
What does this mean?
First class? Benchmark internationally
African?
How African are we really, and how can wedemonstrate & measure this?
Evidence of support for research & teaching?
-
7/29/2019 PM7Workshop_de Jager en Self
12/42
August 2007 7th Northumbria Conference, Spier,South Africa 12
Indicators must be measurable
Surveys Users, e.g. LibQUAL+
Availability
Financial expenditure Totals or per unit
Time expended E.g. on acquisitions, cataloguing, ILL
Counts and tallies of processes and services
Targets to see how well one is doing
-
7/29/2019 PM7Workshop_de Jager en Self
13/42
August 2007 7th Northumbria Conference, Spier,South Africa 13
Constructing a Balanced
Scorecard
Select a limited number of meaningfulindicators for each quadrant
Select targets for each indicator
Four quadrants:
User perspective
Internal processes perspective
Finance perspective
Future/growth perspective
-
7/29/2019 PM7Workshop_de Jager en Self
14/42
August 2007 7th Northumbria Conference, Spier,South Africa
14
User Perspective (1)
User satisfaction
LibQual+ comparative data
satisfaction with the overall quality of the service
provided by the library
Adequate support for teaching & learning(HEQC)
Information literacy activities? How many classes or students are taught at
present?
-
7/29/2019 PM7Workshop_de Jager en Self
15/42
August 2007 7th Northumbria Conference, Spier,South Africa
15
User Perspective (2)
Adequate support for research?
ILL provides research materials for postgraduatestudents & academics
Extent of use:
Increase in use ofelectronic resources (increasein full-text downloads, or downloads per capita)
Access to library portal or web page, Loans per student?
Market penetration?
-
7/29/2019 PM7Workshop_de Jager en Self
16/42
August 2007 7th Northumbria Conference, Spier,South Africa
16
Internal Processes Perspective
Efficiency of library processes:
Turnaround time how long does it take to order;process; shelve?
Ease of accessing resources?
Usability testing?
Staff ratings of own internal services?
Internal surveys?
What do YOU want to manage & improve?
-
7/29/2019 PM7Workshop_de Jager en Self
17/42
August 2007 7th Northumbria Conference, Spier,South Africa
17
Financial Perspective
Library budget as % of institutional budget?
Unit costs & ratios, e.g.
Library expenditure per student? Books purchased per student?
Spend on e-resources as % of acquisitions spend?
Fundraising?
Set target amounts
-
7/29/2019 PM7Workshop_de Jager en Self
18/42
August 2007 7th Northumbria Conference, Spier,South Africa
18
Future/Growth Perspective
Staff development? Staff participation: further education;
Staff participation: in-service training
Staff participation: conferences, workshops Continuous alignment with institutional vision? African materials purchased?
Investment in infrastructure Replacement rate for library computers?
% of library computers/copiers/printers out ofaction?
-
7/29/2019 PM7Workshop_de Jager en Self
19/42
August 2007 7th Northumbria Conference, Spier,South Africa
19
Importance of Targets
Measure quantitatively
Set high but achievable targets
Consider two sets of targets: Complete success
Partial success
Aggregate regularly to provide feedback
Address problems that are revealed
-
7/29/2019 PM7Workshop_de Jager en Self
20/42
August 2007 7th Northumbria Conference, Spier,South Africa
20
Targets at a glanceUser Perspective
40%
40%
20%
Target met
Partial success
Target not met
Internal Perspective
34%
33%
33%
Target met
Partial success
Target not met
Finance Perspective
25%
50%
25%
Target met
Partial success
Target not met
Future/Growth Perspective
25%
25%
50%
Target met
Partial success
Target not met
-
7/29/2019 PM7Workshop_de Jager en Self
21/42
August 2007 7th Northumbria Conference, Spier,South Africa
21
The Balanced Scorecard in
Practice
Jim Self
Director, MIS
University of Virginia Library
-
7/29/2019 PM7Workshop_de Jager en Self
22/42
August 2007 7th Northumbria Conference, Spier,South Africa
22
The BSC at the U.Va. Library
Implemented in 2001
Results tallied FY02 through FY06
Completing metrics for FY08 Tallying results for FY07
A work in progress
-
7/29/2019 PM7Workshop_de Jager en Self
23/42
August 2007 7th Northumbria Conference, Spier,South Africa
23
Choosing the Metrics
--Reflecting Values
What is important?
What are we trying to accomplish?
-
7/29/2019 PM7Workshop_de Jager en Self
24/42
August 2007 7th Northumbria Conference, Spier,South Africa
24
Choosing the Metrics
--Diversity and Balance
Innovations and operations
Variety of measurements
-
7/29/2019 PM7Workshop_de Jager en Self
25/42
August 2007 7th Northumbria Conference, Spier,South Africa
25
Choosing the Metrics
--Ensuring validity
Does the measurement accurately
reflect the reality?
-
7/29/2019 PM7Workshop_de Jager en Self
26/42
August 2007 7th Northumbria Conference, Spier,South Africa
26
Choosing the Metrics
--Being Practical
Use existing measures when possible
Use sampling
Collect data centrally Minimize work by front line
-
7/29/2019 PM7Workshop_de Jager en Self
27/42
August 2007 7th Northumbria Conference, Spier,South Africa
27
What Do We Measure at U.Va.?
Customer survey ratings
Staff survey ratings
Timeliness and cost of service
Usability testing of web resources
Success in fund raising
Comparisons with peers
-
7/29/2019 PM7Workshop_de Jager en Self
28/42
August 2007 7th Northumbria Conference, Spier,South Africa
28
A closer look
Impact of the Scorecard
-
7/29/2019 PM7Workshop_de Jager en Self
29/42
August 2007 7th Northumbria Conference, Spier,
South Africa29
Process improvementIdentifying and dissecting shortfalls
-
7/29/2019 PM7Workshop_de Jager en Self
30/42
August 2007 7th Northumbria Conference, Spier,
South Africa
30
Metric U.4.B: Turnaround time for
user requests
Target1: 75% of user requests for new booksshould be filled within 7 days.
Target2: 50% of user requests for new booksshould be filled within 7 days.
Result FY06: Target1.
79% filled within 7 days.
-
7/29/2019 PM7Workshop_de Jager en Self
31/42
August 2007 7th Northumbria Conference, Spier,
South Africa
31
Metric VII: Turnaround time for
user requests
Target1: 90% of user requests for new booksshould be filled within 7 days.
Target2: 80% of user requests for new booksshould be filled within 7 days.
Result FY03: Target not met.
17% filled within 7 days.
-
7/29/2019 PM7Workshop_de Jager en Self
32/42
August 2007 7th Northumbria Conference, Spier,
South Africa
32
Accountability
Institutionalizing customer feedback
-
7/29/2019 PM7Workshop_de Jager en Self
33/42
August 2007 7th Northumbria Conference, Spier,
South Africa
33
Metric U.1.A: Overall rating in
student and faculty surveys
Target1: An average score of at least 4.00(out of 5.00) from each of the majorconstituencies.
Target2: A score of at least 3.90.
FY06 Result: Target1 Graduate students 4.08
Undergraduates 4.11
-
7/29/2019 PM7Workshop_de Jager en Self
34/42
August 2007 7th Northumbria Conference, Spier,
South Africa
34
Metric I.3.B.Staff Survey Rating
of Internal Customer Service
Target1: A composite rating of at least 4.00,with no unit rated below 3.50.
Target2: A composite rating of 3.50, with nounit below 3.00.
Result FY05: Target1.
Composite score of 4.09. Individual unit ranged from 3.52to 4.57.
-
7/29/2019 PM7Workshop_de Jager en Self
35/42
August 2007 7th Northumbria Conference, Spier,
South Africa
35
Examining priorities
-
7/29/2019 PM7Workshop_de Jager en Self
36/42
August 2007 7th Northumbria Conference, Spier,
South Africa
36
Metric F.2.A: Unit Cost of
Electronic Serial Use
Target1: There should be no increase in unitcost each year.
Target2: Less than 5% annual increase inunit cost.
Result FY03-FY05: Target1.
Result FY06: Target not met.
8.8% increase ($2.10 vs. $1.93)
-
7/29/2019 PM7Workshop_de Jager en Self
37/42
August 2007 7th Northumbria Conference, Spier,
South Africa
37
Gaining resources
-
7/29/2019 PM7Workshop_de Jager en Self
38/42
August 2007 7th Northumbria Conference, Spier,
South Africa
38
Metric F.1.B. Library spending
compared to University
expenditures
Target1: : The University Library willaccount for at least 2.50% of the
Universitys academic division expenditures. Target2: : The Library will account for at
least 2.25% of expenditures. Result FY06: Target1.
2.57% ($25.2M of $972M)
Metric F 1 C Amount of
-
7/29/2019 PM7Workshop_de Jager en Self
39/42
August 2007 7th Northumbria Conference, Spier,
South Africa
39
Metric F.1.C.Amount of
unrestricted development receipts.
Target1: Increase unrestricted (or minimallyrestricted) giving by 10% each year.
Target2: Increase of 5% per year.
Result FY06: Target1.
FY06 unrestricted receipts were $774,000; target was
$374,000.
-
7/29/2019 PM7Workshop_de Jager en Self
40/42
August 2007 7th Northumbria Conference, Spier,
South Africa
40
Metric L.2.C.Compare staff
salaries to peer groups.
Target1: Library faculty salaries should rankin the top 40% of salaries at ARL libraries.
Target2: Rank in top 50%.
Result FY06: Target1.
Ranked 33 of 113. (Top 28%)
-
7/29/2019 PM7Workshop_de Jager en Self
41/42
August 2007 7th Northumbria Conference, Spier,
South Africa
41
To summarize
The Balanced Scorecard
Reflects the organizations vision
Clarifies and communicates the vision
Provides a quick, but comprehensive, pictureof the organizations health
-
7/29/2019 PM7Workshop_de Jager en Self
42/42
August 2007 7th Northumbria Conference, Spier, 42
Assessing the BSC
What are the benefits?
Is it worth the costs?
Are we measuring the right things? What have we learned?
Where do we go from here?