witnessed @ Defining New Metrics for Library Success04.22.2014

presented by Moe Hosseini-Ara

Underlying assumptionsUnderlying assumptionsUnderlying assumptionsUnderlying assumptions

Few libraries have a “culture of assessment”…

• Tend to focus on quantitative “performance”

• Most measures indicate past performance

• Need to do more qualitative performance measures

• Difficult and complex (is the excuse given)

• Have we stopped and asked why we do what we Have we stopped and asked why we do what we Have we stopped and asked why we do what we Have we stopped and asked why we do what we do?do?do?do?

Underlying assumptionsUnderlying assumptionsUnderlying assumptionsUnderlying assumptions

Identifying & illustrating value depends on conversations with your stakeholders

…the first conversation shouldn’t be when measures are presented – if they don’t know you exist…

“More” isn’t necessarily better

…in fact, “more” clouds the issue and the message

Underlying assumptionsUnderlying assumptionsUnderlying assumptionsUnderlying assumptions

Not everything that counts can be counted AND not everything that can be counted counts…

…Wait! What??

- Einstein

Questions driving measuresQuestions driving measuresQuestions driving measuresQuestions driving measures

• What is your library or information centre doing?

• How much is it doing?

• How well is it doing it?

• Who is it doing it for?

• What positive impact is the What positive impact is the What positive impact is the What positive impact is the

service having for them?service having for them?service having for them?service having for them?

• Most important question Most important question Most important question Most important question ----

Are you adding VALUE?Are you adding VALUE?Are you adding VALUE?Are you adding VALUE?

Value

SatisfactionOperational

Three types of measuresThree types of measuresThree types of measuresThree types of measures

Operational/Usage MeasuresOperational/Usage MeasuresOperational/Usage MeasuresOperational/Usage Measures

• We’ve long tracked “how much”• We also need to look at differences:

• Peaks? Dips? Switches?• What are these statistics really telling us? Or, not?

• Who needs to see thesestatistics?

• Are they telling a story that you want to tell?

Customer Satisfaction MeasuresCustomer Satisfaction MeasuresCustomer Satisfaction MeasuresCustomer Satisfaction Measures

• Tends to focus on existing products and/or services

• Usually measured through surveys

• Are we also measuring

importance?

• Survey burnout, plus the

expectation is to score all

“excellent” or “10 out of 10”

The VALUE proposition…Measures?The VALUE proposition…Measures?The VALUE proposition…Measures?The VALUE proposition…Measures?

• Are we adding value?

• What difference are we making?

• What is the impact of our services and/or programs?

• How are we contributing

to the success of….?

Meaningful measuresMeaningful measuresMeaningful measuresMeaningful measures• Matter to you AND your stakeholders• Are conveyed in the stakeholders’ language• More effective when defined using all three types of measures• Demonstrate that your programs and services add value

• Focus attention on what is mostimportant for the organization

• Are critical for managing, planning & decision-making

• Are organization-dependent

The Logic ModelThe Logic ModelThe Logic ModelThe Logic Model

…goes beyond documenting what you did and measures what difference you made in the life of your target audience…how has your audience changed…specifically change in the human condition…

Measuring Impact/ValueMeasuring Impact/ValueMeasuring Impact/ValueMeasuring Impact/ValueChange in the human condition?

• Behavior• Skill• Knowledge• Attitude• Circumstances

• Awareness• Motivation• Condition • Status

IMPACT

Equipment

INPUTINPUTINPUTINPUT

resource resource resource resource perspectiveperspectiveperspectiveperspective

OUTPUTOUTPUTOUTPUTOUTPUT

operational operational operational operational perspectiveperspectiveperspectiveperspective

OUTCOMEOUTCOMEOUTCOMEOUTCOME

user user user user perspectiveperspectiveperspectiveperspective

IMPACTIMPACTIMPACTIMPACT

stakeholder stakeholder stakeholder stakeholder perspectiveperspectiveperspectiveperspective

It’s logical that:It’s logical that:It’s logical that:It’s logical that:

IMPACT

Equipment

INPUTINPUTINPUTINPUTResource Resource Resource Resource PerspectivePerspectivePerspectivePerspective

• Equipment

• Space

• Supplies

• Funding

• Staff

IMPACT

Equipment

OUTPUTOUTPUTOUTPUTOUTPUTOOOOperational perational perational perational PPPPerspectiveerspectiveerspectiveerspective

• Training module

• Program

• Report

• # of Program Attendees

• # of Programs Held

IMPACT

OUTCOMEOUTCOMEOUTCOMEOUTCOMEUser User User User

PerspectivePerspectivePerspectivePerspective

• Increased skill

• Know-how or know-that

• Change in status

• Change in behavior or

attitude

IMPACT

IMPACTIMPACTIMPACTIMPACTStakeholder Stakeholder Stakeholder Stakeholder PerspectivePerspectivePerspectivePerspective

• Lower dropout rates

• Increased employment

levels

• Increased recovery rates

Job Skills Job Skills Job Skills Job Skills TrainingTrainingTrainingTrainingInput(resources)

• 10 hours of staff time• $100 budget required to create the program (supplies and

handouts)

Output(operations)

• 2 hour job skills program created for student job seekers• 4 programs offered over a period of a month • total of 107 participants in attendance

Outcome(user)

• Students gained resume and cover letter writing skills• Knowledge of where to look for jobs and submit applications• Ability to give interviews, proficiency in answering interview

questions

Impact(stakeholder)

• 80% of the participants who attended the program reported that they were either able to get interviews or actual job placements

• University can claim that students are able to get jobs following graduation

Job Skills Job Skills Job Skills Job Skills TrainingTrainingTrainingTrainingInput(resources)

• 10 hours of staff time• $100 budget required to create the program (supplies and

handouts)

Output(operations)

• 2 hour job skills program created for student job seekers• 4 programs offered over a period of a month • total of 107 participants in attendance

Outcome(user)

• Students gained resume and cover letter writing skills• Knowledge of where to look for jobs and submit applications• Ability to give interviews, proficiency in answering interview

questions

Impact(stakeholder)

• 80% of the participants who attended the program reported that they were either able to get interviews or actual job placements

• University can claim that students are able to get jobs following graduation

Job Skills Job Skills Job Skills Job Skills TrainingTrainingTrainingTrainingInput(resources)

• 10 hours of staff time• $100 budget required to create the program (supplies and

handouts)

Output(operations)

• 2 hour job skills program created for job seekers• 4 programs offered over a period of a month • total of 107 participants in attendance

Outcome(user)

• Students gained resume and cover letter writing skills• Knowledge of where to look for jobs and submit applications• Ability to give interviews, proficiency in answering interview

questions

Impact(stakeholder)

• 80% of the participants who attended the program reported that they were either able to get interviews or actual job placements

• University can claim that students are able to get jobs following graduation

Job Skills Job Skills Job Skills Job Skills TrainingTrainingTrainingTrainingInput(resources)

• 10 hours of staff time• $100 budget required to create the program (supplies and

handouts)

Output(operations)

• 2 hour job skills program created for job seekers• 4 programs offered over a period of a month • total of 107 participants in attendance

Outcome(user)

• Participants gained resume and cover letter writing skills• Knowledge of where to look for jobs and submit applications• Ability to give interviews, proficiency in answering interview

questions

Impact(stakeholder)

• 80% of the participants who attended the program reported that they were either able to get interviews or actual job placements

• University can claim that students are able to get jobs following graduation

Job Skills Job Skills Job Skills Job Skills TrainingTrainingTrainingTrainingInput(resources)

• 10 hours of staff time• $100 budget required to create the program (supplies and

handouts)

Output(operations)

• 2 hour job skills program created for job seekers• 4 programs offered over a period of a month • total of 107 participants in attendance

Outcome(user)

• Participants gained resume and cover letter writing skills• Knowledge of where to look for jobs and submit applications• Ability to give interviews, proficiency in answering interview

questions

Impact(stakeholder)

• 80% of the participants who attended the program reported that they were either able to get interviews or actual job placements

• Library can claim that as a result of their program residents are able to get jobs, contribute to economy and lower unemployment

BEGINBEGINBEGINBEGINWITHWITHWITHWITH

THETHETHETHEIN MINDIN MINDIN MINDIN MINDENDENDENDEND

Covey 1989Covey 1989Covey 1989Covey 1989

80% of the participants who attended the program reported that they 80% of the participants who attended the program reported that they 80% of the participants who attended the program reported that they 80% of the participants who attended the program reported that they were either able to get interviews or actual job were either able to get interviews or actual job were either able to get interviews or actual job were either able to get interviews or actual job placementsplacementsplacementsplacements

Library can claim that as a result of their program residents are able to get Library can claim that as a result of their program residents are able to get Library can claim that as a result of their program residents are able to get Library can claim that as a result of their program residents are able to get jobs, contribute to economy and lower unemployment jobs, contribute to economy and lower unemployment jobs, contribute to economy and lower unemployment jobs, contribute to economy and lower unemployment

Measurement Measurement Measurement Measurement FrameworkFrameworkFrameworkFramework

Understand Understand Understand Understand the Contextthe Contextthe Contextthe Context

Align Align Align Align Strategies & Strategies & Strategies & Strategies & ObjectivesObjectivesObjectivesObjectives

Identify Identify Identify Identify Services & Services & Services & Services & ProgramsProgramsProgramsPrograms

Define Define Define Define MeasuresMeasuresMeasuresMeasures

Manage Manage Manage Manage Measurement Measurement Measurement Measurement

DataDataDataData

Translate Translate Translate Translate Data into Data into Data into Data into

Outcomes & Outcomes & Outcomes & Outcomes & ImpactsImpactsImpactsImpacts

Communicate Communicate Communicate Communicate ResultsResultsResultsResults

Understand the Context

Know the Context & the Know the Context & the Know the Context & the Know the Context & the CCCContextersontextersontextersontexters

How do theyHow do theyHow do theyHow do they

measure theirmeasure theirmeasure theirmeasure their

Progress/success?Progress/success?Progress/success?Progress/success?

Know your Know your Know your Know your Stakeholders/InflueStakeholders/InflueStakeholders/InflueStakeholders/Influencers ncers ncers ncers –––– who are who are who are who are they?they?they?they?

How is valueHow is valueHow is valueHow is valuedefined,defined,defined,defined,measured &measured &measured &measured &communicated?communicated?communicated?communicated?

• Libraries days are numbered

• Libraries aren’t relevant

• Everyone has computers and internet

• Need information? It’s all online

• Research happens online not in libraries

• With eBooks and research online believes facilities can be made smaller – no need for print material

Councillor Ural Doomed

Dean of HumanitiesDr. Kee Paper

• Influential in university & broader academic circles

• Focus on research

• Has deep seated beliefs in traditional library services

• Quiet

• No technology

• Libraries are institutions for reading & research

Align Strategies

& Objectives

Align Strategies & ObjectivesAlign Strategies & ObjectivesAlign Strategies & ObjectivesAlign Strategies & Objectives

Stakeholder’s Goals & Objectives Your Goals & Objectives: How they Contribute

Accelerate newcomers contribution to the community & economy

Offer programs and services that connect newcomers with relevant agencies and services

Increased research grants Support the research process and assist with submission and completion of grants

Align objectivesAlign objectivesAlign objectivesAlign objectives

• Articulate & document your goals & objectives

What are you trying to accomplish?

• Clarify how these contribute towards your stakeholders’ objectives & desired outcomes

Identify Services & Programs

Identify ServicesIdentify ServicesIdentify ServicesIdentify Services

What services, programs or products contribute to your realizing an objective?

Stakeholder Goals & Objectives

Your Goals & Objectives

Your Services

Identify ServicesIdentify ServicesIdentify ServicesIdentify Services

Library and Library and Library and Library and University University University University PrioritiesPrioritiesPrioritiesPriorities

What we do…What we do…What we do…What we do…

Define Measures

Define successDefine successDefine successDefine success• What will success look like?

• How will you track progress towards meeting goals?

• What indicators or measures will you use?

• Qualitative? Quantitative?

• What data needs to be collected?

• Keep it key

• Keep it simple

• Be practical

• Don’t get mired – or

overwork frontline staff

Manage Measurement

Data

Manage collectionManage collectionManage collectionManage collection• Determine a collection schedule• Work out agreements with other departments that have needed data

• Are you currently collecting data or indicators that are no longer relevant?

• How long do you need to keep data?• Who is responsible?

Translate Data into

Outcomes & Impacts

Define outcomes & impactDefine outcomes & impactDefine outcomes & impactDefine outcomes & impact

• What will success look like?

• And how will you “get” there?

OUTCOMEOUTCOMEOUTCOMEOUTCOME IMPACTIMPACTIMPACTIMPACT

INPUTINPUTINPUTINPUT OUTPUTOUTPUTOUTPUTOUTPUT

Define outcomes & impactDefine outcomes & impactDefine outcomes & impactDefine outcomes & impact

OUTCOMEOUTCOMEOUTCOMEOUTCOME

User User User User PerspectivePerspectivePerspectivePerspective

• Increased skill

• Know-how or know-that

• Change in status

• Change in behaviour or

attitude

IMPACTIMPACTIMPACTIMPACT

Stakeholder Stakeholder Stakeholder Stakeholder PerspectivePerspectivePerspectivePerspective

• Lower dropout rates

• Increased employment

levels

• Increased recovery rates

Communicate Results

Communicate resultsCommunicate resultsCommunicate resultsCommunicate results

• Actually begins when you are understanding the context

• Your message: “we have contributed towards your goals by……”

• If your goals & objectives are meaningful for your planning & decision-making, your measures will be indispensable

What do you need to stop doing?What do you need to stop doing?What do you need to stop doing?What do you need to stop doing?

Value

SatisfactionOperational

The Sweet SpotThe Sweet SpotThe Sweet SpotThe Sweet Spot

ResourcesResourcesResourcesResources

IMLS IMLS IMLS IMLS WebographyWebographyWebographyWebography

http://www.imls.gov/applicants/webography.aspx

Shaping Shaping Shaping Shaping Outcomes Outcomes Outcomes Outcomes

http://www.shapingoutcomes.org/index.htm

Overcoming Overcoming Overcoming Overcoming Our Habits and Learning to Measure Our Habits and Learning to Measure Our Habits and Learning to Measure Our Habits and Learning to Measure Impact Impact Impact Impact by Moe Hosseini-Ara and Rebecca Jones. Information Today, Inc. June 2013.

http://www.infotoday.com/cilmag/jun13/Hosseini-Ara_Jones--Overcoming-Our-Habits-and-Learning-to-Measure-Impact.shtml

Seven Seven Seven Seven Easy Steps to Help You Measure Easy Steps to Help You Measure Easy Steps to Help You Measure Easy Steps to Help You Measure Up Up Up Up by Moe Hosseini-Ara and Rebecca Jones. Information Today, Inc. July/August 2013.

http://www.infotoday.com/cilmag/jul13/Hosseini-Ara_Jones--Seven-Easy-Steps-to-Help-You-Measure-Up.shtml

MOEHOSSEINI-ARAmhosse@markham.ca

Thank you!

Special thanks to Rebecca Jones (Dysart & Jones) for introducing me to the logic model and allowing

me the honor to teach this material with her at numerous conferences…