Developing Performance Measures

December 13, 2000 OEAS--IE Training Workshop 2

Purpose of Workshop

To assist you in developing – Performance measures

– Measurement methods

To discuss Institutional Effectiveness and changes in reporting requirements

December 13, 2000 OEAS--IE Training Workshop 3

First Validate Your Mission, Goals, and Objectives

O b je ctive 1 O b je ctive 2

G o a l 1

O b je ctive 1 O b je ctive 2

G o a l 2

O b je ctive 1 O b je ctive 2

G o a l 3

U n it M iis io n

Why?

How? Necessary and Sufficient

December 13, 2000 OEAS--IE Training Workshop 4

Performance Measure

A means of objectively assessing programs, products, activities, or services

Should be related to your mission and goals Should indicate how you will measure your

objective Should indicate when your objective will be

measured Should indicate who will do the

measurement

December 13, 2000 OEAS--IE Training Workshop 5

Structure of a Performance Measure Statement

The (description of the measure) based on (approach used to develop the measure)– The (number of students attending workshops)

based on (annual attendance records)– The (satisfaction level of students with the given

service) based on (an annual survey of all graduating seniors conducted by OEAS)

– The (application processing time for applications) based on (the time from receipt of an application in the Admissions Mail room to the key-in of the admission decision as recorded in PeopleSoft)

December 13, 2000 OEAS--IE Training Workshop 6

Two Approaches to Generating Objectives and Performance Measures

First Approach Describe mission Describe vision List goals List objectives Determine at least one

performance measure for each objective

Second Approach Describe mission Describe vision List goals Brainstorm

performance measures Select “preferred”

performance measures Set targets which

become the objectives

December 13, 2000 OEAS--IE Training Workshop 7

Generating Objectives then Performance Measures Example: (Admissions)

– Goal: To provide timely response to undergraduate applicants

– Objective: To reduce the application processing time to less than 14 days for 95% of applications

– Performance measure: The application processing time for applications based on the time from receipt of an application in the Admissions Mail room to the key-in of the admission decision as recorded in PeopleSoft

December 13, 2000 OEAS--IE Training Workshop 8

Generating Performance Measures then Objectives Example: (Admissions)

– Goal: To provide timely response to undergraduate applicants

– Related performance measures• The number of days to process application based

on…

• The average number of applications processed per file manager per day based on…

• The student satisfaction with the timeliness of the admission decision based on…

December 13, 2000 OEAS--IE Training Workshop 9

Evaluating the Quality of Your Performance Measures Does each measure relate to the associated

mission, goal, and objective? Is each measure important to management? Is it possible to collect accurate and reliable data

for each performance measure? Taken together, do the measures accurately reflect

the key results of the program, activity or service? Is there more than one measure for each goal or

objective?

December 13, 2000 OEAS--IE Training Workshop 10

Generating Performance Measures then Objectives Goal: To provide timely response to

undergraduate applicants– Objective 1: to increase the productivity of the

file managers by 10%• PM: Average number of applications processed per

file manager per day

– Objective 2: To reduce the application processing time to less than 14 days for 95% of applications

• PM: Number of days to process each application

December 13, 2000 OEAS--IE Training Workshop 11

Two Ways of Measuring Performance Direct measures of performance

– Time– Error rates– Compliance– Cost– Number of outputs per input– Standardized tests

Indirect measures of performance– Perceived time– Perceived efficiency– Perceived quality

December 13, 2000 OEAS--IE Training Workshop 12

Example: To reduce the application processing time to less than 14 days for 95% of applications

Direct measure: The application processing time for applications based on the time from receipt of an application in the Admissions Mail room to the key-in of the admission decision as recorded in PeopleSoft

Indirect measure: The perceived time based on survey of a random sample of 500 applicants

December 13, 2000 OEAS--IE Training Workshop 13

Categories of Performance Measures Input measures (e.g., staff time, materials,

equipment, resources) are useful in showing resources or effort used to provide services; however does not show effectiveness– You may be a spending a lot of effort doing the wrong

things Output measures (e.g., number of products

produced or services provided) are useful in defining program or service; however, does not reveal quality or efficiency– You may be producing or providing a lot of the wrong

things inefficiently or with poor quality

December 13, 2000 OEAS--IE Training Workshop 14

Categories of Performance Measures Outcome measures (e.g., score on standardized test,

distance from proposed targets) are useful in showing the impact or benefit of the program or service

Efficiency measures (e.g., cost per unit of output, outputs per unit of input, outputs per unit time) are useful in showing productivity and cost effectiveness

Quality measures (e.g., reliability, accuracy, courtesy, competence, responsiveness) are useful in measuring the effectiveness in meeting customer expectations

– Lack of quality can be measured (e.g., error rates)

December 13, 2000 OEAS--IE Training Workshop 15

Example Goal: To provide timely response to undergraduate applicants

Input measure: The number of file managers Output measure: The average number of applications

processed per day Outcome measure: The application processing time for

applications based on the time from receipt of an application in the Admissions Mail room to the key-in of the admission decision as recorded in PeopleSoft

Efficiency measure: The average number of applications processed per file manager per day

Quality measure: The satisfaction of applicants with the application processing time as measured by a survey

December 13, 2000 OEAS--IE Training Workshop 16

Another Example: Developing Performance Measures The mission of the XYZ Foundation is to

promote the social welfare of persons resident or located in the Orlando Metropolitan Region by all appropriate means, including relief of the poor, care of the sick and aged, care and nurture of children, elimination of vermin…

December 13, 2000 OEAS--IE Training Workshop 17

Example Goal, Objective, and Performance Measure Goal: To minimize the number of rats in

the City of Orlando Objective: To reduce the number of rats to

ten per city block in the next year Performance measure: The number of rats

per city block as measured by an extensive search of a random sample of 10 city blocks

December 13, 2000 OEAS--IE Training Workshop 18

PMs Relate to Goal: To minimize the number of rats in the City of Orlando The dollars spent per rat killed based on Budget Office and

exterminator records Percent increase in the number of traps set this year based on

city records The satisfaction of the residents with the extermination

program based on survey of a random sample of residents The dollars spent on extermination based on the Budget

Office records The number of rats remaining per city block obtained by

conducting an extensive search of a random sample of the city blocks in Orlando.

The approximate number of rats killed during the year per city block based on the exterminator reports

December 13, 2000 OEAS--IE Training Workshop 19

Evaluating the Quality of Your Performance Measures Does each measure clearly relate to the associated

mission, goal, and/or objective? Is each measure important to management? Is it possible to collect accurate and reliable data for

each performance measure? Taken together, do the measures accurately reflect

the key results of the program, activity or service? Is there more than one measure for each goal and/or

objective? Are your measures primarily outcome, efficiency or

quality measures?

December 13, 2000 OEAS--IE Training Workshop 20

Evaluate: Related, Important, Measurable, Comprehensive, and Preferred Type

MINIMIZE THE NUMBER OF RATS IN ORLANDO The dollars spent per rat killed based on Budget Office and

exterminator records Percent increase in the number of traps set this year based

on city records The satisfaction of the residents with the extermination

program based on survey of a random sample of residents The dollars spent on extermination based on the Budget

Office records The number of rats remaining per city block obtained by

conducting an extensive search of a random sample of the city blocks in Orlando.

The approximate number of rats killed during the year per city block based on the exterminator reports

December 13, 2000 OEAS--IE Training Workshop 21

Some Resulting Objectives Supporting the GoalMINIMIZE THE NUMBER OF RATS IN

ORLANDO

To improve the cost effectiveness by 10% by Fall 2001

To obtain high to very high satisfaction rating of 90% of city residents

To reduce the number of rats to ten per city block in the next year

December 13, 2000 OEAS--IE Training Workshop 22

Activity: Develop Performance Measures for each Goal Try to develop an input, output, outcome,

efficiency, and quality performance measure for each goal

Identify whether each is a direct or indirect measure– Can you identify a direct measure for the ones

that are currently indirect?– Can you identify an indirect measure for the

ones that are currently direct?

December 13, 2000 OEAS--IE Training Workshop 23

Develop Objectives

Evaluate quality of performance measure– Related– Important– Measurable– Comprehensive– Preferred Type (outcome, efficiency, quality)

Develop objectives