Download - Technical Assistance Guide Prepared for Web Cast on AT Act Outcomes Wednesday, October 22, 2003
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
Understanding the Government-Understanding the Government-Wide Shift in Accountability: Wide Shift in Accountability:
Using Performance Management Using Performance Management to Plan for Outcomes & to Plan for Outcomes & Demonstrate Results Demonstrate Results
Understanding the Government-Understanding the Government-Wide Shift in Accountability: Wide Shift in Accountability:
Using Performance Management Using Performance Management to Plan for Outcomes & to Plan for Outcomes & Demonstrate Results Demonstrate Results
Technical Assistance GuideTechnical Assistance GuidePrepared for Web Cast on AT Act Outcomes Prepared for Web Cast on AT Act Outcomes
Wednesday, October 22, 2003Wednesday, October 22, 2003
by Margaret L. Campbell, Ph.D. by Margaret L. Campbell, Ph.D.
National Institute on Disability and Rehabilitation National Institute on Disability and Rehabilitation ResearchResearch
U.S. Department of EducationU.S. Department of Education
Technical Assistance GuideTechnical Assistance GuidePrepared for Web Cast on AT Act Outcomes Prepared for Web Cast on AT Act Outcomes
Wednesday, October 22, 2003Wednesday, October 22, 2003
by Margaret L. Campbell, Ph.D. by Margaret L. Campbell, Ph.D.
National Institute on Disability and Rehabilitation National Institute on Disability and Rehabilitation ResearchResearch
U.S. Department of EducationU.S. Department of Education
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
2
Purpose of Technical Assistance Purpose of Technical Assistance GuideGuide
Purpose of Technical Assistance Purpose of Technical Assistance GuideGuide
To provide NIDRR staff, grantees, applicants and reviewers with a comprehensive resource for understanding the government-wide shift in accountability and how the concepts and tools of performance management can be used to: Plan for outputs & outcomes Improve programs Monitor progress, and Document results and communicate
successes
To provide NIDRR staff, grantees, applicants and reviewers with a comprehensive resource for understanding the government-wide shift in accountability and how the concepts and tools of performance management can be used to: Plan for outputs & outcomes Improve programs Monitor progress, and Document results and communicate
successes
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
Part 1:Part 1:Part 1:Part 1:
The “What and the Why” of the Accountability Shift
The “What and the Why” of the Accountability Shift
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
4
What is the Government-Wide Shift in What is the Government-Wide Shift in AccountabilityAccountability??
What is the Government-Wide Shift in What is the Government-Wide Shift in AccountabilityAccountability??
• Shift from primary focus on: Dollars (how much is spent & on what) Activities & processes (what you are
doing to/with whom and how well you are doing it)
Productivity – (how much are you doing)
• To an expanded focus on: Performance -- (are you meeting stated
objectives and standards of quality), and
Results – (what you are achieving and who’s benefiting?)
• Shift from primary focus on: Dollars (how much is spent & on what) Activities & processes (what you are
doing to/with whom and how well you are doing it)
Productivity – (how much are you doing)
• To an expanded focus on: Performance -- (are you meeting stated
objectives and standards of quality), and
Results – (what you are achieving and who’s benefiting?)
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
5
Shift in Accountability is Not Shift in Accountability is Not “Either- Or” Situation “Either- Or” Situation
Shift in Accountability is Not Shift in Accountability is Not “Either- Or” Situation “Either- Or” Situation
Rather amounts to an added dimension of accountability for “results.”
Still must meet accepted standards of practice for: Continuous quality improvementProgram development and
delivery of services, andConsumer involvement Systems change & advocacy
Rather amounts to an added dimension of accountability for “results.”
Still must meet accepted standards of practice for: Continuous quality improvementProgram development and
delivery of services, andConsumer involvement Systems change & advocacy
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
6
What Does “Accountability for What Does “Accountability for Results” Mean?Results” Mean?
What Does “Accountability for What Does “Accountability for Results” Mean?Results” Mean?
In general, “accountability for results” refers to accomplishments that are under the direct or indirect influence of project resources and activities & occur within the boundaries of a funding/budget cycle.
Within the NIDRR context, this translates into accountable for three types of results:
Outputs – planned and produced to date
Short-Term Outcomes – anticipated and actual, and in some cases
Intermediate Outcomes – anticipated or actual
In general, “accountability for results” refers to accomplishments that are under the direct or indirect influence of project resources and activities & occur within the boundaries of a funding/budget cycle.
Within the NIDRR context, this translates into accountable for three types of results:
Outputs – planned and produced to date
Short-Term Outcomes – anticipated and actual, and in some cases
Intermediate Outcomes – anticipated or actual
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
7
Origins of the Accountability Shift in Origins of the Accountability Shift in Government: GPRAGovernment: GPRA
Origins of the Accountability Shift in Origins of the Accountability Shift in Government: GPRAGovernment: GPRA
In 1993, Congress passed the Government In 1993, Congress passed the Government Results and Reporting Act, which Results and Reporting Act, which established uniform requirements for established uniform requirements for sstrategic planning, atrategic planning, annual planning and reporting, and agency-grantee partnerships.
Key Provision of GPRA: Provides a new definition of program evaluation – “an assessment, through objective measurement and statistical analysis, of the manner and extent to which federal programs achieve intended objectives.”
In 1993, Congress passed the Government In 1993, Congress passed the Government Results and Reporting Act, which Results and Reporting Act, which established uniform requirements for established uniform requirements for sstrategic planning, atrategic planning, annual planning and reporting, and agency-grantee partnerships.
Key Provision of GPRA: Provides a new definition of program evaluation – “an assessment, through objective measurement and statistical analysis, of the manner and extent to which federal programs achieve intended objectives.”
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
8
Recent Policy Initiatives ReinforcingRecent Policy Initiatives Reinforcingthe Accountability Shiftthe Accountability Shift
Recent Policy Initiatives ReinforcingRecent Policy Initiatives Reinforcingthe Accountability Shiftthe Accountability Shift
The President’s Management Agenda (PMA) FY 2002, with its emphasis on: Performance management &
measurement Objective criteria to assess program
results (i.e., the PART) Performance-based budgeting
The Dept. of Education’s New Strategic Goals, 2002-2007, with an emphasis on: Transforming Education into an
Evidenced-based Field.
The President’s Management Agenda (PMA) FY 2002, with its emphasis on: Performance management &
measurement Objective criteria to assess program
results (i.e., the PART) Performance-based budgeting
The Dept. of Education’s New Strategic Goals, 2002-2007, with an emphasis on: Transforming Education into an
Evidenced-based Field.
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
Part 2:Part 2:Part 2:Part 2:
The Language and Logic of Outcomes Planning and
Managing for Results(Key Concepts and Definitions)
The Language and Logic of Outcomes Planning and
Managing for Results(Key Concepts and Definitions)
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
10
What is Meant by What is Meant by “Managing for Results?”“Managing for Results?”
What is Meant by What is Meant by “Managing for Results?”“Managing for Results?”
“Managing for results” requires developing a new mindset based on outcomes-oriented goals and performance measures, as opposed to activity-oriented goals and objectives .
The primary focus is on planning for what outputs and outcomes will be achieved not on what activities will be conducted.
For most of us, this involves re-expressing existing program goals and objectives into the language of planned outputs and anticipated outcomes.
“Managing for results” requires developing a new mindset based on outcomes-oriented goals and performance measures, as opposed to activity-oriented goals and objectives .
The primary focus is on planning for what outputs and outcomes will be achieved not on what activities will be conducted.
For most of us, this involves re-expressing existing program goals and objectives into the language of planned outputs and anticipated outcomes.
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
11
Concepts and Tools Concepts and Tools You Need to SucceedYou Need to SucceedConcepts and Tools Concepts and Tools You Need to SucceedYou Need to Succeed
Performance management Performance measurement Performance goals & measures The Performance Spectrum Logic modeling – i.e., reverse
mapping Components of Performance
measurement Target systems and/or populations
Performance management Performance measurement Performance goals & measures The Performance Spectrum Logic modeling – i.e., reverse
mapping Components of Performance
measurement Target systems and/or populations
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
12
What is Performance Management vs. What is Performance Management vs. Performance Measurement? Performance Measurement?
What is Performance Management vs. What is Performance Management vs. Performance Measurement? Performance Measurement?
Performance management is an approach to planning and evaluation that utilizes the concepts and tools of performance measurement and logic modeling to identify performance goals and assess progress towards goals.
Performance measurement “involves the ongoing monitoring and reporting of program accomplishments, particularly progress towards pre-established goals” (see GPRA requirements).
Performance management is an approach to planning and evaluation that utilizes the concepts and tools of performance measurement and logic modeling to identify performance goals and assess progress towards goals.
Performance measurement “involves the ongoing monitoring and reporting of program accomplishments, particularly progress towards pre-established goals” (see GPRA requirements).
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
13
What are Performance Goals?What are Performance Goals?What are Performance Goals?What are Performance Goals?
According to OMB, performance goals are the desired levels of performance expressed as measurable objectives against which actual achievement can be compared.
To be complete, performance goals should incorporate performance measures and performance targets into timeframes.
According to OMB, performance goals are the desired levels of performance expressed as measurable objectives against which actual achievement can be compared.
To be complete, performance goals should incorporate performance measures and performance targets into timeframes.
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
14
Definitions of Performance Goals Definitions of Performance Goals (Cont’d.)(Cont’d.)
Definitions of Performance Goals Definitions of Performance Goals (Cont’d.)(Cont’d.)
Performance measures are the quantitative or qualitative metrics and indicators that are used to demonstrate progress towards goals and document results/ accomplishments.
Performance targets are the quantifiable or otherwise measurable characteristics that tell how well a program or project must accomplish a performance measure.
Performance measures are the quantitative or qualitative metrics and indicators that are used to demonstrate progress towards goals and document results/ accomplishments.
Performance targets are the quantifiable or otherwise measurable characteristics that tell how well a program or project must accomplish a performance measure.
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
15
The Spectrum of Performance: The Spectrum of Performance: Outcomes-Oriented GoalsOutcomes-Oriented Goals
The Spectrum of Performance: The Spectrum of Performance: Outcomes-Oriented GoalsOutcomes-Oriented Goals
Although the PART emphasizes outcome goals, that focus on changes and/or improvements in the target system, performance goals can also be stated in terms of “planned outputs” (and sometimes “quality of activities”).
The key: All performance goals should clearly distinguish between components of performance and there should be a logical connections among them with outputs supporting outcomes (i.e., outcome-oriented performance goals).
Although the PART emphasizes outcome goals, that focus on changes and/or improvements in the target system, performance goals can also be stated in terms of “planned outputs” (and sometimes “quality of activities”).
The key: All performance goals should clearly distinguish between components of performance and there should be a logical connections among them with outputs supporting outcomes (i.e., outcome-oriented performance goals).
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
16
What is Logic Modeling?What is Logic Modeling?What is Logic Modeling?What is Logic Modeling?
A vehicle for dialogue, planning, program improvement and evaluation
A graphic representation or “blueprint” of the key elements of a program or project, and how it will work under certain conditions to “solve” identified problems.
A helpful tool for identifying outcomes and depicting the “chain of events” that link inputs and activities to outputs and outcomes.
A vehicle for dialogue, planning, program improvement and evaluation
A graphic representation or “blueprint” of the key elements of a program or project, and how it will work under certain conditions to “solve” identified problems.
A helpful tool for identifying outcomes and depicting the “chain of events” that link inputs and activities to outputs and outcomes.
17
Short-TermOutcomes
ProgramActivities
IntermediateOutcomes
LongerTerm
Outcomes
Customers Who Use Outputs
Outputs
PROGRAM DELIVERED
RESULTS FROM PROGRAM
EXTERNAL CONDITIONS INFLUENCING SUCCESS (+/-) ANTECEDENT/ MEDIATING
MEASURES
HOW WHY
What Does A Basic Logic Model Look Like? Prepared by the NIDRR PPB&E Division1
1 Source: Adapted from McLaughlin, J. A. & Jordan, G. B. (In Press/2003) Logic models: a tool for describing program theory and performance, in Wholey, et. al.: Handbook of Practical Evaluation. Jossey-Bass. The lead author may be contacted by email at:[email protected]
Note:The above logic model does not include the standards by which the quality, relevance, and productivity of performance will be measured and evaluated across the spectrum. For NIDRR grantees this is provided by the “centers of excellence” (CoE) criteria. Updated on June 27, 2003
Span of Accountability for Results
Inputs &Infrastructure
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
18
Inputs & Infrastructure: The human and financial resources and systems needed to conduct a high quality, outcomes-oriented program or project. Examples: agency priorities and
requirements, host institutional support and leveraging, staff competencies and expertise, management practices and evaluation plans, partnerships and collaborations, and previous accomplishments.
Inputs & Infrastructure: The human and financial resources and systems needed to conduct a high quality, outcomes-oriented program or project. Examples: agency priorities and
requirements, host institutional support and leveraging, staff competencies and expertise, management practices and evaluation plans, partnerships and collaborations, and previous accomplishments.
What are the Components of What are the Components of Performance Measurement?Performance Measurement?
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
19
What are the Components of What are the Components of PerformancePerformance Measurement? (Cont’d) Measurement? (Cont’d)
What are the Components of What are the Components of PerformancePerformance Measurement? (Cont’d) Measurement? (Cont’d)
Program Activities: Are the action steps, tasks, procedures, and services performed in conjunction with implementing your planned program of public awareness, technical assistance and training, Outreach, and interagency coordination to carry out objectives and produce results.
Program Activities: Are the action steps, tasks, procedures, and services performed in conjunction with implementing your planned program of public awareness, technical assistance and training, Outreach, and interagency coordination to carry out objectives and produce results.
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
20
What are the Components of What are the Components of PerformancePerformance Measurement? (Cont’d) Measurement? (Cont’d)
What are the Components of What are the Components of PerformancePerformance Measurement? (Cont’d) Measurement? (Cont’d)
Outputs: Are the direct results of program activities and consist of the findings or conclusions, products and services produced and reported to external audiences. Outputs typically are reported using statistics
or quantitative counts – e.g., the # of individuals trained, newsletters distributed, publications produced, & participants attended.
But they also can be described qualitatively in terms of the nature of the findings or discoveries.
Outputs: Are the direct results of program activities and consist of the findings or conclusions, products and services produced and reported to external audiences. Outputs typically are reported using statistics
or quantitative counts – e.g., the # of individuals trained, newsletters distributed, publications produced, & participants attended.
But they also can be described qualitatively in terms of the nature of the findings or discoveries.
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
21
What are the Components of What are the Components of PerformancePerformance Measurement? (Cont’d) Measurement? (Cont’d)
What are the Components of What are the Components of PerformancePerformance Measurement? (Cont’d) Measurement? (Cont’d)
• More On Outputs: Outputs are important in their own
right as indicators of productivity; they also are the essential building blocks of outcomes.
However, not every output will have a corresponding outcome, nor should it.
• More On Outputs: Outputs are important in their own
right as indicators of productivity; they also are the essential building blocks of outcomes.
However, not every output will have a corresponding outcome, nor should it.
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
22
What are the Components of What are the Components of PerformancePerformance Measurement? (Cont’d) Measurement? (Cont’d)
What are the Components of What are the Components of PerformancePerformance Measurement? (Cont’d) Measurement? (Cont’d)
Utility: Although not an official component of the performance spectrum as defined by GPRA, NIDRR has included utility because of its centrality to our Mission, and because demonstrating the “perceived usefulness” of outputs to sub-groups within the broader target systems constitutes an important bridge between outputs and short-term outcomes.
Utility: Although not an official component of the performance spectrum as defined by GPRA, NIDRR has included utility because of its centrality to our Mission, and because demonstrating the “perceived usefulness” of outputs to sub-groups within the broader target systems constitutes an important bridge between outputs and short-term outcomes.
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
23
What are the Components of What are the Components of Performance Measurement (Cont’d)Performance Measurement (Cont’d)
What are the Components of What are the Components of Performance Measurement (Cont’d)Performance Measurement (Cont’d)
Outcomes: Are the anticipated or actual effects of program activities and outputs and constitute changes or improvements in the target systems being served. Although the emphasis is on planning for outcomes, outcomes may also occur as unexpected changes.
Outcomes are the most complex component of PM and are difficult to define outside of the context of a specific program and target system.
Outcomes: Are the anticipated or actual effects of program activities and outputs and constitute changes or improvements in the target systems being served. Although the emphasis is on planning for outcomes, outcomes may also occur as unexpected changes.
Outcomes are the most complex component of PM and are difficult to define outside of the context of a specific program and target system.
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
24
What are the Components of What are the Components of Performance Measurement (Cont’d)Performance Measurement (Cont’d)
What are the Components of What are the Components of Performance Measurement (Cont’d)Performance Measurement (Cont’d)
Factors to take into account in planning for and documenting outcomes: The basic outcomes equation is: Outcomes
= change/improvement in the target population or system, which is a function of:
Outputs + Documented External Use
Outcomes typically occur sequentially over time and, as a result, may affect more than one target system. Because of this, we refer to “chain of outcomes,” starting with short-term outcomes and progressing to intermediate and longer-term outcomes.
Factors to take into account in planning for and documenting outcomes: The basic outcomes equation is: Outcomes
= change/improvement in the target population or system, which is a function of:
Outputs + Documented External Use
Outcomes typically occur sequentially over time and, as a result, may affect more than one target system. Because of this, we refer to “chain of outcomes,” starting with short-term outcomes and progressing to intermediate and longer-term outcomes.
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
25
Types of OutcomesTypes of OutcomesTypes of OutcomesTypes of Outcomes
Short-term outcomes are expected or actual changes or improvements in the identified target system that are more under the direct influence of center activities. Short-term outcomes represent the first level of change that must occur in order to bring about intermediate outcomes. Typically, short-term outcomes are focused on changes/improvements in the learning & knowledge system at the individual or environmental level of analysis.
Short-term outcomes are expected or actual changes or improvements in the identified target system that are more under the direct influence of center activities. Short-term outcomes represent the first level of change that must occur in order to bring about intermediate outcomes. Typically, short-term outcomes are focused on changes/improvements in the learning & knowledge system at the individual or environmental level of analysis.
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
26
Types of OutcomesTypes of OutcomesTypes of OutcomesTypes of Outcomes
Intermediate outcomes are the “gold-standard” in terms of accountability for results. They consist of expected or actual changes or improvements in the “action” system that occur in part as a result of the use or adoption of project outputs. Unlike short-term outcomes that occur under the direct influence of program activities, there may be other causal factors contributing to the achievement of intermediate.
Intermediate outcomes are the “gold-standard” in terms of accountability for results. They consist of expected or actual changes or improvements in the “action” system that occur in part as a result of the use or adoption of project outputs. Unlike short-term outcomes that occur under the direct influence of program activities, there may be other causal factors contributing to the achievement of intermediate.
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
27
Types of Outcomes (Cont’d)Types of Outcomes (Cont’d)Types of Outcomes (Cont’d)Types of Outcomes (Cont’d)
Longer-term outcomes are the desired end-results of a program, and constitute changes or improvement in the overall condition of a population or system. Given their scope, longer-term outcomes usually take more than one funding cycle to achieve, and therefore are not required under the “accountability for results” umbrella. Their primary function is to serve as critical anchor points in the planning process.
Longer-term outcomes are the desired end-results of a program, and constitute changes or improvement in the overall condition of a population or system. Given their scope, longer-term outcomes usually take more than one funding cycle to achieve, and therefore are not required under the “accountability for results” umbrella. Their primary function is to serve as critical anchor points in the planning process.
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
28
Examples of Types of Outcomes (Changes Examples of Types of Outcomes (Changes or Improvements) at the Individual Levelor Improvements) at the Individual Level
Examples of Types of Outcomes (Changes Examples of Types of Outcomes (Changes or Improvements) at the Individual Levelor Improvements) at the Individual Level
Focus on Learning and Knowledge
Focus on Action Focus on Conditions
Awareness
Knowledge
Attitudes
Skills
Opinion
Motivation
Function
Behavior
Practice and/or Clinical care
Advocacy
Choice or decision-making
Empowerment/Self-Efficacy
Physical or Mental Health Status
Employability
Independent Living
Community Integration, Participation
Economic Self Sufficiency
Quality of Life
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
29
Examples of Types of Outcomes (Changes or Examples of Types of Outcomes (Changes or Improvements) at the Environmental LevelImprovements) at the Environmental Level
Examples of Types of Outcomes (Changes or Examples of Types of Outcomes (Changes or Improvements) at the Environmental LevelImprovements) at the Environmental Level
Focus on Learning and Knowledge
Focus on Action Focus on Conditions
Level or Availability of New
Knowledge
Models, frameworks or guidelines
Methods or Tools
Products and/or devices
Services
Technologies
Use or Adoption of New:
Knowledge
Models, frameworks or guidelines
Methods or Tools
Products and/or devices
Services
Technologies
Standards
Policies, laws and/or regulations
Capacity of systems or society:
Employment Rates
Rates of poverty among PWD
Rates of Secondary Conditions
Rates of Premature Mortality
Delivery Systems
Environmental Access (barrier elimination)
Rates of participation
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
30
What is a Target System?What is a Target System?What is a Target System?What is a Target System?
Formally defined, a “target system” refers to a unit of analysis or bounded set of interacting elements.
In the NIDRR context, a target system or population refers to the level(s) of the environment and/or to the group(s) of individuals that program activities and outputs are intended to affect.
Specifying the particular sub-groups or sample populations within the broader target system that your program intends to affect is key to identifying anticipated outcomes.
Formally defined, a “target system” refers to a unit of analysis or bounded set of interacting elements.
In the NIDRR context, a target system or population refers to the level(s) of the environment and/or to the group(s) of individuals that program activities and outputs are intended to affect.
Specifying the particular sub-groups or sample populations within the broader target system that your program intends to affect is key to identifying anticipated outcomes.
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
31
Examples of the Types of Target Examples of the Types of Target Systems Typically Involved in NIDRR-Systems Typically Involved in NIDRR-
Funded ProjectsFunded Projects
Examples of the Types of Target Examples of the Types of Target Systems Typically Involved in NIDRR-Systems Typically Involved in NIDRR-
Funded ProjectsFunded Projects Individual level systems: persons with
disability and/or their family members, clinicians, service providers, other researchers, policy makers, and industry representatives.
Environmental-level systems: the knowledge base in a field, treatment/practice standards, product development standards, availability of new methods, tools, products, services, and technologies, policies and/or service delivery systems, manufacturers, and societal conditions.
Individual level systems: persons with disability and/or their family members, clinicians, service providers, other researchers, policy makers, and industry representatives.
Environmental-level systems: the knowledge base in a field, treatment/practice standards, product development standards, availability of new methods, tools, products, services, and technologies, policies and/or service delivery systems, manufacturers, and societal conditions.
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
Part 3:Part 3:Part 3:Part 3:
How to Apply Performance Management Tools to Plan for
Outcomes & Manage for Results
How to Apply Performance Management Tools to Plan for
Outcomes & Manage for Results
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
33
Objective of Outcomes PlanningObjective of Outcomes PlanningObjective of Outcomes PlanningObjective of Outcomes Planning
To identify a limited number of high-priority, problem-focused, reasonably ambitious and measurable outcomes-oriented performance goals that:
Reflect NIDRR priorities or statutory requirements, and
Capture the anticipated effects and benefits of project activities & outputs on identified target populations and/or systems.
To identify a limited number of high-priority, problem-focused, reasonably ambitious and measurable outcomes-oriented performance goals that:
Reflect NIDRR priorities or statutory requirements, and
Capture the anticipated effects and benefits of project activities & outputs on identified target populations and/or systems.
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
34
Two Ways to Get to Outcomes:Two Ways to Get to Outcomes:Prospective vs. Retrospective PlanningProspective vs. Retrospective Planning
Two Ways to Get to Outcomes:Two Ways to Get to Outcomes:Prospective vs. Retrospective PlanningProspective vs. Retrospective Planning
Prospective outcomes planning is the preferred strategy, but it requires starting with a “managing for results” mindset and identifying performance goals and measures to guide activities and monitor progress (slides 35- 42 ).
Retrospective outcomes planning is less than ideal, but is how most of us beginning to get our feet wet. It involves identifying outcomes after-the-fact and collecting data to support them (slides .
Prospective outcomes planning is the preferred strategy, but it requires starting with a “managing for results” mindset and identifying performance goals and measures to guide activities and monitor progress (slides 35- 42 ).
Retrospective outcomes planning is less than ideal, but is how most of us beginning to get our feet wet. It involves identifying outcomes after-the-fact and collecting data to support them (slides .
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
35
Prospective Outcomes Planning Prospective Outcomes Planning Step 1: Preparation Step 1: Preparation Prospective Outcomes Planning Prospective Outcomes Planning Step 1: Preparation Step 1: Preparation
Describe the overall problem you are trying to “solve” and specify what part or parts of the problem will be addressed by your program activities (i.e., theory of the problem clarifies scope of your project).
Identify the specific sub-groups or sample populations within the broader target systems that you anticipate will change/ improve as a result of your program.
Explain how your program activities & outputs will change or benefit the specific target systems or populations you have identified (i.e., theory of the program)
Describe the overall problem you are trying to “solve” and specify what part or parts of the problem will be addressed by your program activities (i.e., theory of the problem clarifies scope of your project).
Identify the specific sub-groups or sample populations within the broader target systems that you anticipate will change/ improve as a result of your program.
Explain how your program activities & outputs will change or benefit the specific target systems or populations you have identified (i.e., theory of the program)
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
36
Prospective Outcomes Planning Prospective Outcomes Planning Step 2: Constructing a Logic Model Step 2: Constructing a Logic Model Prospective Outcomes Planning Prospective Outcomes Planning Step 2: Constructing a Logic Model Step 2: Constructing a Logic Model
Translate requirements and objectives of program into anticipated outcomes, starting with longer-term, end-results and working backwards to identify intermediate and short-term outcomes (see Worksheet below);
Identify the specific target populations and systems to be affected by each type of outcome
Plan the outputs that will be produced to bring about anticipated changes in target systems;
Design the activities that must be conducted to produce planned outputs;
Identify the resources needed to implement activities; and
Document the contextual factors that could influence the success of your program.
Translate requirements and objectives of program into anticipated outcomes, starting with longer-term, end-results and working backwards to identify intermediate and short-term outcomes (see Worksheet below);
Identify the specific target populations and systems to be affected by each type of outcome
Plan the outputs that will be produced to bring about anticipated changes in target systems;
Design the activities that must be conducted to produce planned outputs;
Identify the resources needed to implement activities; and
Document the contextual factors that could influence the success of your program.
Worksheet for Constructing a Logic Model of Performance for Your Program
Prepared by the NIDRR PPB&E Division1
Inputs and Infrastructure
What resources and management practices are you
using, or developing, to
conduct planned activities and
produce desired results consistent with the priority
and program requirements?
Program Activities
What activities are you currently conducting, or plan to conduct, across program requirements to fulfill objectives, produce intended outputs, and assess their relevance and utility for target audiences? How well are you implementing these activities; and what is your progress to date, including problems encountered and actions taken?
Outputs
What products, services, and information are you providing, or do you plan to provide to target audiences or customers to support anticipated outcomes? And what is your productivity to date?
Utility
What evidence do you have of the potential merit and/or usefulness of your outputs?
Short-Term Outcomes
What changes or improvements will occur in awareness, understanding or knowledge and for which target populations as a result of your program activities and outputs? And, what measurable indicators will you use to track progress towards achieving these short-term outcomes/ performance goals?
Longer-term Outcomes
1 Source: Adapted from McLaughlin, J. A. & Jordan, G. B. (In Press/2003) Logic models: a tool for describing program theory and performance, in Wholey, et. al.: Handbook of Practical Evaluation. Jossey-Bass. The lead author may be contacted by email at:[email protected]
Note:The above logic model does not include the standards by which the quality, relevance, and productivity of performance will be measured and evaluated across the spectrum. For NIDRR centers and model systems this is provided by the “centers of excellence” (CoE) criteria.
Revised: June 27, 2003
Internal and External Factors That Can Influence Success
Important Tip: When planning, begin with the end in mind and work from right to left.When implementing, begin with inputs and infrastructure and move from left to right
IntermediateOutcomes
What changes or improvements do you anticipate will occur in behavior, practice and/or policy, and for which target populations as a partial result of the use or adoption of new knowledge generated by your program? And, what measurable indicators will you use to track progress towards achieving these intermediate outcomes/ performance goals?
What are the long-term changes or improvements in societal conditions that you hope to contribute to overtime through your program activities, outputs and short-term and intermediate outcomes?
Span of Accountability for Results
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
38
Prospective Outcomes Planning Prospective Outcomes Planning Step 3: Formulate Performance GoalsStep 3: Formulate Performance Goals Prospective Outcomes Planning Prospective Outcomes Planning Step 3: Formulate Performance GoalsStep 3: Formulate Performance Goals
Convert outcome statements into performance goals by: Identifying or developing the
performance measures that will be used to document progress and demonstrate accountability for results (i.e., outputs, short-term and intermediate outcomes).
Establishing a baseline and identifying performance targets that must be met to achieve outcomes.
Establishing the timeframe within which progress & results will occur.
Convert outcome statements into performance goals by: Identifying or developing the
performance measures that will be used to document progress and demonstrate accountability for results (i.e., outputs, short-term and intermediate outcomes).
Establishing a baseline and identifying performance targets that must be met to achieve outcomes.
Establishing the timeframe within which progress & results will occur.
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
39
Prospective Outcomes PlanningProspective Outcomes PlanningStep 4: Evaluation of PerformanceStep 4: Evaluation of PerformanceProspective Outcomes PlanningProspective Outcomes PlanningStep 4: Evaluation of PerformanceStep 4: Evaluation of Performance
Develop an evaluation plan that identifies the types of evidence you need to collect to determine how well your program is working and if it is achieving the anticipated results.
Specify the sources and methods of data collection you will use to measure performance and demonstrate progress towards outcomes-oriented goals (see Table on next slide for examples of data sources & methods).
Develop an evaluation plan that identifies the types of evidence you need to collect to determine how well your program is working and if it is achieving the anticipated results.
Specify the sources and methods of data collection you will use to measure performance and demonstrate progress towards outcomes-oriented goals (see Table on next slide for examples of data sources & methods).
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
40
Examples of Sources and Methods of Data Examples of Sources and Methods of Data Collection for Demonstrating OutcomesCollection for Demonstrating OutcomesExamples of Sources and Methods of Data Examples of Sources and Methods of Data Collection for Demonstrating OutcomesCollection for Demonstrating Outcomes
Secondary Data Analysis
Literature Reviews Data sets/ public use data files
Citation analysis Documents
Record reviews Practitioners
Focus Group Key Informants
Interviews or surveys End-users & Other Stakeholders
Observation Family Members
Usability Assessments, pre/post tests
Participants (consumers, persons with disabilities)
Method of Collecting Information
Source of Information
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
41
Prospective Outcomes PlanningProspective Outcomes PlanningStep 5: Evaluation of PerformanceStep 5: Evaluation of PerformanceProspective Outcomes PlanningProspective Outcomes PlanningStep 5: Evaluation of PerformanceStep 5: Evaluation of Performance
Integrate your performance management plan into the ongoing operation of your program activities to measure performance, collect outcomes data and monitor progress.
Integrate your performance management plan into the ongoing operation of your program activities to measure performance, collect outcomes data and monitor progress.
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
42
Prospective Outcomes PlanningProspective Outcomes PlanningStep 6: Analysis & DocumentationStep 6: Analysis & DocumentationProspective Outcomes PlanningProspective Outcomes PlanningStep 6: Analysis & DocumentationStep 6: Analysis & Documentation Analyze the qualitative and quantitative
performance data to document progress towards anticipated outcomes.
Interpret the data to determine if an outcome has occurred or is likely to occur within the specified timeframe. Making this determination relies on qualitative judgments of quantitative and/or qualitative data.
Document results and communicate accomplishments to external audiences is an evidence-based claims-making process, in which grantees present evidence to support the claim that their program activities & outputs have resulted in changes/ improvements in the identified target system.
Analyze the qualitative and quantitative performance data to document progress towards anticipated outcomes.
Interpret the data to determine if an outcome has occurred or is likely to occur within the specified timeframe. Making this determination relies on qualitative judgments of quantitative and/or qualitative data.
Document results and communicate accomplishments to external audiences is an evidence-based claims-making process, in which grantees present evidence to support the claim that their program activities & outputs have resulted in changes/ improvements in the identified target system.
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
43
Retrospective Outcome Identification: Retrospective Outcome Identification: Problems with ApproachProblems with ApproachRetrospective Outcome Identification: Retrospective Outcome Identification: Problems with ApproachProblems with Approach
Missing pre-established performance goals to guide activities and outputs
Missing specification of target systems where outcomes are expected to occur
Missing a logic model that clarifies the interconnections among activities, outputs and outcomes and between short-term and intermediate outcomes
Missing performance measures and systematic outcomes data to monitor progress and demonstrate results
Missing pre-established performance goals to guide activities and outputs
Missing specification of target systems where outcomes are expected to occur
Missing a logic model that clarifies the interconnections among activities, outputs and outcomes and between short-term and intermediate outcomes
Missing performance measures and systematic outcomes data to monitor progress and demonstrate results
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
44
Retrospective Outcome Identification: Retrospective Outcome Identification: Suggested Implementation StrategiesSuggested Implementation StrategiesRetrospective Outcome Identification: Retrospective Outcome Identification: Suggested Implementation StrategiesSuggested Implementation Strategies
Form an AT Act Work Group to translate statutory requirements into a “limited number of high-priority” (i.e., 2-3) and measurable outcomes-oriented performance goals for each core activity: public awareness, TA & training, outreach, and interagency coordination. (suggested # of outcomes is
Review and categorize accomplishments produced to date into outputs and types of outcomes (short-term, intermediate and long-term).
Form an AT Act Work Group to translate statutory requirements into a “limited number of high-priority” (i.e., 2-3) and measurable outcomes-oriented performance goals for each core activity: public awareness, TA & training, outreach, and interagency coordination. (suggested # of outcomes is
Review and categorize accomplishments produced to date into outputs and types of outcomes (short-term, intermediate and long-term).
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
45
Retrospective Outcome Identification: Retrospective Outcome Identification: Implementation Strategies (Cont’d.)Implementation Strategies (Cont’d.)Retrospective Outcome Identification: Retrospective Outcome Identification: Implementation Strategies (Cont’d.)Implementation Strategies (Cont’d.)
Link accomplishments to newly formulated performance goals and identify the types of data available and the data elements needed to demonstrate progress towards outcomes and document the contribution of AT Act project activities and outputs.
Revise web-based annual performance reporting form for FY 2004 to collect necessary data and rethink how existing data can be analyzed to provide evidence of progress or achievement of outcomes.
Link accomplishments to newly formulated performance goals and identify the types of data available and the data elements needed to demonstrate progress towards outcomes and document the contribution of AT Act project activities and outputs.
Revise web-based annual performance reporting form for FY 2004 to collect necessary data and rethink how existing data can be analyzed to provide evidence of progress or achievement of outcomes.
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
46
Common Weaknesses in Prospective & Common Weaknesses in Prospective & Retrospective Outcomes IdentificationRetrospective Outcomes IdentificationCommon Weaknesses in Prospective & Common Weaknesses in Prospective & Retrospective Outcomes IdentificationRetrospective Outcomes Identification
1. Lack of specificity in “what” and “where”
2. Too many outcomes and expressed too narrowly or too broadly
3. Outcomes expressed as speculations vs. as expectations
4. Confusion between levels of the performance spectrum (i.e., activities vs. outputs and outcomes and short-term vs. intermediate outcomes)
5. Missing supporting links between outputs and short-term outcomes
7. Missing evidence of outcomes
1. Lack of specificity in “what” and “where”
2. Too many outcomes and expressed too narrowly or too broadly
3. Outcomes expressed as speculations vs. as expectations
4. Confusion between levels of the performance spectrum (i.e., activities vs. outputs and outcomes and short-term vs. intermediate outcomes)
5. Missing supporting links between outputs and short-term outcomes
7. Missing evidence of outcomes
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
47
Exercise: What’s Missing in the Exercise: What’s Missing in the Following AT Act Project Outcomes? Following AT Act Project Outcomes?
•Outreach: Within the last grant cycle, State X provided technical assistance to 4, 127 individual and demonstration of loan of equipment to 1,415 individuals.
• Training: State X provided 102 training sessions attended by 3,8288 which resulted in increased knowledge levels that have prompted systems to address AT needs and responsibilities within policies and procedures.
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
48
Exercise: Illustration of Reformulated Exercise: Illustration of Reformulated Outcome StatementOutcome Statement
•Original Statement Systems Change Intermediate Outcome: State AT program worked with the state election officials on the HAVA state plan which resulted in adoption of the FEC 2002 access standards for voting machines.
• Suggested Reformulation: Conduct legislative initiatives that result in the development (short-term outcome) and adoption by states (intermediate outcome) of new standards for voting machines to increase access for adults with moderate to severe disabilities.
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
Part 4:Part 4:Part 4:Part 4:
Lessons Learned:
The Utility and Challenges of Implementing Performance
Measurement
Lessons Learned:
The Utility and Challenges of Implementing Performance
Measurement
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
50
Benefits of Utilizing Benefits of Utilizing Performance Management Performance Management
Benefits of Utilizing Benefits of Utilizing Performance Management Performance Management
Provides a common language and integrated framework to facilitate: Strategic planning & program design Identification of performance goals Program integration – coordinates the
contributions of multiple projects and tasks Objective measurement of progress towards
goals Program monitoring & improvement Program accountability -- documentation of
results, and Communication of success stories
Provides a common language and integrated framework to facilitate: Strategic planning & program design Identification of performance goals Program integration – coordinates the
contributions of multiple projects and tasks Objective measurement of progress towards
goals Program monitoring & improvement Program accountability -- documentation of
results, and Communication of success stories
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
51
Challenges of Applying Performance Challenges of Applying Performance Management Concepts & Tools Management Concepts & Tools
Challenges of Applying Performance Challenges of Applying Performance Management Concepts & Tools Management Concepts & Tools
Developing a “managing for results” mindset. Incorporating requirements of performance
measurement into the traditional standards of excellence for service delivery, consumer involvement, and productivity.
Learning to re-express core program objectives into the language of outcomes-oriented performance goals (outputs and outcomes).
Deciding what evidence you need to collect and how to measure performance and determine how your program is working (i.e., quality and relevance/utility) and if you are achieving planned outputs and anticipated outcomes.
Developing a “managing for results” mindset. Incorporating requirements of performance
measurement into the traditional standards of excellence for service delivery, consumer involvement, and productivity.
Learning to re-express core program objectives into the language of outcomes-oriented performance goals (outputs and outcomes).
Deciding what evidence you need to collect and how to measure performance and determine how your program is working (i.e., quality and relevance/utility) and if you are achieving planned outputs and anticipated outcomes.
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
52
Challenges of Applying Performance Challenges of Applying Performance Management (Cont’d)Management (Cont’d)
Challenges of Applying Performance Challenges of Applying Performance Management (Cont’d)Management (Cont’d)
Integrating the tools of performance management into the operation of your program.
Using the information gathered to improve program management, evaluate performance, achieve results, and provide evidence to support claims of short-term & intermediate outcomes.
Documenting results and communicating successes using a “claims-making” process based on qualitative judgments of quantitative and/qualitative data
Integrating the tools of performance management into the operation of your program.
Using the information gathered to improve program management, evaluate performance, achieve results, and provide evidence to support claims of short-term & intermediate outcomes.
Documenting results and communicating successes using a “claims-making” process based on qualitative judgments of quantitative and/qualitative data
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
53
Lessons Learned from Implementing Lessons Learned from Implementing Performance Management in Business Performance Management in Business
& Government& Government
Lessons Learned from Implementing Lessons Learned from Implementing Performance Management in Business Performance Management in Business
& Government& Government
“What gets measured gets done.” (Drucker)
“If you don’t measure results, you can’t tell success from failure “ (C. Mindel). According to OMB, the number one cause of
federal agencies getting a low score on the new Program Assessment Reporting Tool (PART) is NOT having results-oriented performance measures to gauge the outcomes of a program (http://www.transparentgovernment.org/tg/analysis.htm).
“What gets measured gets done.” (Drucker)
“If you don’t measure results, you can’t tell success from failure “ (C. Mindel). According to OMB, the number one cause of
federal agencies getting a low score on the new Program Assessment Reporting Tool (PART) is NOT having results-oriented performance measures to gauge the outcomes of a program (http://www.transparentgovernment.org/tg/analysis.htm).
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
54
Lessons Learned from Implementing Lessons Learned from Implementing Performance Management (Cont’d)Performance Management (Cont’d)
Lessons Learned from Implementing Lessons Learned from Implementing Performance Management (Cont’d)Performance Management (Cont’d)
Accountability for results involves a partnership -- NIDRR can’t satisfy GPRA and PART alone! To succeed in the new era of performance-based budgeting, NIDRR needs data on accomplishments from grantees like you.
Key Assumption: All grantees have accomplishments; what missing are the performance goals and outcome data.
Key Assumption: “If you can demonstrate results, you can win public support” (C. Mindel).
Accountability for results involves a partnership -- NIDRR can’t satisfy GPRA and PART alone! To succeed in the new era of performance-based budgeting, NIDRR needs data on accomplishments from grantees like you.
Key Assumption: All grantees have accomplishments; what missing are the performance goals and outcome data.
Key Assumption: “If you can demonstrate results, you can win public support” (C. Mindel).
N I
D R
RN
I D
R R
National Institute on
Disability and Rehabilitation
Research
Thank youThank you Thank youThank you