nidrr accountability

N I

D R

RN

I D

R R

National Institute on

Disability and Rehabilitation

Research

Understanding the Government-Understanding the Government-Wide Shift in Accountability: Wide Shift in Accountability:

Using Performance Management Using Performance Management to Plan for Outcomes & to Plan for Outcomes & Demonstrate Results Demonstrate Results

Understanding the Government-Understanding the Government-Wide Shift in Accountability: Wide Shift in Accountability:

Using Performance Management Using Performance Management to Plan for Outcomes & to Plan for Outcomes & Demonstrate Results Demonstrate Results

Technical Assistance GuideTechnical Assistance GuidePrepared for Web Cast on AT Act Outcomes Prepared for Web Cast on AT Act Outcomes

Wednesday, October 22, 2003Wednesday, October 22, 2003

by Margaret L. Campbell, Ph.D. by Margaret L. Campbell, Ph.D.

National Institute on Disability and Rehabilitation National Institute on Disability and Rehabilitation ResearchResearch

U.S. Department of EducationU.S. Department of Education

Technical Assistance GuideTechnical Assistance GuidePrepared for Web Cast on AT Act Outcomes Prepared for Web Cast on AT Act Outcomes

Wednesday, October 22, 2003Wednesday, October 22, 2003

by Margaret L. Campbell, Ph.D. by Margaret L. Campbell, Ph.D.

National Institute on Disability and Rehabilitation National Institute on Disability and Rehabilitation ResearchResearch

U.S. Department of EducationU.S. Department of Education

N I

D R

RN

I D

R R



Research

2

Purpose of Technical Assistance Purpose of Technical Assistance GuideGuide

Purpose of Technical Assistance Purpose of Technical Assistance GuideGuide

To provide NIDRR staff, grantees, applicants and reviewers with a comprehensive resource for understanding the government-wide shift in accountability and how the concepts and tools of performance management can be used to: Plan for outputs & outcomes Improve programs Monitor progress, and Document results and communicate

successes

To provide NIDRR staff, grantees, applicants and reviewers with a comprehensive resource for understanding the government-wide shift in accountability and how the concepts and tools of performance management can be used to: Plan for outputs & outcomes Improve programs Monitor progress, and Document results and communicate

successes

N I

D R

RN

I D

R R



Research

Part 1:Part 1:Part 1:Part 1:

The “What and the Why” of the Accountability Shift

The “What and the Why” of the Accountability Shift

N I

D R

RN

I D

R R



Research

4

What is the Government-Wide Shift in What is the Government-Wide Shift in AccountabilityAccountability??

What is the Government-Wide Shift in What is the Government-Wide Shift in AccountabilityAccountability??

• Shift from primary focus on: Dollars (how much is spent & on what) Activities & processes (what you are

doing to/with whom and how well you are doing it)

Productivity – (how much are you doing)

• To an expanded focus on: Performance -- (are you meeting stated

objectives and standards of quality), and

Results – (what you are achieving and who’s benefiting?)

• Shift from primary focus on: Dollars (how much is spent & on what) Activities & processes (what you are

doing to/with whom and how well you are doing it)

Productivity – (how much are you doing)

• To an expanded focus on: Performance -- (are you meeting stated

objectives and standards of quality), and

Results – (what you are achieving and who’s benefiting?)

N I

D R

RN

I D

R R



Research

5

Shift in Accountability is Not Shift in Accountability is Not “Either- Or” Situation “Either- Or” Situation

Shift in Accountability is Not Shift in Accountability is Not “Either- Or” Situation “Either- Or” Situation

Rather amounts to an added dimension of accountability for “results.”

Still must meet accepted standards of practice for: Continuous quality improvementProgram development and

delivery of services, andConsumer involvement Systems change & advocacy

Rather amounts to an added dimension of accountability for “results.”

Still must meet accepted standards of practice for: Continuous quality improvementProgram development and

delivery of services, andConsumer involvement Systems change & advocacy

N I

D R

RN

I D

R R



Research

6

What Does “Accountability for What Does “Accountability for Results” Mean?Results” Mean?

What Does “Accountability for What Does “Accountability for Results” Mean?Results” Mean?

In general, “accountability for results” refers to accomplishments that are under the direct or indirect influence of project resources and activities & occur within the boundaries of a funding/budget cycle.

Within the NIDRR context, this translates into accountable for three types of results:

Outputs – planned and produced to date

Short-Term Outcomes – anticipated and actual, and in some cases

Intermediate Outcomes – anticipated or actual

In general, “accountability for results” refers to accomplishments that are under the direct or indirect influence of project resources and activities & occur within the boundaries of a funding/budget cycle.

Within the NIDRR context, this translates into accountable for three types of results:

Outputs – planned and produced to date

Short-Term Outcomes – anticipated and actual, and in some cases

Intermediate Outcomes – anticipated or actual

N I

D R

RN

I D

R R



Research

7

Origins of the Accountability Shift in Origins of the Accountability Shift in Government: GPRAGovernment: GPRA

Origins of the Accountability Shift in Origins of the Accountability Shift in Government: GPRAGovernment: GPRA

In 1993, Congress passed the Government In 1993, Congress passed the Government Results and Reporting Act, which Results and Reporting Act, which established uniform requirements for established uniform requirements for sstrategic planning, atrategic planning, annual planning and reporting, and agency-grantee partnerships.

Key Provision of GPRA: Provides a new definition of program evaluation – “an assessment, through objective measurement and statistical analysis, of the manner and extent to which federal programs achieve intended objectives.”

In 1993, Congress passed the Government In 1993, Congress passed the Government Results and Reporting Act, which Results and Reporting Act, which established uniform requirements for established uniform requirements for sstrategic planning, atrategic planning, annual planning and reporting, and agency-grantee partnerships.

Key Provision of GPRA: Provides a new definition of program evaluation – “an assessment, through objective measurement and statistical analysis, of the manner and extent to which federal programs achieve intended objectives.”

N I

D R

RN

I D

R R



Research

8

Recent Policy Initiatives ReinforcingRecent Policy Initiatives Reinforcingthe Accountability Shiftthe Accountability Shift

Recent Policy Initiatives ReinforcingRecent Policy Initiatives Reinforcingthe Accountability Shiftthe Accountability Shift

The President’s Management Agenda (PMA) FY 2002, with its emphasis on: Performance management &

measurement Objective criteria to assess program

results (i.e., the PART) Performance-based budgeting

The Dept. of Education’s New Strategic Goals, 2002-2007, with an emphasis on: Transforming Education into an

Evidenced-based Field.

The President’s Management Agenda (PMA) FY 2002, with its emphasis on: Performance management &

measurement Objective criteria to assess program

results (i.e., the PART) Performance-based budgeting

The Dept. of Education’s New Strategic Goals, 2002-2007, with an emphasis on: Transforming Education into an

Evidenced-based Field.

N I

D R

RN

I D

R R



Research


The Language and Logic of Outcomes Planning and

Managing for Results(Key Concepts and Definitions)

The Language and Logic of Outcomes Planning and

Managing for Results(Key Concepts and Definitions)

N I

D R

RN

I D

R R



Research

10

What is Meant by What is Meant by “Managing for Results?”“Managing for Results?”

What is Meant by What is Meant by “Managing for Results?”“Managing for Results?”

“Managing for results” requires developing a new mindset based on outcomes-oriented goals and performance measures, as opposed to activity-oriented goals and objectives .

The primary focus is on planning for what outputs and outcomes will be achieved not on what activities will be conducted.

For most of us, this involves re-expressing existing program goals and objectives into the language of planned outputs and anticipated outcomes.

“Managing for results” requires developing a new mindset based on outcomes-oriented goals and performance measures, as opposed to activity-oriented goals and objectives .

The primary focus is on planning for what outputs and outcomes will be achieved not on what activities will be conducted.

For most of us, this involves re-expressing existing program goals and objectives into the language of planned outputs and anticipated outcomes.

N I

D R

RN

I D

R R



Research

11

Concepts and Tools Concepts and Tools You Need to SucceedYou Need to SucceedConcepts and Tools Concepts and Tools You Need to SucceedYou Need to Succeed

Performance management Performance measurement Performance goals & measures The Performance Spectrum Logic modeling – i.e., reverse

mapping Components of Performance

measurement Target systems and/or populations

Performance management Performance measurement Performance goals & measures The Performance Spectrum Logic modeling – i.e., reverse

mapping Components of Performance

measurement Target systems and/or populations

N I

D R

RN

I D

R R



Research

12

What is Performance Management vs. What is Performance Management vs. Performance Measurement? Performance Measurement?

What is Performance Management vs. What is Performance Management vs. Performance Measurement? Performance Measurement?

Performance management is an approach to planning and evaluation that utilizes the concepts and tools of performance measurement and logic modeling to identify performance goals and assess progress towards goals.

Performance measurement “involves the ongoing monitoring and reporting of program accomplishments, particularly progress towards pre-established goals” (see GPRA requirements).

Performance management is an approach to planning and evaluation that utilizes the concepts and tools of performance measurement and logic modeling to identify performance goals and assess progress towards goals.

Performance measurement “involves the ongoing monitoring and reporting of program accomplishments, particularly progress towards pre-established goals” (see GPRA requirements).

N I

D R

RN

I D

R R



Research

13

What are Performance Goals?What are Performance Goals?What are Performance Goals?What are Performance Goals?

According to OMB, performance goals are the desired levels of performance expressed as measurable objectives against which actual achievement can be compared.

To be complete, performance goals should incorporate performance measures and performance targets into timeframes.

According to OMB, performance goals are the desired levels of performance expressed as measurable objectives against which actual achievement can be compared.

To be complete, performance goals should incorporate performance measures and performance targets into timeframes.

N I

D R

RN

I D

R R



Research

14

Definitions of Performance Goals Definitions of Performance Goals (Cont’d.)(Cont’d.)

Definitions of Performance Goals Definitions of Performance Goals (Cont’d.)(Cont’d.)

Performance measures are the quantitative or qualitative metrics and indicators that are used to demonstrate progress towards goals and document results/ accomplishments.

Performance targets are the quantifiable or otherwise measurable characteristics that tell how well a program or project must accomplish a performance measure.

Performance measures are the quantitative or qualitative metrics and indicators that are used to demonstrate progress towards goals and document results/ accomplishments.

Performance targets are the quantifiable or otherwise measurable characteristics that tell how well a program or project must accomplish a performance measure.

N I

D R

RN

I D

R R



Research

15

The Spectrum of Performance: The Spectrum of Performance: Outcomes-Oriented GoalsOutcomes-Oriented Goals

The Spectrum of Performance: The Spectrum of Performance: Outcomes-Oriented GoalsOutcomes-Oriented Goals

Although the PART emphasizes outcome goals, that focus on changes and/or improvements in the target system, performance goals can also be stated in terms of “planned outputs” (and sometimes “quality of activities”).

The key: All performance goals should clearly distinguish between components of performance and there should be a logical connections among them with outputs supporting outcomes (i.e., outcome-oriented performance goals).

Although the PART emphasizes outcome goals, that focus on changes and/or improvements in the target system, performance goals can also be stated in terms of “planned outputs” (and sometimes “quality of activities”).

The key: All performance goals should clearly distinguish between components of performance and there should be a logical connections among them with outputs supporting outcomes (i.e., outcome-oriented performance goals).

N I

D R

RN

I D

R R



Research

16

What is Logic Modeling?What is Logic Modeling?What is Logic Modeling?What is Logic Modeling?

A vehicle for dialogue, planning, program improvement and evaluation

A graphic representation or “blueprint” of the key elements of a program or project, and how it will work under certain conditions to “solve” identified problems.

A helpful tool for identifying outcomes and depicting the “chain of events” that link inputs and activities to outputs and outcomes.

A vehicle for dialogue, planning, program improvement and evaluation

A graphic representation or “blueprint” of the key elements of a program or project, and how it will work under certain conditions to “solve” identified problems.

A helpful tool for identifying outcomes and depicting the “chain of events” that link inputs and activities to outputs and outcomes.

17

Short-TermOutcomes

ProgramActivities

IntermediateOutcomes

LongerTerm

Outcomes

Customers Who Use Outputs

Outputs

PROGRAM DELIVERED

RESULTS FROM PROGRAM

EXTERNAL CONDITIONS INFLUENCING SUCCESS (+/-) ANTECEDENT/ MEDIATING

MEASURES

HOW WHY

What Does A Basic Logic Model Look Like? Prepared by the NIDRR PPB&E Division1

1 Source: Adapted from McLaughlin, J. A. & Jordan, G. B. (In Press/2003) Logic models: a tool for describing program theory and performance, in Wholey, et. al.: Handbook of Practical Evaluation. Jossey-Bass. The lead author may be contacted by email at:[email protected]

Note:The above logic model does not include the standards by which the quality, relevance, and productivity of performance will be measured and evaluated across the spectrum. For NIDRR grantees this is provided by the “centers of excellence” (CoE) criteria. Updated on June 27, 2003

Span of Accountability for Results

Inputs &Infrastructure

mailto:[email protected]

N I

D R

RN

I D

R R



Research

18

Inputs & Infrastructure: The human and financial resources and systems needed to conduct a high quality, outcomes-oriented program or project. Examples: agency priorities and

requirements, host institutional support and leveraging, staff competencies and expertise, management practices and evaluation plans, partnerships and collaborations, and previous accomplishments.

Inputs & Infrastructure: The human and financial resources and systems needed to conduct a high quality, outcomes-oriented program or project. Examples: agency priorities and

requirements, host institutional support and leveraging, staff competencies and expertise, management practices and evaluation plans, partnerships and collaborations, and previous accomplishments.

What are the Components of What are the Components of Performance Measurement?Performance Measurement?

N I

D R

RN

I D

R R



Research

19

What are the Components of What are the Components of PerformancePerformance Measurement? (Cont’d) Measurement? (Cont’d)


Program Activities: Are the action steps, tasks, procedures, and services performed in conjunction with implementing your planned program of public awareness, technical assistance and training, Outreach, and interagency coordination to carry out objectives and produce results.

Program Activities: Are the action steps, tasks, procedures, and services performed in conjunction with implementing your planned program of public awareness, technical assistance and training, Outreach, and interagency coordination to carry out objectives and produce results.

N I

D R

RN

I D

R R



Research

20



Outputs: Are the direct results of program activities and consist of the findings or conclusions, products and services produced and reported to external audiences. Outputs typically are reported using statistics

or quantitative counts – e.g., the # of individuals trained, newsletters distributed, publications produced, & participants attended.

But they also can be described qualitatively in terms of the nature of the findings or discoveries.

Outputs: Are the direct results of program activities and consist of the findings or conclusions, products and services produced and reported to external audiences. Outputs typically are reported using statistics

or quantitative counts – e.g., the # of individuals trained, newsletters distributed, publications produced, & participants attended.

But they also can be described qualitatively in terms of the nature of the findings or discoveries.

N I

D R

RN

I D

R R



Research

21



• More On Outputs: Outputs are important in their own

right as indicators of productivity; they also are the essential building blocks of outcomes.

However, not every output will have a corresponding outcome, nor should it.

• More On Outputs: Outputs are important in their own

right as indicators of productivity; they also are the essential building blocks of outcomes.

However, not every output will have a corresponding outcome, nor should it.

N I

D R

RN

I D

R R



Research

22



Utility: Although not an official component of the performance spectrum as defined by GPRA, NIDRR has included utility because of its centrality to our Mission, and because demonstrating the “perceived usefulness” of outputs to sub-groups within the broader target systems constitutes an important bridge between outputs and short-term outcomes.

Utility: Although not an official component of the performance spectrum as defined by GPRA, NIDRR has included utility because of its centrality to our Mission, and because demonstrating the “perceived usefulness” of outputs to sub-groups within the broader target systems constitutes an important bridge between outputs and short-term outcomes.

N I

D R

RN

I D

R R



Research

23

What are the Components of What are the Components of Performance Measurement (Cont’d)Performance Measurement (Cont’d)


Outcomes: Are the anticipated or actual effects of program activities and outputs and constitute changes or improvements in the target systems being served. Although the emphasis is on planning for outcomes, outcomes may also occur as unexpected changes.

Outcomes are the most complex component of PM and are difficult to define outside of the context of a specific program and target system.

Outcomes: Are the anticipated or actual effects of program activities and outputs and constitute changes or improvements in the target systems being served. Although the emphasis is on planning for outcomes, outcomes may also occur as unexpected changes.

Outcomes are the most complex component of PM and are difficult to define outside of the context of a specific program and target system.

N I

D R

RN

I D

R R



Research

24



Factors to take into account in planning for and documenting outcomes: The basic outcomes equation is: Outcomes

= change/improvement in the target population or system, which is a function of:

Outputs + Documented External Use

Outcomes typically occur sequentially over time and, as a result, may affect more than one target system. Because of this, we refer to “chain of outcomes,” starting with short-term outcomes and progressing to intermediate and longer-term outcomes.

Factors to take into account in planning for and documenting outcomes: The basic outcomes equation is: Outcomes

= change/improvement in the target population or system, which is a function of:

Outputs + Documented External Use

Outcomes typically occur sequentially over time and, as a result, may affect more than one target system. Because of this, we refer to “chain of outcomes,” starting with short-term outcomes and progressing to intermediate and longer-term outcomes.

N I

D R

RN

I D

R R



Research

25

Types of OutcomesTypes of OutcomesTypes of OutcomesTypes of Outcomes

Short-term outcomes are expected or actual changes or improvements in the identified target system that are more under the direct influence of center activities. Short-term outcomes represent the first level of change that must occur in order to bring about intermediate outcomes. Typically, short-term outcomes are focused on changes/improvements in the learning & knowledge system at the individual or environmental level of analysis.

Short-term outcomes are expected or actual changes or improvements in the identified target system that are more under the direct influence of center activities. Short-term outcomes represent the first level of change that must occur in order to bring about intermediate outcomes. Typically, short-term outcomes are focused on changes/improvements in the learning & knowledge system at the individual or environmental level of analysis.

N I

D R

RN

I D

R R



Research

26

Types of OutcomesTypes of OutcomesTypes of OutcomesTypes of Outcomes

Intermediate outcomes are the “gold-standard” in terms of accountability for results. They consist of expected or actual changes or improvements in the “action” system that occur in part as a result of the use or adoption of project outputs. Unlike short-term outcomes that occur under the direct influence of program activities, there may be other causal factors contributing to the achievement of intermediate.

Intermediate outcomes are the “gold-standard” in terms of accountability for results. They consist of expected or actual changes or improvements in the “action” system that occur in part as a result of the use or adoption of project outputs. Unlike short-term outcomes that occur under the direct influence of program activities, there may be other causal factors contributing to the achievement of intermediate.

N I

D R

RN

I D

R R



Research

27

Types of Outcomes (Cont’d)Types of Outcomes (Cont’d)Types of Outcomes (Cont’d)Types of Outcomes (Cont’d)

Longer-term outcomes are the desired end-results of a program, and constitute changes or improvement in the overall condition of a population or system. Given their scope, longer-term outcomes usually take more than one funding cycle to achieve, and therefore are not required under the “accountability for results” umbrella. Their primary function is to serve as critical anchor points in the planning process.

Longer-term outcomes are the desired end-results of a program, and constitute changes or improvement in the overall condition of a population or system. Given their scope, longer-term outcomes usually take more than one funding cycle to achieve, and therefore are not required under the “accountability for results” umbrella. Their primary function is to serve as critical anchor points in the planning process.

N I

D R

RN

I D

R R



Research

28

Examples of Types of Outcomes (Changes Examples of Types of Outcomes (Changes or Improvements) at the Individual Levelor Improvements) at the Individual Level

Examples of Types of Outcomes (Changes Examples of Types of Outcomes (Changes or Improvements) at the Individual Levelor Improvements) at the Individual Level

Focus on Learning and Knowledge

Focus on Action Focus on Conditions

Awareness

Knowledge

Attitudes

Skills

Opinion

Motivation

Function

Behavior

Practice and/or Clinical care

Advocacy

Choice or decision-making

Empowerment/Self-Efficacy

Physical or Mental Health Status

Employability

Independent Living

Community Integration, Participation

Economic Self Sufficiency

Quality of Life

N I

D R

RN

I D

R R



Research

29

Examples of Types of Outcomes (Changes or Examples of Types of Outcomes (Changes or Improvements) at the Environmental LevelImprovements) at the Environmental Level

Examples of Types of Outcomes (Changes or Examples of Types of Outcomes (Changes or Improvements) at the Environmental LevelImprovements) at the Environmental Level

Focus on Learning and Knowledge

Focus on Action Focus on Conditions

Level or Availability of New

Knowledge

Models, frameworks or guidelines

Methods or Tools

Products and/or devices

Services

Technologies

Use or Adoption of New:

Knowledge

Models, frameworks or guidelines

Methods or Tools

Products and/or devices

Services

Technologies

Standards

Policies, laws and/or regulations

Capacity of systems or society:

Employment Rates

Rates of poverty among PWD

Rates of Secondary Conditions

Rates of Premature Mortality

Delivery Systems

Environmental Access (barrier elimination)

Rates of participation

N I

D R

RN

I D

R R



Research

30

What is a Target System?What is a Target System?What is a Target System?What is a Target System?

Formally defined, a “target system” refers to a unit of analysis or bounded set of interacting elements.

In the NIDRR context, a target system or population refers to the level(s) of the environment and/or to the group(s) of individuals that program activities and outputs are intended to affect.

Specifying the particular sub-groups or sample populations within the broader target system that your program intends to affect is key to identifying anticipated outcomes.

Formally defined, a “target system” refers to a unit of analysis or bounded set of interacting elements.

In the NIDRR context, a target system or population refers to the level(s) of the environment and/or to the group(s) of individuals that program activities and outputs are intended to affect.

Specifying the particular sub-groups or sample populations within the broader target system that your program intends to affect is key to identifying anticipated outcomes.

N I

D R

RN

I D

R R



Research

31

Examples of the Types of Target Examples of the Types of Target Systems Typically Involved in NIDRR-Systems Typically Involved in NIDRR-

Funded ProjectsFunded Projects

Examples of the Types of Target Examples of the Types of Target Systems Typically Involved in NIDRR-Systems Typically Involved in NIDRR-

Funded ProjectsFunded Projects Individual level systems: persons with

disability and/or their family members, clinicians, service providers, other researchers, policy makers, and industry representatives.

Environmental-level systems: the knowledge base in a field, treatment/practice standards, product development standards, availability of new methods, tools, products, services, and technologies, policies and/or service delivery systems, manufacturers, and societal conditions.

Individual level systems: persons with disability and/or their family members, clinicians, service providers, other researchers, policy makers, and industry representatives.

Environmental-level systems: the knowledge base in a field, treatment/practice standards, product development standards, availability of new methods, tools, products, services, and technologies, policies and/or service delivery systems, manufacturers, and societal conditions.

N I

D R

RN

I D

R R



Research


How to Apply Performance Management Tools to Plan for

Outcomes & Manage for Results

How to Apply Performance Management Tools to Plan for

Outcomes & Manage for Results

N I

D R

RN

I D

R R



Research

33

Objective of Outcomes PlanningObjective of Outcomes PlanningObjective of Outcomes PlanningObjective of Outcomes Planning

To identify a limited number of high-priority, problem-focused, reasonably ambitious and measurable outcomes-oriented performance goals that:

Reflect NIDRR priorities or statutory requirements, and

Capture the anticipated effects and benefits of project activities & outputs on identified target populations and/or systems.

To identify a limited number of high-priority, problem-focused, reasonably ambitious and measurable outcomes-oriented performance goals that:

Reflect NIDRR priorities or statutory requirements, and

Capture the anticipated effects and benefits of project activities & outputs on identified target populations and/or systems.

N I

D R

RN

I D

R R



Research

34

Two Ways to Get to Outcomes:Two Ways to Get to Outcomes:Prospective vs. Retrospective PlanningProspective vs. Retrospective Planning

Two Ways to Get to Outcomes:Two Ways to Get to Outcomes:Prospective vs. Retrospective PlanningProspective vs. Retrospective Planning

Prospective outcomes planning is the preferred strategy, but it requires starting with a “managing for results” mindset and identifying performance goals and measures to guide activities and monitor progress (slides 35- 42 ).

Retrospective outcomes planning is less than ideal, but is how most of us beginning to get our feet wet. It involves identifying outcomes after-the-fact and collecting data to support them (slides .

Prospective outcomes planning is the preferred strategy, but it requires starting with a “managing for results” mindset and identifying performance goals and measures to guide activities and monitor progress (slides 35- 42 ).

Retrospective outcomes planning is less than ideal, but is how most of us beginning to get our feet wet. It involves identifying outcomes after-the-fact and collecting data to support them (slides .

N I

D R

RN

I D

R R



Research

35

Prospective Outcomes Planning Prospective Outcomes Planning Step 1: Preparation Step 1: Preparation Prospective Outcomes Planning Prospective Outcomes Planning Step 1: Preparation Step 1: Preparation

Describe the overall problem you are trying to “solve” and specify what part or parts of the problem will be addressed by your program activities (i.e., theory of the problem clarifies scope of your project).

Identify the specific sub-groups or sample populations within the broader target systems that you anticipate will change/ improve as a result of your program.

Explain how your program activities & outputs will change or benefit the specific target systems or populations you have identified (i.e., theory of the program)

Describe the overall problem you are trying to “solve” and specify what part or parts of the problem will be addressed by your program activities (i.e., theory of the problem clarifies scope of your project).

Identify the specific sub-groups or sample populations within the broader target systems that you anticipate will change/ improve as a result of your program.

Explain how your program activities & outputs will change or benefit the specific target systems or populations you have identified (i.e., theory of the program)

N I

D R

RN

I D

R R



Research

36

Prospective Outcomes Planning Prospective Outcomes Planning Step 2: Constructing a Logic Model Step 2: Constructing a Logic Model Prospective Outcomes Planning Prospective Outcomes Planning Step 2: Constructing a Logic Model Step 2: Constructing a Logic Model

Translate requirements and objectives of program into anticipated outcomes, starting with longer-term, end-results and working backwards to identify intermediate and short-term outcomes (see Worksheet below);

Identify the specific target populations and systems to be affected by each type of outcome

Plan the outputs that will be produced to bring about anticipated changes in target systems;

Design the activities that must be conducted to produce planned outputs;

Identify the resources needed to implement activities; and

Document the contextual factors that could influence the success of your program.

Translate requirements and objectives of program into anticipated outcomes, starting with longer-term, end-results and working backwards to identify intermediate and short-term outcomes (see Worksheet below);

Identify the specific target populations and systems to be affected by each type of outcome

Plan the outputs that will be produced to bring about anticipated changes in target systems;

Design the activities that must be conducted to produce planned outputs;

Identify the resources needed to implement activities; and

Document the contextual factors that could influence the success of your program.

Worksheet for Constructing a Logic Model of Performance for Your Program

Prepared by the NIDRR PPB&E Division1

Inputs and Infrastructure

What resources and management practices are you

using, or developing, to

conduct planned activities and

produce desired results consistent with the priority

and program requirements?

Program Activities

What activities are you currently conducting, or plan to conduct, across program requirements to fulfill objectives, produce intended outputs, and assess their relevance and utility for target audiences? How well are you implementing these activities; and what is your progress to date, including problems encountered and actions taken?

Outputs

What products, services, and information are you providing, or do you plan to provide to target audiences or customers to support anticipated outcomes? And what is your productivity to date?

Utility

What evidence do you have of the potential merit and/or usefulness of your outputs?

Short-Term Outcomes

What changes or improvements will occur in awareness, understanding or knowledge and for which target populations as a result of your program activities and outputs? And, what measurable indicators will you use to track progress towards achieving these short-term outcomes/ performance goals?

Longer-term Outcomes

1 Source: Adapted from McLaughlin, J. A. & Jordan, G. B. (In Press/2003) Logic models: a tool for describing program theory and performance, in Wholey, et. al.: Handbook of Practical Evaluation. Jossey-Bass. The lead author may be contacted by email at:[email protected]

Note:The above logic model does not include the standards by which the quality, relevance, and productivity of performance will be measured and evaluated across the spectrum. For NIDRR centers and model systems this is provided by the “centers of excellence” (CoE) criteria.

Revised: June 27, 2003

Internal and External Factors That Can Influence Success

Important Tip: When planning, begin with the end in mind and work from right to left.When implementing, begin with inputs and infrastructure and move from left to right

IntermediateOutcomes

What changes or improvements do you anticipate will occur in behavior, practice and/or policy, and for which target populations as a partial result of the use or adoption of new knowledge generated by your program? And, what measurable indicators will you use to track progress towards achieving these intermediate outcomes/ performance goals?

What are the long-term changes or improvements in societal conditions that you hope to contribute to overtime through your program activities, outputs and short-term and intermediate outcomes?

Span of Accountability for Results

mailto:[email protected]

N I

D R

RN

I D

R R



Research

38

Prospective Outcomes Planning Prospective Outcomes Planning Step 3: Formulate Performance GoalsStep 3: Formulate Performance Goals Prospective Outcomes Planning Prospective Outcomes Planning Step 3: Formulate Performance GoalsStep 3: Formulate Performance Goals

Convert outcome statements into performance goals by: Identifying or developing the

performance measures that will be used to document progress and demonstrate accountability for results (i.e., outputs, short-term and intermediate outcomes).

Establishing a baseline and identifying performance targets that must be met to achieve outcomes.

Establishing the timeframe within which progress & results will occur.

Convert outcome statements into performance goals by: Identifying or developing the

performance measures that will be used to document progress and demonstrate accountability for results (i.e., outputs, short-term and intermediate outcomes).

Establishing a baseline and identifying performance targets that must be met to achieve outcomes.

Establishing the timeframe within which progress & results will occur.

N I

D R

RN

I D

R R



Research

39

Prospective Outcomes PlanningProspective Outcomes PlanningStep 4: Evaluation of PerformanceStep 4: Evaluation of PerformanceProspective Outcomes PlanningProspective Outcomes PlanningStep 4: Evaluation of PerformanceStep 4: Evaluation of Performance

Develop an evaluation plan that identifies the types of evidence you need to collect to determine how well your program is working and if it is achieving the anticipated results.

Specify the sources and methods of data collection you will use to measure performance and demonstrate progress towards outcomes-oriented goals (see Table on next slide for examples of data sources & methods).

Develop an evaluation plan that identifies the types of evidence you need to collect to determine how well your program is working and if it is achieving the anticipated results.

Specify the sources and methods of data collection you will use to measure performance and demonstrate progress towards outcomes-oriented goals (see Table on next slide for examples of data sources & methods).

N I

D R

RN

I D

R R



Research

40

Examples of Sources and Methods of Data Examples of Sources and Methods of Data Collection for Demonstrating OutcomesCollection for Demonstrating OutcomesExamples of Sources and Methods of Data Examples of Sources and Methods of Data Collection for Demonstrating OutcomesCollection for Demonstrating Outcomes

Secondary Data Analysis

Literature Reviews Data sets/ public use data files

Citation analysis Documents

Record reviews Practitioners

Focus Group Key Informants

Interviews or surveys End-users & Other Stakeholders

Observation Family Members

Usability Assessments, pre/post tests

Participants (consumers, persons with disabilities)

Method of Collecting Information

Source of Information

N I

D R

RN

I D

R R



Research

41

Prospective Outcomes PlanningProspective Outcomes PlanningStep 5: Evaluation of PerformanceStep 5: Evaluation of PerformanceProspective Outcomes PlanningProspective Outcomes PlanningStep 5: Evaluation of PerformanceStep 5: Evaluation of Performance

Integrate your performance management plan into the ongoing operation of your program activities to measure performance, collect outcomes data and monitor progress.

Integrate your performance management plan into the ongoing operation of your program activities to measure performance, collect outcomes data and monitor progress.

N I

D R

RN

I D

R R



Research

42

Prospective Outcomes PlanningProspective Outcomes PlanningStep 6: Analysis & DocumentationStep 6: Analysis & DocumentationProspective Outcomes PlanningProspective Outcomes PlanningStep 6: Analysis & DocumentationStep 6: Analysis & Documentation Analyze the qualitative and quantitative

performance data to document progress towards anticipated outcomes.

Interpret the data to determine if an outcome has occurred or is likely to occur within the specified timeframe. Making this determination relies on qualitative judgments of quantitative and/or qualitative data.

Document results and communicate accomplishments to external audiences is an evidence-based claims-making process, in which grantees present evidence to support the claim that their program activities & outputs have resulted in changes/ improvements in the identified target system.

Analyze the qualitative and quantitative performance data to document progress towards anticipated outcomes.

Interpret the data to determine if an outcome has occurred or is likely to occur within the specified timeframe. Making this determination relies on qualitative judgments of quantitative and/or qualitative data.

Document results and communicate accomplishments to external audiences is an evidence-based claims-making process, in which grantees present evidence to support the claim that their program activities & outputs have resulted in changes/ improvements in the identified target system.

N I

D R

RN

I D

R R



Research

43

Retrospective Outcome Identification: Retrospective Outcome Identification: Problems with ApproachProblems with ApproachRetrospective Outcome Identification: Retrospective Outcome Identification: Problems with ApproachProblems with Approach

Missing pre-established performance goals to guide activities and outputs

Missing specification of target systems where outcomes are expected to occur

Missing a logic model that clarifies the interconnections among activities, outputs and outcomes and between short-term and intermediate outcomes

Missing performance measures and systematic outcomes data to monitor progress and demonstrate results

Missing pre-established performance goals to guide activities and outputs

Missing specification of target systems where outcomes are expected to occur

Missing a logic model that clarifies the interconnections among activities, outputs and outcomes and between short-term and intermediate outcomes

Missing performance measures and systematic outcomes data to monitor progress and demonstrate results

N I

D R

RN

I D

R R



Research

44

Retrospective Outcome Identification: Retrospective Outcome Identification: Suggested Implementation StrategiesSuggested Implementation StrategiesRetrospective Outcome Identification: Retrospective Outcome Identification: Suggested Implementation StrategiesSuggested Implementation Strategies

Form an AT Act Work Group to translate statutory requirements into a “limited number of high-priority” (i.e., 2-3) and measurable outcomes-oriented performance goals for each core activity: public awareness, TA & training, outreach, and interagency coordination. (suggested # of outcomes is

Review and categorize accomplishments produced to date into outputs and types of outcomes (short-term, intermediate and long-term).

Form an AT Act Work Group to translate statutory requirements into a “limited number of high-priority” (i.e., 2-3) and measurable outcomes-oriented performance goals for each core activity: public awareness, TA & training, outreach, and interagency coordination. (suggested # of outcomes is

Review and categorize accomplishments produced to date into outputs and types of outcomes (short-term, intermediate and long-term).

N I

D R

RN

I D

R R



Research

45

Retrospective Outcome Identification: Retrospective Outcome Identification: Implementation Strategies (Cont’d.)Implementation Strategies (Cont’d.)Retrospective Outcome Identification: Retrospective Outcome Identification: Implementation Strategies (Cont’d.)Implementation Strategies (Cont’d.)

Link accomplishments to newly formulated performance goals and identify the types of data available and the data elements needed to demonstrate progress towards outcomes and document the contribution of AT Act project activities and outputs.

Revise web-based annual performance reporting form for FY 2004 to collect necessary data and rethink how existing data can be analyzed to provide evidence of progress or achievement of outcomes.

Link accomplishments to newly formulated performance goals and identify the types of data available and the data elements needed to demonstrate progress towards outcomes and document the contribution of AT Act project activities and outputs.

Revise web-based annual performance reporting form for FY 2004 to collect necessary data and rethink how existing data can be analyzed to provide evidence of progress or achievement of outcomes.

N I

D R

RN

I D

R R



Research

46

Common Weaknesses in Prospective & Common Weaknesses in Prospective & Retrospective Outcomes IdentificationRetrospective Outcomes IdentificationCommon Weaknesses in Prospective & Common Weaknesses in Prospective & Retrospective Outcomes IdentificationRetrospective Outcomes Identification

1. Lack of specificity in “what” and “where”

2. Too many outcomes and expressed too narrowly or too broadly

3. Outcomes expressed as speculations vs. as expectations

4. Confusion between levels of the performance spectrum (i.e., activities vs. outputs and outcomes and short-term vs. intermediate outcomes)

5. Missing supporting links between outputs and short-term outcomes

7. Missing evidence of outcomes

1. Lack of specificity in “what” and “where”

2. Too many outcomes and expressed too narrowly or too broadly

3. Outcomes expressed as speculations vs. as expectations

4. Confusion between levels of the performance spectrum (i.e., activities vs. outputs and outcomes and short-term vs. intermediate outcomes)

5. Missing supporting links between outputs and short-term outcomes

7. Missing evidence of outcomes

N I

D R

RN

I D

R R



Research

47

Exercise: What’s Missing in the Exercise: What’s Missing in the Following AT Act Project Outcomes? Following AT Act Project Outcomes?

•Outreach: Within the last grant cycle, State X provided technical assistance to 4, 127 individual and demonstration of loan of equipment to 1,415 individuals.

• Training: State X provided 102 training sessions attended by 3,8288 which resulted in increased knowledge levels that have prompted systems to address AT needs and responsibilities within policies and procedures.

N I

D R

RN

I D

R R



Research

48

Exercise: Illustration of Reformulated Exercise: Illustration of Reformulated Outcome StatementOutcome Statement

•Original Statement Systems Change Intermediate Outcome: State AT program worked with the state election officials on the HAVA state plan which resulted in adoption of the FEC 2002 access standards for voting machines.

• Suggested Reformulation: Conduct legislative initiatives that result in the development (short-term outcome) and adoption by states (intermediate outcome) of new standards for voting machines to increase access for adults with moderate to severe disabilities.

N I

D R

RN

I D

R R



Research


Lessons Learned:

The Utility and Challenges of Implementing Performance

Measurement

Lessons Learned:

The Utility and Challenges of Implementing Performance

Measurement

N I

D R

RN

I D

R R



Research

50

Benefits of Utilizing Benefits of Utilizing Performance Management Performance Management

Benefits of Utilizing Benefits of Utilizing Performance Management Performance Management

Provides a common language and integrated framework to facilitate: Strategic planning & program design Identification of performance goals Program integration – coordinates the

contributions of multiple projects and tasks Objective measurement of progress towards

goals Program monitoring & improvement Program accountability -- documentation of

results, and Communication of success stories

Provides a common language and integrated framework to facilitate: Strategic planning & program design Identification of performance goals Program integration – coordinates the

contributions of multiple projects and tasks Objective measurement of progress towards

goals Program monitoring & improvement Program accountability -- documentation of

results, and Communication of success stories

N I

D R

RN

I D

R R



Research

51

Challenges of Applying Performance Challenges of Applying Performance Management Concepts & Tools Management Concepts & Tools

Challenges of Applying Performance Challenges of Applying Performance Management Concepts & Tools Management Concepts & Tools

Developing a “managing for results” mindset. Incorporating requirements of performance

measurement into the traditional standards of excellence for service delivery, consumer involvement, and productivity.

Learning to re-express core program objectives into the language of outcomes-oriented performance goals (outputs and outcomes).

Deciding what evidence you need to collect and how to measure performance and determine how your program is working (i.e., quality and relevance/utility) and if you are achieving planned outputs and anticipated outcomes.

Developing a “managing for results” mindset. Incorporating requirements of performance

measurement into the traditional standards of excellence for service delivery, consumer involvement, and productivity.

Learning to re-express core program objectives into the language of outcomes-oriented performance goals (outputs and outcomes).

Deciding what evidence you need to collect and how to measure performance and determine how your program is working (i.e., quality and relevance/utility) and if you are achieving planned outputs and anticipated outcomes.

N I

D R

RN

I D

R R



Research

52

Challenges of Applying Performance Challenges of Applying Performance Management (Cont’d)Management (Cont’d)

Challenges of Applying Performance Challenges of Applying Performance Management (Cont’d)Management (Cont’d)

Integrating the tools of performance management into the operation of your program.

Using the information gathered to improve program management, evaluate performance, achieve results, and provide evidence to support claims of short-term & intermediate outcomes.

Documenting results and communicating successes using a “claims-making” process based on qualitative judgments of quantitative and/qualitative data

Integrating the tools of performance management into the operation of your program.

Using the information gathered to improve program management, evaluate performance, achieve results, and provide evidence to support claims of short-term & intermediate outcomes.

Documenting results and communicating successes using a “claims-making” process based on qualitative judgments of quantitative and/qualitative data

N I

D R

RN

I D

R R



Research

53

Lessons Learned from Implementing Lessons Learned from Implementing Performance Management in Business Performance Management in Business

& Government& Government

Lessons Learned from Implementing Lessons Learned from Implementing Performance Management in Business Performance Management in Business

& Government& Government

“What gets measured gets done.” (Drucker)

“If you don’t measure results, you can’t tell success from failure “ (C. Mindel). According to OMB, the number one cause of

federal agencies getting a low score on the new Program Assessment Reporting Tool (PART) is NOT having results-oriented performance measures to gauge the outcomes of a program (http://www.transparentgovernment.org/tg/analysis.htm).

“What gets measured gets done.” (Drucker)

“If you don’t measure results, you can’t tell success from failure “ (C. Mindel). According to OMB, the number one cause of

federal agencies getting a low score on the new Program Assessment Reporting Tool (PART) is NOT having results-oriented performance measures to gauge the outcomes of a program (http://www.transparentgovernment.org/tg/analysis.htm).

N I

D R

RN

I D

R R



Research

54

Lessons Learned from Implementing Lessons Learned from Implementing Performance Management (Cont’d)Performance Management (Cont’d)

Lessons Learned from Implementing Lessons Learned from Implementing Performance Management (Cont’d)Performance Management (Cont’d)

Accountability for results involves a partnership -- NIDRR can’t satisfy GPRA and PART alone! To succeed in the new era of performance-based budgeting, NIDRR needs data on accomplishments from grantees like you.

Key Assumption: All grantees have accomplishments; what missing are the performance goals and outcome data.

Key Assumption: “If you can demonstrate results, you can win public support” (C. Mindel).

Accountability for results involves a partnership -- NIDRR can’t satisfy GPRA and PART alone! To succeed in the new era of performance-based budgeting, NIDRR needs data on accomplishments from grantees like you.

Key Assumption: All grantees have accomplishments; what missing are the performance goals and outcome data.

Key Assumption: “If you can demonstrate results, you can win public support” (C. Mindel).

N I

D R

RN

I D

R R



Research

Thank youThank you Thank youThank you