introduction to performance measurement for americorps*vista 1-800-548-3646 washington dulles...

Introduction toIntroduction toPerformance MeasurementPerformance Measurement

for AmeriCorps*VISTAfor AmeriCorps*VISTA

1-800-548-3646 www.projectstar.org1-800-548-3646 www.projectstar.org

Washington Dulles Hilton Washington Dulles Hilton

September 12-14, 2005September 12-14, 2005

Day One - Afternoon Day One - Afternoon SessionSession

This session will cover: Six Steps of Performance

Measurement (PM) Types of Results: Outputs,

Intermediate and End Outcomes Use of the Logic Model as a

Planning Tool

Performance measurementPerformance measurementis a six-step process foris a six-step process for

regularly measuringregularly measuringyour project results.your project results.

Performance Measurement:The Six Steps

Performance Measurement Six Steps

The logic model describes The logic model describes your AmeriCorps*VISTA your AmeriCorps*VISTA

project.project.Use it to identify your Use it to identify your

project activities and what project activities and what you will measure to gauge you will measure to gauge the success of your project. the success of your project.

1. Develop Logic Model

& Project Plan

2. DevelopPM Plan

Identifying your data Identifying your data sources and data sources and data

collectors.collectors.Determine a timeline Determine a timeline for data collection, for data collection,

analysis, and analysis, and reporting reporting

& Project Plan

2. DevelopPM Plan

3. Developand Test

Instruments

Adapt existing Adapt existing instruments or instruments or

develop your own; develop your own; pilot test to make sure pilot test to make sure

instruments collect instruments collect valid, appropriate valid, appropriate

informationinformation

& Project Plan

2. DevelopPM Plan

3. Developand Test

Instruments4. Collect

Go out and collect Go out and collect your data (e.g., fill your data (e.g., fill

out attendance out attendance sheets, administer sheets, administer surveys, conduct surveys, conduct

interviews)interviews)

& Project Plan

2. DevelopPM Plan

3. Developand Test

5. AnalyzeData

Aggregate and Aggregate and analyze data; analyze data;

generate generate summary info summary info

(totals, averages, (totals, averages, etc)etc)

& Project Plan

2. DevelopPM Plan

3. Developand Test

5. AnalyzeData

6. ReportResults

Write progress Write progress report; include report; include

your performance your performance measurement measurement

resultsresults

& Project Plan

2. DevelopPM Plan

3. Developand Test

5. AnalyzeData

6. ReportResults

ContinuousProgram

Improvement

What Are Results?What Are Results?Outputs and outcomes your AmeriCorps*VISTA project hopes to achieve by engaging in service activitiesOutputs: Services delivered and products completed by VISTA membersOutcomes: Changes or benefits that occur as a result of services delivered

Intermediate Outcomes: Changes that are short of the “final” result you hope to achieveeach yearEnd Outcomes: The most significantchanges that your AmeriCorps*VISTAproject hopes to achieve each year

Sample ScenarioSample Scenario

Community-based organization (CBO)Community-based organization (CBO)has existing mentoring program, buthas existing mentoring program, butfaces challenges in recruiting and faces challenges in recruiting and retaining volunteer mentors.retaining volunteer mentors.

CBO unable to deliver high-quality service CBO unable to deliver high-quality service to youth.to youth.

VISTA members will serve with CBO to VISTA members will serve with CBO to develop system to recruit, train and develop system to recruit, train and support volunteer mentoring program.support volunteer mentoring program.

Position descriptions for volunteer mentors Written procedures for screening volunteer

mentors Tracking database Training curriculum for new volunteer mentors Written procedures to provide ongoing support

to volunteer mentors

Outputs answer the question, “How much service did we do?” They do NOT answer the question, “What changed as a result of the service?”

OutputsOutputs

Trained agency personnel demonstrate

understanding of how to use new system

and procedures (new knowledge and skills).

Agency personnel use new systems and

procedures to recruit, train, and support

volunteer mentors (changes in behavior).

Intermediate outcomes are important milestones on the way to achieving end outcomes. Achieving intermediate outcomes can lead to attaining end outcomes.

Intermediate OutcomesIntermediate Outcomes

More youth are served by volunteer mentors under the new system than were served under the previous system (increased beneficiaries served).

More youth experience a close and stable relationship with a caring adult, because a greater proportion of volunteer mentors complete their service commitments (improved quality of service).

End outcomes are changes that reflect the overarching goals of your AmeriCorps*VISTA project.

End OutcomesEnd Outcomes

Examples of ResultsExamples of Results

OUTPUTS INTERMEDIATE OUTCOME(S)

END OUTCOMES

Individual Development Account trainings providedParticipants

IDA program participants accumulate savings to use for purchase of assets

IDA program participants use accumulated savings to purchase assets they had planned to purchase

Training sessions providedParent Network Collaborative meetings

Network referral system developed

Network agencies expand quality and range of services

ExerciseExercise

Identifying Types of ResultsIdentifying Types of Results

Benefits of Developing a Logic ModelBenefits of Developing a Logic Model Helps to build consensus about

program goals and desired results

Clarifies relationship between keyprogram elements (inputs, activities, outputs, and outcomes)

Brings “fuzzy goals” down to earth

Begins to identify things to measure

Provides solid foundation for developing project plan

AmeriCorps*VISTA Logic ModelAmeriCorps*VISTA Logic Model

A Logic Model = a planning tool that diagrams how you intend to achieve your AmeriCorps*VISTA project goals by

combining resources (inputs) and service activities to produce specific results (outputs and

outcomes) that address poverty-related community needs.

Community Need

Community Need = Unresolved poverty-related problem or

issue in the community that your VISTA project will

address

Community Need

Goals Inputs

Activities

Outputs

Intermediate Outcomes

EndOutcomes

End Outcomes=The most significant changes achieved

by your projectAnswer the questions:

“So what”? “What changed?”

Goals = Broad purposes toward which your

VISTA project is directed

Community Need

Inputs= Resources your VISTA project will need

to engage in the identified activities and produce outputs and outcomes

(human, financial, organizational, community)

Community Need

Goals Inputs

Activities=What VISTA members do to address the community need and achieve desired

results

Community Need

Goals Inputs

Activities

Outputs:Reflect the amount of service completed

by VISTA members.Outputs do not answer the questions:

“what changed as a result of the work?” or “how did people or the organization

benefit?”

Community Need

Goals Inputs

Activities

Outputs

Intermediate Outcomes=Initial changes achieved by your project; milestones on the way to achieving end

outcomes

Community Need

Goals Inputs

Activities

Outputs

Community Need

Goals Inputs

Activities

Outputs

EndOutcomes

End Outcomes=The most significant changes achieved

by your projectAnswer the questions:

“So what”? “What changed?”

Community Need

Goals Inputs

Activities

Outputs

EndOutcomes

Program Planning

Intended Results

Community Need

Goals Inputs

Activities

Outputs

EndOutcomes

Logic Model for a Mentoring Program Logic Model for a Mentoring Program (Year 3)(Year 3)

Children of incarcerated parents lack social and emotional support for positive youth development that can help to break the cycle of poverty.

Develop, enhance systemsto recruit, train, and support volunteer mentors.

2 VISTAs

Supervisor

Program staff

Partners

Assist with implementa-tion, assess performance of new systems, identify promising practices.

Policiesand guidance revised for volunteer recruitment and sustain-ability systems.

Improved recruitment: more youth are served by more volunteer mentors.

Improved retention: more volunteer mentors complete service.

Improved program quality: more volunteer mentors form closer bonds with youth; mentoring relationships last longer.

ExerciseExercise

The Logic Model PuzzleThe Logic Model Puzzle

ExerciseExercise

Develop a Logic Model Develop a Logic Model

for Your Projectfor Your Project

End of Day ReviewEnd of Day Review

++What Worked Well What Worked Well

About Today?About Today?

∆∆How Could Tomorrow How Could Tomorrow

Be Made Even Be Made Even Better?Better?

Day Two – Morning SessionDay Two – Morning Session

This session will cover:This session will cover:

1.1. Moving from a Logic Model to the Moving from a Logic Model to the Project PlanProject Plan

2.2. CNCS Requirements for VISTACNCS Requirements for VISTA

3.3. Components of a Project PlanComponents of a Project Plan

4.4. Indicators, Targets, InstrumentsIndicators, Targets, Instruments

AmeriCorps*VISTA Performance Measurement Requirements

One project plan containing one or more One project plan containing one or more goalsgoals

Identify activities and results for each goalIdentify activities and results for each goal Results are written as outputs, intermediate Results are written as outputs, intermediate

outcomes, or end outcomesoutcomes, or end outcomes Label 3-5 of these results as “performance Label 3-5 of these results as “performance

measures”measures”– One OutputOne Output– One Intermediate OutcomeOne Intermediate Outcome– One End OutcomeOne End Outcome

Identify indicators, targets, and instruments Identify indicators, targets, and instruments for each performance measurefor each performance measure

VISTA Project Plan VISTA Project Plan ComponentsComponents

Goals (Same) Activities (formerly “Objectives”) Results (New)

Output(s)Intermediate Outcome(s)End Outcome(s)

AmeriCorps*VISTA Project PlanAmeriCorps*VISTA Project Plan

Column AGoals, Activities, and Results

Column BDate(s)

Column CProject Reporting

GOAL 1

Activities

Results Outputs Intermediate Outcomes End Outcomes

GOAL 2

Activities

GOAL 3 Etc.

Column BDate(s)

GOAL 1

Activities

GOAL 2

Activities

GOAL 3 Etc.

Column BDate(s)

GOAL 1

Activities

GOAL 2

Activities

GOAL 3 Etc.

From Logic Model to Project PlanFrom Logic Model to Project Plan

Column BDate(s)

Goal(s)

Activities

Results

Outputs

End Outcomes

Community Need

Goals Inputs

Activities

Outputs

EndOutcomes

Multi-Year Goals,Multi-Year Goals,Annual Activities and Annual Activities and ResultsResults

GoalsGoals are are multi-yearmulti-year..

Activities and results may vary from one Activities and results may vary from one

year to the next.year to the next.

Each year the project builds on the previous Each year the project builds on the previous

year’s activities and results.year’s activities and results.

Only list Only list activitiesactivities and and resultsresults for for current current

yearyear..

ExerciseExercise

Identify One Goal for Your ProjectIdentify One Goal for Your Project For that Goal, identify the Activities For that Goal, identify the Activities

for Year One and Possible Results:for Year One and Possible Results:– OutputsOutputs– Intermediate outcome(s)Intermediate outcome(s)– End outcome(s)End outcome(s)

AmeriCorps*VISTA Project PlanAmeriCorps*VISTA Project PlanColumn AGoals, Activities, and Results

Column BDate(s)

GOAL 1

GOAL 2

Activities

Results Output “Performance Measure” Indicator Target Instrument Intermediate Outcome “Performance Measure” Indicator Target Instrument End Outcome “Performance Measure” Indicator Target Instrument

AmeriCorps*VISTA Project PlanAmeriCorps*VISTA Project PlanColumn AGoals, Activities, and Results

Column BDate(s)

GOAL 1

GOAL 2

Activities

Results Output “Performance Measure” Indicator Target Instrument Intermediate Outcome “Performance Measure” Indicator Target Instrument End Outcome “Performance Measure” Indicator Target Instrument

What is an Indicator?What is an Indicator?

Information you collect to determineInformation you collect to determineprogress toward achieving resultsprogress toward achieving results

Concrete and measurableConcrete and measurable

Answers the question, “What is myAnswers the question, “What is my evidence that something has occurred?”that something has occurred?”

Output indicator usually begins with Output indicator usually begins with “Number of...”“Number of...”

Outcome indicator usually begins with Outcome indicator usually begins with “Percent of...”“Percent of...”

Example of Result and IndicatorsExample of Result and Indicators

Desired Result: Desired Result: Increase service to clientsIncrease service to clients

Possible Indicators:Possible Indicators:

- Greater number of clients served Greater number of clients served (number of clients served, compared to last (number of clients served, compared to last year)year)

- Clients receive more servicesClients receive more services(number of services provided, per client, (number of services provided, per client, compared to last year)compared to last year)

- Clients receive services of greater value Clients receive services of greater value (dollar value of services provided, per client, (dollar value of services provided, per client, compared to last year)compared to last year)

What is a Target?What is a Target? Describes amount of change youDescribes amount of change you

anticipate achieving each year dueanticipate achieving each year dueto efforts of VISTA members.to efforts of VISTA members.

Based on indicator – specific, measurable Based on indicator – specific, measurable amount of change that will signify amount of change that will signify success

Outcome Target:Outcome Target:– Who will experience change? will experience change?– How many people will experience change? people will experience change?– How much change will occur (compared to change will occur (compared to

baseline)?baseline)?– Over Over what time period will change occur (one will change occur (one

year)?year)?

TargetTarget describes amount/level of describes amount/level ofchange you anticipate achieving inchange you anticipate achieving inone year due to efforts of VISTA members.one year due to efforts of VISTA members.

Based on Based on indicatorindicator (concrete, measurable) (concrete, measurable)

Rely on Rely on experienceexperience and knowledge of and knowledge of community community to judge how much change to to judge how much change to expect in one year.expect in one year.

Keep in mind your Keep in mind your resources.resources.

Be realistic!Be realistic!

Formulating Annual TargetsFormulating Annual Targets

Indicator and Target Indicator and Target ExamplesExamplesOutput Outcome

Result Volunteer Volunteer database database developed.developed.

Improve Improve volunteer base.volunteer base.

Indicator Volunteer Volunteer database database developed.developed.

Number ofNumber of new new volunteers volunteers recruited.recruited.

Target OneOne volunteer volunteer database database developed.developed.

300300 new new volunteers volunteers recruited.recruited.

What is an Instrument?What is an Instrument?

Document or form used to collectDocument or form used to collectinformation from data sourceinformation from data source

Data source: Person, place or group that Data source: Person, place or group that provides you with information to measure provides you with information to measure indicatorsindicators

Examples: Attendance rosters, tally sheets, Examples: Attendance rosters, tally sheets, questionnaires, checklistsquestionnaires, checklists

Adapt existing instruments or develop your Adapt existing instruments or develop your ownown

Advice about InstrumentsAdvice about Instruments

Make sure instrument will measure Make sure instrument will measure

indicatorsindicators

Consider who will Consider who will administeradminister instrument instrument

Consider who will Consider who will complete complete instrumentinstrument

Consider Consider whenwhen to administer instrument to administer instrument

Consider data Consider data accessibilityaccessibility

Project Plan: Performance MeasuresProject Plan: Performance MeasuresColumn AGoals, Activities, and Results

Column BDate(s)

ACTIVITIES (Year 1)

RESULTS (Year 1)Outputs:1. Volunteer mentor recruitment systems are developed.2. Agency personnel receive orientation in new systems

“Performance Measure”Indicator: # agency personnel oriented Target: 2 F/T and 4 P/T agency personnel oriented Instrument: Orientation Log

Qtr 1–3Qtr 1–3

Column BDate(s)

ACTIVITIES (Year 1)

Intermediate Outcome: Agency personnel understand how to use new systems. “Performance Measure” Indicator: # personnel who correctly identify staff roles and who can find info in new databaseTarget: 5 personnel can recruit, screen and train mentorsInstrument: Mentoring System User Checklist

Qtr 1–3Qtr 1–3

Column BDate(s)

ACTIVITIES (Year 1)

Intermediate Outcome: Agency personnel understand how to use new systems. “Performance Measure” Indicator: # personnel who correctly identify staff roles and who can find info in new databaseTarget: 5 personnel can recruit, screen and train mentorsInstrument: Mentoring System User Checklist

End Outcome: Agency personnel use new systems. “Performance Measure”Indicator: # agency personnel using # new systems Target: Four agency personnel, two systemsInstrument: Year-end Agency Interview

Qtr 1–3Qtr 1–3

ExerciseExercise

Identifying Indicators, Identifying Indicators, Targets and How Measured Targets and How Measured

for Outcomesfor Outcomes

ExerciseExercise

Add Enhancements to Your Add Enhancements to Your Project Plan ResultsProject Plan Results

Day Two – Afternoon Day Two – Afternoon SessionSession

1.1. Selecting InstrumentsSelecting Instruments

2.2. Data Collection StrategiesData Collection Strategies

3.3. Data Aggregation and AnalysisData Aggregation and Analysis

4.4. ReportingReporting

Strategies to Ensure Successful Strategies to Ensure Successful ReportingReporting

From the beginning…

Align work plan and instruments Plan for collecting the information Understand how to analyze data Anticipate how you will report

What Do I Report On?What Do I Report On? Actual Activities

Actual Output– Actual target as compared to anticipated target– How data collected

Actual Intermediate Outcome– Actual target as compared to anticipated target– How data collected

Actual End Outcome– Actual target as compared to anticipated target– How data collected

Data Collection Issues

Can you access the data source (beneficiaries, organizations)?

Does an instrument (site or project) exist? Are they willing to give you data? (burden,

capacity) Can you get the data when you need it? Are your helpers (e.g. VISTAs) trained? Do you have a mechanism to collect

summarized data from sites?

Methods

Logs, Tally Sheets, Attendance Rosters Survey, Questionnaire (Check list, Index, Scale) Interview (Interview Guide)

Service Recipients Interview Expert/Provider/Key Informant

Observation (Observation Guide) Case Study (Measurable change of a case over time) Focus Group (Focus Group Guide) Secondary Data (Existing records and information) Test / Experimental Designs

Instrument Considerations Connect to Project Plan results

– Measures the identified PM Aim for appropriate method

– Measures the correct population (data source) Lower burden of use

– e.g. Captive audience, (part of project)– Balance between “best” method and

“manageable” method Assure that sites have instruments or

summary tool

ExerciseExercise

Instrument and Data Instrument and Data Collection Collection Review and DiscussionReview and Discussion

Data AnalysisData Analysis

Quantitative FrequenciesFrequencies RangesRanges PercentsPercents MeansMeans Pre-post comparisonPre-post comparison

Qualitative Content analysisContent analysis

– Pre-existing Pre-existing themesthemes

– Emergent themesEmergent themes

ExerciseExercise

Data Aggregation and Data Aggregation and Analysis Analysis

Reporting Tips

Provide evidence. Avoid hearsay and generalizations.

Be candid about your efforts. Don’t claim too little. Don’t claim too much.

Tie report to Project Plan. State results in same format as original target in Project Plan. Pay attention to enhancements: Pay attention to enhancements:

indicator, target, and instrumentsindicator, target, and instruments

Report TemplateReport TemplateReporting Format for Each Reporting Format for Each

Performance MeasurePerformance Measure

Actual TargetActual Target::

1) 1) Compare the actual to the anticipated target stated in Compare the actual to the anticipated target stated in Column A. Column A.

2) 2) Met target? If target not met, explain. Met target? If target not met, explain.

3) 3) Include any additional information about your result.Include any additional information about your result.

How Data Collected: How Data Collected:

1) 1) Describe how the result was measured. Describe how the result was measured.

Reporting Each ResultReporting Each ResultActual Target:

1) Compare the actual target to the anticipated target stated in Column A. Interpret results; explain what it means

2) Met target? If target not met, explain. Include numbers not just percentages.

Explicitly state that you did (not) meet target

3) Include any additional information about your result.Describes challenges, next steps

Describing Your ResultsDescribing Your Results

Actual Target: Actual Target: 1) Compare the actual target to the anticipated target 1) Compare the actual target to the anticipated target stated in Column A. [Always include an actual number] stated in Column A. [Always include an actual number]

“10 interview respondents (86% of collaborative) identified at least three beneficial practices resulting from the partner collaboration. This exceeds our target of one project improvement. The top two included an increase in clients served and…”

Reporting Format for Each Reporting Format for Each ResultResult

How Data Collected:

1) Describe how the result was measured. What was used to track the result? If applicable, how was the instrument administered?

(When was it administered? How often?) If applicable, include total number approached as well

as those who responded.

Describing Your ResultsDescribing Your ResultsHow Data Collected:

1) Describe how the result was measured

“During the 2nd and 3rd weeks of April, the Project Supervisor, and two VISTAs conducted telephone interviews during the afternoon and evening…”

“...Of the 12 collaborative partners called, 10 completed the interview that assessed their practices resulting from the collaborative…”

Describing Your Work Before Describing Your Work Before Results Are AvailableResults Are Available

“...We don’t have any results yet.”

What do I Report When I Have What do I Report When I Have “Nothing to Report” Yet? “Nothing to Report” Yet?

We are…..Development/piloting of instruments

Introducing instrument to sites

Gaining access to data

Training data collectors

Challenges and proposed solutions

Administering pre-survey data. (If you have baseline data, report it early!)

Say something! Don’t wait.

Describing Your Work Before Describing Your Work Before Results Are Available (Improved)Results Are Available (Improved)

How Data Collected: 1) Describe how the result was measured

“...In preparation for implementing performance measurement, we drafted a telephone interview protocol containing six questions designed to assess partner collaborative practices. During the first week of March, two collaborative members reviewed the draft and six VISTAs were trained to conduct interviews...”

Day Three – Morning Day Three – Morning SessionSession

1.1. ReportingReporting

2.2. Using the Results of Performance Using the Results of Performance Measurement Measurement

3.3. eGrants and VISTAeGrants and VISTA

4.4. Closing - Receive box lunch and Closing - Receive box lunch and departure departure

ExerciseExercise

Progress Report Critique Progress Report Critique Using ChecklistUsing Checklist

ExerciseExercise

Communicating SuccessCommunicating SuccessWays to Use PM Results to Strengthen Ways to Use PM Results to Strengthen

and Promote Your Projectand Promote Your Project

Additional Resources:Additional Resources: Contact your CNCS State OfficeContact your CNCS State Office Contact Project STARContact Project STAR

• www.projectstar.orgwww.projectstar.org• 1-800-548-36561-800-548-3656• star@aiweb.comstar@aiweb.com

introduction to performance measurement for americorps*vista 1-800-548-3646 washington dulles...

logic model project

project results

year slide

pm plan

reporting slide

new volunteer mentors

volunteer mentors outputs

vista members outcomes

Documents

03 477 3646 gb - trovaprezzi.it

dulles middle school home of the dulles vikings. dulles...

residence inn by marriott dulles airport at dulles 28 centre...

americorps boot camp - california · americorps boot camp...

dulles schools newsletter - loudoun county public...

dulles international airport, dulles, va to 4:06 pm - 5:11...

indian standard 3646

eisenhower, dulles fail

e-911psap epl 3646 effective dates - mississippi · pdf...

americorps program guide | california americorps

welcome to the national service family! · americorps vista...

americorps - volunteer florida · americorps state and...

inspiron 3646 owner's manual - gfk...

dulles middle school

can we ever improve the airport...

[xls] · web view9433.25 401009371 1656.5 401009376 883.25...

americorps 101 key americorps grant terminology for ...

transportation from washington dulles …transportation from...

dulles schools newsletter · 2019-02-07 · 1 dulles...

about americorps - ywca 2019...about americorps • about...