part 2. evaluation formative summative why evaluate ? makes good economic sense allows for...
TRANSCRIPT
![Page 1: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/1.jpg)
Part 2
![Page 2: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/2.jpg)
Last Week Why evaluate Results-based approach Purposes and uses of evaluation Reviewed Kirkpatrick model
![Page 3: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/3.jpg)
This Week More on Kirkpatrick model Types of data collection Types of data Evaluation instruments Tips for Survey/Questionnaire
* Will vary from note taking H.O.
![Page 4: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/4.jpg)
Warm Up Exercise
F
![Page 5: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/5.jpg)
Levels of Evaluations
• Donald Kirkpatrick - 4 Levels
I. Reaction – Were the participants pleased with interventionII. Learning – What did the participants learnIII. Behavior – Did the participants change their behavior based on what was learning?IV. Results – Did the change in behavior positively affect the organization
![Page 6: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/6.jpg)
Exercise # 3
• What are some advantages and limitations in each of the four levels of Kirkpatrick’s evaluation model?
(Use note-taking handout)
![Page 7: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/7.jpg)
Collect Post Intervention Data(Handout 3)
1. Surveys2. Questionnaires3. On the job observation4. Post intervention interviews5. Focus groups6. Program assignments7. Action plans8. Performance contracts9. Follow up session10. Performance monitoring
![Page 8: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/8.jpg)
Exercise # 4
• Using the 10 items in the Post Intervention Data handout (3):
Each group describe (fabricate) a situation where you might gather post intervention data for a level 2/3/4 evaluation. Be prepared to explain your rationale.
![Page 9: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/9.jpg)
Hard Data
Hard data can be group into four categories
1. Output – of the work unit
2. Quality – how well produced or serviced
3. Cost – improvement in costs
4. Time – savings
![Page 10: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/10.jpg)
Soft Data
• Work habits – absenteeism, tardiness, violations
• Climate – number of grievances, complaints, job satisfaction
• Satisfaction – Favorable reactions, employee loyalty, increased confidence
![Page 11: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/11.jpg)
Exercise # 5
• List some advantages and disadvantages / limitations when collecting hard and soft data.
• (Use Notetaking H.O. p. 3/4)
![Page 12: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/12.jpg)
Evaluation Instruments
Validity – does it measure what it is supposed to measure
• Content validity – how well does the instrument measure the content/objectives of the program
• Construct validity –How well does it measure the construct (abstract variable such as KSA)
• Concurrent validity – How well does the instrument measure up against other instruments
• Predictive validity – how well can it predict future behavior
![Page 13: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/13.jpg)
Part 3
![Page 14: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/14.jpg)
Last Week
More on Kirkpatrick model Types of data collection Types of data
![Page 15: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/15.jpg)
This Week
Developing evaluation instruments
The survey process
* New Handouts
![Page 16: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/16.jpg)
Exercise # 5
• List some five things you would do to enhance the chances of getting a good number returns for surveys/ questionnaires.
![Page 17: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/17.jpg)
Survey Process -- Tips
• Communicate – in advance– the purpose
• Signed introductory letter
• Explain who’ll see data• Use anonymous input?
![Page 18: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/18.jpg)
More Tips• Keep it simple• Simplify the
response process– Bubble format– SASE
• Utilize local support
![Page 19: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/19.jpg)
More Tips
• Consider incentives• Use follow-up
reminders• Send a copy of the
results to the participants
![Page 20: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/20.jpg)
Action PlanningAction Planning
• Most common type of follow-up assignments.
• Developed by participants.
• Contains detailed steps to accomplish measurable objectives.
• Shows what is to be done, by whom, when.
• Must be monitored
![Page 21: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/21.jpg)
Action Plans• Communicate the action plan requirement early and
explain its value (avoids resistance)
• Describe the action-planning process at the beginning of the program (outline)
• Teach the action-planning process
• Allow time to develop the plan
• Have the facilitator approve the action plans
• Require participants to assign a monetary value for each improvement (helps ROI later)
![Page 22: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/22.jpg)
Action Plans• Ask participants to isolate the effects of the
program
• Ask participants to provide a confidence level
for estimates
• Require action plans to be presented to the groups by participants (peer review) if possible
• Explain the follow up mechanism
• Collect action plans
• Summarize the data and calculate ROI
![Page 23: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/23.jpg)
Converting Data to Monetary Benefits
• Focus on a unit of measure• Determine the value of each unit• Calculate the change in performance• Determine an annual amount for
the change
• Calculate the total value of the improvement
![Page 24: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/24.jpg)
Ways to Put Value on Units
• Cost of quality
• Converting Employee time
• Using Historical Costs
• Using Internal and External Experts
• External Databases
• Estimates from the participants
• Estimates from Supervisors
• Estimates from Senior Managers
• Using HRD staff estimates
![Page 25: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/25.jpg)
Credibility• Source of the Data• Source for the study• Motives of evaluators• Methodology of the study• Assumptions made
in the analysis• Realism of the outcome data• Types of data• Scope of analysis
![Page 26: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/26.jpg)
Guidelines for Study
• Credible and reliable sources for estimates
• Present material in an unbiased, objective way
• Fully explain methods (step by step)
• Define assumptions and compare to other studies
• Consider factoring or adjusting output values when they appear unrealistic
• Use hard data whenever possible
![Page 27: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/27.jpg)
Identifying intangible Measures
(Not based upon monetary values)
• Employee satisfaction
• Stress reduction
• Employee turnover
• Customer satisfaction, retention
• Team effectiveness
![Page 28: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/28.jpg)
Determining Costs• Collect costs on every intervention• Costs will not be precise (hard to be perfect)• Be practical - work with accounting department• Define which costs to collect, categories, sources• Computerize• Cost accumulation (track accounts)• Cost estimation (forumulas - page 227)• Fully load with all costs possible - be truthful• Overhead, benefits, perpherial costs, etc
![Page 29: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/29.jpg)
Data Analysis
• Statistics (use a professional)
• Use terms appropriately (ie, Significant difference)
• Statistical deception (erroneous conclusions)
![Page 30: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/30.jpg)
Return on Investment
• Compares costs to benefits
• Complicated
• Usually annualized
• Business casespecific
• Communicate the formulaused
![Page 31: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/31.jpg)
I. Reaction and Planned Action – measure’s participants reactions and plans to change
II. Learning – Measures KSA
III. Job Applications – Measures change of behavior on the job and specific use of the training material
IV. Business results – Measures impact of the program
V. Return on investments – Measures the monetary value of the results and costs for the program, usually expressed as a percentage
Phillips ROI Framework
![Page 32: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/32.jpg)
Level 1 Reaction Participants
Level 2 Learning Participants
Level 3 JobApplications
ImmediateManagers
Level 4 BusinessImpact
Immediate/SeniorManagers
Level 5 Return onInvestment
Senior ManagersExecutives
Evaluation as a Customer Satisfaction Tool
![Page 33: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/33.jpg)
From Level 4 to Level 5Requires Three Steps:
1. Level 4 data must be converted to monetary values
2. Cost of the intervention mustbe tabulated
3. Calculate the formula
![Page 34: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/34.jpg)
ROI Process ModelROI Process Model
CollectData
IsolateEffects ofTraining,
ConvertData toMonetaryValve
Calculate theReturn onInvestment
IdentifyIntangibleBenefits
TabulateProgram Costs
![Page 35: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/35.jpg)
ROI Formula
ROI (%) =
Net Program Benefits
Program Costs
X 100
![Page 36: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/36.jpg)
Two Methods
• 1. Cost/Benefit RatioAn early model that compares the intervention’s costs to its benefits in a ratio form. For every one dollar invested in the intervention, X dollars in benefits were returned.
• 2. ROI FormulaUses net program benefits divided by costs, and expressed as a percent.
![Page 37: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/37.jpg)
Cost / Benefit
CBR = Program Benefits
Program Costs
![Page 38: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/38.jpg)
ROI Formula
ROI (%) =
Net Program Benefits
Program Costs
X 100
![Page 39: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/39.jpg)
Cautions With Using ROI
• Make sure needs assessment has been completed
• Include one or more strategies for isolating the effects of training
• Use the reliable, credible sources in making estimates
• Be conservative when developing benefits and costs
• Use caution when comparing the ROI in training and development with other financial returns
• Involve management in developing the return
• Approach sensitive and controversial issues carefully
• Do not boast about a high return (internal politics)
![Page 40: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/40.jpg)
Implementation Issues
• Identify an internal champion (cheerleader)
• Develop an implementation leader
• Assign responsibilities so everyone will know their assigned tasks and outcomes
• Set targets (annual)
• Develop a project plan, timetable
• Revise/Develop Policies and Procedures (Page 367)
• Assess the climate – gap analysis, SWOT, barriers
![Page 41: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/41.jpg)
Preparing Your Staff
• Involve the staff in the process
• Using Evaluation Data as a Learning Tool
• Identify and remove obstacles(complex, time, motivation, correct use of results)
![Page 42: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/42.jpg)
ROI Administration
Which programs to select?Large target audiencesImportant to corporate strategiesExpensiveHigh VisibilityComprehensive needs assessment
![Page 43: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/43.jpg)
ROI Administration
• Reporting ProgressStatus meetings (facilitated by expert)Report progressAdd evaluation areas
• Establish Discussion Groups
• Train the Management Tool
![Page 44: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/44.jpg)
Timing of Evaluation
1. During the program
2. Time series – multiple measures
3. Post tests – timing
![Page 45: Part 2. Evaluation FORMATIVE SUMMATIVE Why Evaluate ? Makes good economic sense Allows for accountability Answers the increased scrutiny of Budgets Fulfills](https://reader030.vdocuments.us/reader030/viewer/2022032605/56649e7c5503460f94b7e8b9/html5/thumbnails/45.jpg)
Questionnaire Content IssuesQuestionnaire Content Issues
• Progress with objectives
• Action plan status
• Relevance of intervention
• Use of program materials
• Knowledge/skill application
• Skill frequency
• Changes in the work unit
• Measurable improvements/accomplishments
• Monetary impact
• Confidence level
• Improvement linked with the intervention
• Investment perception
• Linkage with output measures
• Barriers
• Enablers
• Management support
• Other solutions
• Target audience recommendations
• Suggestions for improvement