module 7 g&m
DESCRIPTION
TRANSCRIPT
T E A M 4
MODULE 7: CHAPTER 17 G&M
What is evaluation to you? Our text lists five program issues for evaluation:● Quality● Suitability● Effectiveness● Efficiency● Importance
How do these components relate to your place of work? Can you provide examples?
EVALUATION PRE-DISCUSSION
�Evaluation-estimating value
�Two steps1.Compare results and objectives
2.Appraise or judge the value of the differences assessed
What is the difference between “measurement” and “evaluation”?
EVALUATION
�A systematic process, reliant upon multiple skills
● �Collecting information
● Interpreting data�● Drawing conclusions�● Communicating outcomes�
The authors list two major purposes of evaluations. What are they?
EVALUATION
ADDIE MODEL
1. Identify All Clients and Stakeholders and Clarify Their Needs2. Identify the Performance Improvement Initiative to Be Evaluated3. Identify and Clarify the Purposes for the Evaluation4. Determine the Critical Research Questions That the Evaluation
Must Address5. Develop an Evaluation Design6. Analyze Resources and Constraints7. Determine the Best Data Collection Methods8. Plan Reporting and Communications Actions
Can one person explain a step of the process from their own experience at work?
THE PROCESS
KIRKPATRICK’S FOUR LEVELS OF EVALUATION MODEL
Most of us said that our organizations utilized the Kirkpatrick model of evaluation, but in a limited capacity (reaction and learning)
What do you think would have been an effective method of incorporating Levels 3-4 (behavior and impact) ?
KIRKPATRICK
● Helps training professionals to understand evaluation in a systematic way
● A straightforward system for discussing training outcomes
● recognizes that single outcome measures cannot adequately reflect complexity of organizational training programs (Bates, 2004)
Are there any other advantages of Kirkpatrick’s model that you would like to add?
KIRKPATRICK ADVANTAGES
● Model is incomplete- oversimplified view of training effectiveness. Other factors influence training outcomes.o learning culture of the organizationo organizational or work unit goals and valueso interpersonal supporto climate for learning transfero adequacy of material resources
(Bates, 2004)
KIRKPATRICK LIMITATIONS
● Assumption of Causal Linkages- model assumes that the criteria represent a causal relationship between the levels of evaluationo research has failed to confirm causal linkageso “ if training is going to be effective , it is
important that trainees react favorably” and “without learning, no change in behavior will occur” (Kirkpatrick, 1994)
(Bates, 2004)
KIRKPATRICK LIMITATIONS
● Incremental Importance of Information- model assumes that each level of evaluation provides data that is more informative than the lasto perception that establishing level 4 results provide
the most useful informationo “in practice, however, the weak conceptual
linkages inherent in the model and resulting data it generates do not provide an adequate basis for this assumption” (Bates, 2004)
KIRKPATRICK LIMITATIONS
•“Evaluation is the systematic process of delineating, obtaining, reporting, and applying descriptive and judgmental information about some object’s merit, worth, probity [moral correctness], feasibility, safety, significance, or equity"(Stufflebeam & Shinkfield, 2007)
EVALUATION
Context- Input-Process-Product
STUFFLEBEAM’S CIPP MODEL (1983)
What needs to be done? ContextHow should it be done? InputIs it being done? ProcessDid it succeed? Product
STUFFLEBEAM’S CIPP MODEL
Uses for CIPP model:● Conduct a needs analysis● Evaluation of alternatives for addressing needs● Monitor design/implementation of interventions● Helps to examine outcomes of intervention
regarding impact to the organization
STUFFLEBEAM’S CIPP MODEL
● Addresses concerns of decision-makers for justifying the investment in interventions/initiatives
● Provides a framework for comparing alternatives for future investments
COST BENEFIT MODEL (KEARSLEY, 1986)
● Provides the expected benefit or return on investment
● Expressed as a percentage or in actual dollars● Identify the benefits of intervention ($), divide by
the cost (%) or subtract costs
RETURN ON INVESTMENT
● Helps to decide how to best allocate resources● Disadvantage: most interventions or initiatives
provide benefits which are hard to quantify
RETURN ON INVESTMENT
1. What do you think is the difference between the Cost Benefit Model and the ROI model?
2. Provide examples of when these models should be used.
QUESTIONS:
7 step model1. Determine purpose, objectives, participants (who
wants this information)2. Assess information needs3. Consider proper protocol4. Describe population to be studied, selet subjects5. Identify other variables6. Formulate a study design7. Formulate a management plan
FORMATIVE EVALUATIONS
● Most useful for evaluating instruction● May be used to for performance improvement
and change interventions
FORMATIVE EVALUATIONS
“...evaluative inquiry can not only be a means of accumulating information for decision making and action..but that it also be equally concerned with questioning and debating the value of what we do in organizations” (Preskill and Torres, 1999)
EVALUATIVE INQUIRY
Evaluative inquiry is a way of fostering individual learning and team learning within an organization , about issues that are critical to its purpose and what it values (Parsons, 2009)
EVALUATIVE INQUIRY
● Collaboration● Organizational learning and change● Links learning and performance● Diverse perspectives
EVALUATIVE INQUIRY
● A study is needed to evaluate and redesign an online master’s degree program consisting of 12 courses in informatics
● Educators are concerned about the quality of online education courses
● Meaningful assessment is essential for improving quality of such programs
CASE STUDY FOR EVALUATION
Considering the evaluation models that we have discussed:
1. Which model(s) would you consider appropriate for this case? Why?
2. Design an evaluation program to include Steps 1-5 as described by Rothwell and Kazanas ( Gilley & Maycunich, p. 430-432)
CASE STUDY FOR EVALUATION
Bates, R. (2004) A critical analysis of evaluation practice; the Kirkpatrick model and the principle
of beneficence. Evaluation and program planning,27(3),341-34
Parsons, B. (2009) Evaluative inquiry for complex times. OD Practitioner, 41(1).
Preskill, H. & Torres, R. T.(1999). building capacity for organizational learning through evaluative inquiry. Evaluation, 5(1) 42-60
REFERENCES