1
Effectiveness of the Intervention October 14th, 2011
National Product Service Center
Rent-A-Center, Inc.
Table of Contents:
Introduction ........................................................................................................................... 2
1.0 TAP (Topic, Audience & Purpose) ................................................................................... 2
1.2 Topic ........................................................................................................................... 3 1.3 Audience ..................................................................................................................... 3 1.4 Purpose ....................................................................................................................... 3 1.5 Format ......................................................................................................................... 3
2.0 Problem .......................................................................................................................... 3
2.1 Learners ...................................................................................................................... 4 2.2 Instructors .................................................................................................................... 4 2.3 Core problems with existing instructional design .......................................................... 4 2.4 Overview of existing skills, knowledge and limitations.................................................. 4
3.0 Effectiveness of the Intervention ..................................................................................... 5
3.1 Training Content .......................................................................................................... 5 3.2 Training Design ........................................................................................................... 5 3.3. Training Instructor (Facilitator) .................................................................................... 6 3.4 Training Results ........................................................................................................... 6 3.5 Summary ..................................................................................................................... 6
Appendix A. Training Evaluation Questionnaire .................................................................... 8
Appendix B. Training Evaluation Questionnaire Results ..................................................... 10
Appendix C. Kirkpatrick’s Four Levels of Evaluation .......................................................... 11
Figure 1. Instructional System Design Concept Map ...................................................... 12
References ......................................................................................................................... 13
Effectiveness of the Intervention – Product Service Systems Orientation
Tony F. Fuller
2
Effectiveness of the Intervention October 14th, 2011
Introduction
One of the outcomes of instructional design (ID) must be an assessment of the impact of the
instruction design. For those that utilize the ADDIE design model this is the last step in the
process, E – Evaluation. The assessment of the impact of the ID is discussed in Section
3.0, Effectiveness of the Intervention. Section 1 – Topic, Audience and Purpose (TAP) and
Section 2, Problem will reintroduce the audience to the scope of the original User Needs
Analysis.
1.0 TAP (Topic, Audience & Purpose)
“My Client” is a National Product Service Division (NPS) that is part of a $3 billion
organization. NPS consists of 22 facilities geographically located in the continental United
States, Hawaii and Puerto Rico. These facilities will conduct more than a million repairs
annually on consumer goods, primarily electronics, for more than 3,000 company stores.
This division was formally created as part of the company in 1992.
Having created service systems for another large retailer during the mid to late
1980’s and then personally having a key role in the formation of the service division for my
client during the early 1990’s I am very familiar the activities and processes within the
service division.
The problem being faced is that over the years the processes within the service
division have not changed significantly, and there has been very little turn over in personnel
in the previous two decades. Over the years formal training within the division on their core
systems had not been necessary given their low turnover rates. With the company looking
at emerging markets the service division would like to create a formal training program on its
core system.
I have personally and professionally known the Senior Vice President of the service
division since the mid 1980’s. This will help facilitate the process since I do not have to
establish my creditability in getting the job done. Conversely, I know the SVP of the service
3
Effectiveness of the Intervention October 14th, 2011
division to be very direct and goals oriented which will help to eliminate scope creep and will
insure whatever resources and time necessary, from the national product service division, to
make this a successful endeavor will be provided.
1.2 Topic Product Service Systems Orientation
1.3 Audience Attendees will be a mixed blending of work associates, supervisors and managers.
Participants will be motivated to learn, as they fully understand this training is designed to
make their work flow easier and less manual than it currently is. Each individual will be
looking for how it integrates into their specific day to day activities.
1.4 Purpose At the conclusion of training individuals should have a firm grasp of how to navigate the
system, understanding of the characteristics or functionality of the system, which section of
the applications apply to their job role and basic knowledge of how to utilize these portions of
the application. This course is not designed to make user “experts”, rather participants will
have the fundamental knowledge required to use the application as part of their normal daily
tasks.
1.5 Format The course of instruction is to be conducted as an instructor led program of learning.
Instructor Led Training (ILT) means that the course of instruction will not only be conducted
by a person, but that person will be available to monitor; review and assess; and answer any
questions or concerns throughout the course.
2.0 Problem
4
Effectiveness of the Intervention October 14th, 2011
Needs will be reviewed and presented in two different sections: (1) learners and (2)
instructors.
2.1 Learners The problem being faced is that over the years the processes within the service division
have not changed significantly, and there has been very little turn over in personnel in the
previous two decades. Over the years formal training within the division on their core
systems had not been necessary given their low turnover rates. With the company looking
at emerging markets the service division would like to create a formal training program on its
core system.
2.2 Instructors The instructor(s) cannot just “teach the class”. The instructor must not only be an instructor,
but must also be a subject matter expert (SME); an individual capable of demonstrating their
knowledge of the lesson plan. The benefit of the instructor being a SME is the learning can
go beyond the scope of the objectives and goals. While in most situations it is desirable for
the ID to provide only that which the client has asked for, in this case it is desirable for the
instructor to be part of the client base. This will add a level of creditability to completion of
the goals and objectives.
2.3 Core problems with existing instructional design Fundamentally there is not a problem with the existing instructional design (ID). –The root
problem is there is NOT any existing instructional design.
2.4 Overview of existing skills, knowledge and limitations When conducting a review of the existing skills and knowledge there does not appear to be a
problem in this area. However, there is a limiting factor on a much broader scale; one that
extends beyond the individual level of the prescribed audience. This limitation has to do with
preparation for the future, and growth of the national product service division.
5
Effectiveness of the Intervention October 14th, 2011
While the existing audience has not been exposed to the positive impacts of ID, they
have achieved a sufficient level of knowledge to perform their prescribed tasks.
3.0 Effectiveness of the Intervention
Effectiveness of the intervention must be measured both qualitatively and quantitatively. For
this several instruments were designed to measure the learners’ capabilities and the
effectiveness of the instructional design (ID). An end of course Training Evaluation
Questionnaire (Appendix A) was administered to each of the learners at the completion of
the course. Each of a series of questions was proposed to the learner utilizing a Likert
scale. This scale was graded 1 through 5. With 1 = “Strongly disagree”, or the lowest, most
negative impression; 3 = “Neither agree nor disagree”, or an adequate impression; and 5 =
“Strongly agree”, or the highest, most positive impression. Percentages may not sum to
100% due to rounding to the nearest percentage.
3.1 Training Content As determined by the results of the end of course Training Evaluation Questionnaire
(Appendix B) Training Content received an 89% acceptance rating at level 5 and an 7%
acceptance rating at level 4 for an overall acceptance rating of 96%. 4% of the learners
rated the Training Content at level 3 with no respondents replying at levels 1 or 2. All nine
learners responded to each of the three questions presented. This section consisted of
three post implementation questions.
3.2 Training Design As determined by the results of the end of course Training Evaluation Questionnaire
(Appendix B) Training Design received an 38% acceptance rating at level 5 and an 56%
acceptance rating at level 4 for an overall acceptance rating of 94%. 4% of the learners
rated the Training Design at level 3 with an additional 2% of the learners giving a rating at
level 2. This section consisted of five post implementation questions.
6
Effectiveness of the Intervention October 14th, 2011
3.3. Training Instructor (Facilitator) The Training Instructor (Facilitator) received a 100% rating at level five as compiled from the
results of the Training Evaluation Questionnaire (Appendix B). This section consisted of two
post implementations questions.
3.4 Training Results As determined by the results of the end of course Training Evaluation Questionnaire
(Appendix B) Training Results received an 89% acceptance rating at level 5 and an 11%
acceptance rating at level 4 for an overall acceptance rating of 100%. This section consisted
of two post implementation questions.
3.5 Summary Overall 69% of the learners rated the intervention at a level 5 with 27% responding at level 4.
This resulted in an overall acceptable rating of 96% with 3% responding at level 3 and 1%
responding unfavourably at level 2. No learners indicated N/A on their answers.
While the administration and interpretation of an end of course evaluation can be
used to determine the overall impact of an intervention it is usually acceptable to receive an
early indicator of an acceptable level of performance.
It should be noted that under Kirkpatrick’s Four Level Training Evaluation Model
(Appendix C) that only the requirements for a level one evaluation were completed.
However, the conclusion for level one is the ID was effective and adequately supported the
client’s needs.
Additionally, it should be noted that as part of the post implementation assessment
several forms of assessment materials were utilized to determine a level of understanding
and competencies from the learners. These instruments were not utilized as part of the
impact assessment. Instruments of this type are best utilized as part of a level two
evaluation. As part of the level two evaluation the results from these instruments should be
compared to the same instruments being given to the learners at a prescribed interval, i.e.,
7
Effectiveness of the Intervention October 14th, 2011
30 days, 60 days, etc. Normally, level two would be completed in the form of pre and post-
tests. The purpose of level two is to measure “Did the learners learn anything?”. This will
allow for the assessment or impact measurement at level two.
8
Effectiveness of the Intervention October 14th, 2011
Appendix A. Training Evaluation Questionnaire
Training Evaluation Questionnaire
Course Name: _________________________________ Training Location: _________________________________ Participant Name (optional): _________________________ Date: ________________ Job Title: ________________________________________ Years in present position? <1 1-3 3-5 5+ INSTRUCTIONS Please circle your response to the items. Rate aspects of the Training on a 1 to 5 scale: 1 = “Strongly disagree, “or the lowest, most negative impression 3 = “Neither agree nor disagree, “or an adequate impression 5 = “Strongly agree,” or the highest, most positive impression Choose N/A if the item is not appropriate or not applicable to this Training. Your feedback is sincerely appreciated. Thank you.
TRAINING CONTENT (circle your response to each item.)
1. I was well informed about the objectives of this training.
1 2 3 4 5 N/A
2. This training lived up to my expectations.
1 2 3 4 5 N/A
3. The content is relevant. 1 2 3 4 5 N/A TRAINING DESIGN (circle your response to each item.) 4. The training objectives were clear to
me. 1 2 3 4 5 N/A
5. The training activities stimulated my learning.
1 2 3 4 5 N/A
9
Effectiveness of the Intervention October 14th, 2011
Appendix A. Training Evaluation Questionnaire (cont)
TRAINING DESIGN (cont) (circle your response to each item.) 6. The activities in this training gave me
sufficient practice and feedback. 1 2 3 4 5 N/A
7. The difficulty level of this training was appropriate.
1 2 3 4 5 N/A
8. The pace of this training was appropriate.
1 2 3 4 5 N/A
TRAINING INSTRUCTOR (FACILITATOR) (circle your response to each item.) 9. The instructor was well prepared. 1 2 3 4 5 N/A 10. The instructor was helpful. 1 2 3 4 5 N/A
TRAINING RESULTS (circle your response to each item.) 11. I accomplished the objectives of this
training. 1 2 3 4 5 N/A
12. I will be able to use what I learned in this training.
1 2 3 4 5 N/A
COMMENTS: ________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
10
Effectiveness of the Intervention October 14th, 2011
Appendix B. Training Evaluation Questionnaire Results
TRAINING CONTENT Likert Scale
1 2 3 4 5 N/A
1. I was well informed about the objectives of this training. 1 1 7
2. This training lived up to my expectations. 1 8
3. The content is relevant. 9
RAW SCORES 0 0 1 2 24 0
RAW PERCENTAGES 0% 0% 4% 7% 89% 0%
TRAINING DESIGN Likert Scale
1 2 3 4 5 N/A
1. The training objectives were clear to me.
1 8
2. The training activities stimulated my learning.
1 1 1 6
3. The activities in this training gave me sufficient practice and feedback.
7 2
4. The difficulty level of this training was appropriate.
1 8
5. The pace of this training was appropriate.
8 1
RAW SCORES 0 1 2 25 17 0
RAW PERCENTAGES 0% 2% 4% 56% 38% 0%
TRAINING INSTRUCTOR (FACILITATOR) Likert Scale
1 2 3 4 5 N/A
1. The instructor was well prepared. 9
2. The instructor was helpful. 9
RAW SCORES 0 0 0 0 18 0
RAW PERCENTAGES 0% 0% 0% 0% 100% 0%
TRAINING RESULTS Likert Scale
1 2 3 4 5 N/A
1. I accomplished the objectives of this training.
1 8
2. I will be able to use what I learned in this training.
1 8
RAW SCORES 0 0 0 2 16 0
RAW PERCENTAGES 0% 0% 0% 11% 89% 0%
TOTAL RESULTS Likert Scale
1 2 3 4 5 N/A
RAW SCORES 0 1 3 29 75 0
RAW PERCENTAGES 0% 1% 3% 27% 69% 0%
11
Effectiveness of the Intervention October 14th, 2011
Appendix C. Kirkpatrick’s Four Levels of Evaluation
The four-levels of evaluation consist of (Kirkpatrick, 1994): 1. Reaction – how learners react to the learning process
Evaluation at this level measures how the learners react to the training. This level is
often measured with attitude questionnaires that are passed out after most training
classes. This level measures one thing: the learner’s perception (reaction) of the
course).
2. Learning – the extent to which the learners gain knowledge and skills
This is the extent to which participants change attitudes, improve knowledge, and
increase skill as a result of participating in the learning process. It addresses the
question: Did the participants learn anything? The learning evaluation requires some
type of post-testing to ascertain what skills were learned during the training. In addition,
the post-testing is only valid when combined with pre-testing, so that you can
differentiate between what they already knew prior to training and what they actually
learned during the training program.
3. Behavior – capability to perform the learned skills while on the job
This evaluation involves testing the students capabilities to perform learned skills while
on the job, rather than in the classroom. Level three evaluations can be performed
formally (testing) or informally (observation). It determines if the correct performance is
now occurring by answering the question, “Do people use their newly acquired learning
on the job?”
4. Results – includes such items as monetary, efficiency, moral, etc.
This is the final results that occur. It measures the training program's effectiveness, that
is, “What impact has the training achieved?” These impacts can include such items as
monetary, efficiency, moral, teamwork, etc.
12
Effectiveness of the Intervention October 14th, 2011
(www.nwlink.com/~donclark/hrd/ahold/isd.html)
Figure 1. Instructional System Design Concept Map
13
Effectiveness of the Intervention October 14th, 2011
References
Kirkpatrick, D. L. (1994). Evaluating Training Programs. San Francisco: Berrett-Koehler Publishers, Inc.