determining the productivity of instructional systems
TRANSCRIPT
The Journal of Human Resource and Adult Learning * November 2006 1
Determining the Productivity of Instructional Systems
Dr. Philip McGee, Program Coordinator of the Masters in Human Resource Development,
Clemson University, South Carolina
ABSTRACT
The Productivity Model described in this article is applied in such a way as to determine the productivity of
instructional systems by describing a simple to use strategy for assessing and evaluating instructional systems. Data
generated by the model enables instructional designers to determine not only the effectiveness and efficiency of any
instructional system, but also its overall productivity. Data generated by the Productivity Model may be used not only to
assess an instructional system, but may also be used as an evaluation tool enabling instructional designers to pin-point
where and when to take corrective actions.
INTRODUCTION
If you enter the keywords “instructional design” + “productivity” into any Internet search engine you will
generate a number of links to online papers and sites extolling the benefits and advantages of using instructional system
principles and techniques to design and develop training programs and materials that produce predictable learning
outcomes. The same holds true for searching online libraries and databases of journal articles.
However, when one examines the literature related to evaluating training systems, more often than not, one is
referred to the work of Donald Kirkpatrick (1998) and Jack Phillips (1997). Unfortunately, neither Kirkpatrick’s Four
Levels of Evaluation nor Phillips’ work determining the ROI of training systems always provide the type of information
needed by instructional designers to evaluate and improve the instructional systems for which they are responsible.
This is not to say the topic of instructional design and productivity has not been addressed in the past. Ivor Davies
wrote on the issue of designing instructional systems that are both effective and efficient in his book Instructional
Technique (1981). Davies based his writings on the definitions developed by Peter Drucker (1974) who wrote that
“Efficiency is concerned with doing things right.” While, “Effectiveness is doing right things.”
Now, thirty years later, William Rothwell and H. C. Kazanas in their book Mastering the Instructional Design
Process (2003) state that “These terms have no universally accepted definitions.” They continue their discussion of
effectiveness and efficiency by quoting Drucker, as did Davies, and then press on to discuss that instructional design is
guided by a model of human performance.
It is at this point that I would like to offer a new evaluation model using the concepts of effectiveness and
efficiency, and to explore how this model can be used as a strategy for improving the productivity of instructional
systems.
A MODEL FOR DETERMINING THE PRODUCTIVITY OF INSTRUCTIONAL SYSTEMS
The Oxford English Dictionary (1989) states that productivity is the rate of output per unit of input or output/input =
productivity. As with calculating the mean for a set of scores, valuable insights can be lost when data is reduced to a
single measure. It is with this concern in mind that I would like to describe the Productivity Model and how it can be
applied by instructional designers and trainers. Let us begin with the assumption that any system can be described as
being composed of three basic elements as illustrated below by Figure 1.
The Journal of Human Resource and Adult Learning * November 2006 2
Figure 1. Elements common to all systems
First, all systems produce results. Granted, the results produced by any given system may at times be difficult to
predict. Secondly, to achieve results, something must happen; something must be done. In other words, activity must
take place. Third, in order to have activity, resources must be used. These three elements of a system hold true whether
the system is mechanical, electrical, biological, financial, social or educational.
The next step to understanding this model is to examine the dynamic relationships that exist between the elements
of a system. We will start with the relationship between results and activity.
Effectiveness
It can be said that something is effective only when an activity produces a predetermined result. For example,
when an instructional designer develops a program of instruction, the designer should first strive to create an
instructional system that produces not only the desired results, but also a system that produces the desired results with a
high degree of predictability. It is only when a system does what it is supposed to, with a high degree of predictability,
that we able to say that it is effective.
Figure 2. The Effectiveness Dimension
Effectiveness is concerned with "how well" something works i.e., produces predetermined desired results. This
concern for "how well" is the basis for the concept we call quality. As designers of systems, we must strive first for
effectiveness. For without it, there is little reason to proceed with the design of any system.
Efficiency
Efficiency is the dynamic relationship that exists between resources and an activity. Efficiency is concerned with
quantity and asks the question, "How much?" However, this is a dangerous question to ask. Dangerous in that if we cut
resources too much, we run the risk of producing poor results.
Figure 3. The Efficiency Dimension
Resources + Activity = Results
Resources + Activity = Results
Effectiveness Dimension Qualitative Measures Concern for Quality
Asks: How Well?
Resources + Activity = Results
Efficiency Dimension Quantitative Measures
Concern for Quality Asks: How much?
The Journal of Human Resource and Adult Learning * November 2006 3
This is not to say that we should not be concerned with resources and their associated costs. We should, because
within every system there is an optimum balance between resources and activity, and activity and results. This optimum
balance is known as productivity.
Figure 4. Productivity defined as a measure of effectiveness and efficiency
Productivity
We can see this balance between effectiveness and efficiency in the illustration below in Figure 5. System (A),
which is neither effective nor efficient, cannot be said to be productive. System (B), while very effective, cannot be said
to be productive, because it is not efficient. System (C), while efficient, is not effective. Therefore it cannot be
considered productive. Only system (D), which is both effective and efficient, can be said to be productive. In
application, it is possible to set performance standards for any system in terms of effectiveness and efficiency, and to
determine corrective courses of action when evaluative data is generated and graphed.
Figure 5. The Slope of Productivity
Applying the Model to Improving the Productivity of Training Systems
If we are to measure and improve the productivity of a training system, we must focus our attention on two
distinct measures. The first is the quality of the instruction, i.e., what should be taught? And the second is the quantity
Efficiency Dimension Quantitative Measures
Concern for Quality
Asks: How much?
Resources + Activity = Results Productivity
Effectiveness Dimension Qualitative Dimension
Concern for Quality
Asks: How well?
Efficiency Low
Low
High
High
Eff
ecti
ven
ess
Slope of Productivity
A
B
C
D
The Journal of Human Resource and Adult Learning * November 2006 4
of resources needed to deliver the instruction, i.e., by what means should the curriculum be taught? This
interrelationship is illustrated below in Figure 6.
Figure 6. Curriculum development versus Instructional Tactics
Let me expand on this concept. Instructional quality is the major concern of curriculum development. Curriculum
development answers the question, “What should be taught?” It is during the development of the instructional
curriculum that a performance standard for the curriculum is established and program content is identified.
Instructional quality is often measured by how well participants are able to predictably achieve the objectives of the
program. This data is most often gained through traditional testing procedures. For example, a particular curriculum
may produce results, wherein an average participant achieves a 95% level of competency on the material presented.
Figure 7. Average competency level achieved versus delivery cost per participant
Instructional quantity, on the other hand, falls into the domain of instructional tactics, which is the process of
determining and selecting the most efficient method and media for delivering a program of instruction (curriculum).
The goal of instructional tactics is to answer the question, “By what means should the curriculum be taught?” Again, a
performance standard must be established by which to measure this dimension. Common standards are money, time,
instructional staff, equipment required, i.e., instructional resources.
Instructional Tactics Asks: By what means should the
curriculum be taught?
Resources + Activity = Results Productivity
Curriculum Development
Asks: What should be taught?
100%
Delivery Cost per Participant
Avera
ge C
om
pete
ncy L
ev
el
Ach
iev
ed
95%
90%
85%
$150
$125
$100
$75
Instructional System
The Journal of Human Resource and Adult Learning * November 2006 5
In order to determine the productivity of an instructional system, we must consider both the results produced by
the curriculum and the instructional resources required to deliver the curriculum. Keep in mind that productivity is a
ratio or composite measure of both the effectiveness and efficiency of a system. For example, in Figure 7 above we
have an instructional system in which 95% of the participants achieve competency using a media/method to deliver the
training which has a per participant cost of $125. By simply plotting the data using an X-Y graph and noting where the
data points are located in relation to the Slope of Productivity, it is possible to determine the effectiveness, efficiency,
and productivity for any given system.
Taking Corrective Action
The power of this evaluation strategy is that it enables trainers and instructional developers to identify where they
should take corrective action. Using Figure 8, it can be seen that to improve the instructional system, attention should be
focused upon decreasing the cost of delivering the training.
Figure 8. Determining corrective action using the Productivity Model
In other situations, given different data, trainers and designers may want to improve their curriculums. This brings us to
the following:
Factors that Influence the Effectiveness of an Instructional System (Curriculum Development)
- Needs Assessments
- Assessment of Learners
- Analysis of Work Settings
- Job, Task, or Content Analysis
- Statements of Performance Objectives
- Performance Measurements (test items)
- Sequence Performance Objectives
Factors that Influence the Efficiency of an Instructional System (Instructional Tactics)
- Instructional Techniques
- Designs for Instructional Materials
100%
Delivery Cost per Participant
Avera
ge C
om
pete
ncy L
ev
el A
ch
ieved
95%
90%
85%
$150 $125 $100 $75
Instructional System
The Journal of Human Resource and Adult Learning * November 2006 6
- Media
- Delivery mode / medium
- Instructional Resources, i.e., money, time, instructional staff, or required equipment.
Comparing Two or More Instructional Systems
Trainers and instructional developers are often asked to compare and contrast two or more instructional systems
and to make a recommendation. Using the Productivity Model, this becomes a simple task as illustrated below.
Let us say that we have two instructional systems that deliver the same information and skills. 85% of the trainees,
who use Instructional System A, achieve competency for an average delivery cost of $100 per participant. However,
95% of the trainees who use Instructional System B achieve competency using media and methods which have a per
participant cost of $125. To determine which of the two instructional systems is the most productive we need only to
graph out the two instructional systems as illustrated in Figure 9.
Figure 9. Comparing two or more instructional systems using the Productivity Model
Because Instructional System B falls higher and closer to the line we call the "Slope of Productivity," we can determine
it is the most productive. It should be noted that using this strategy, it is possible to compare any number instructional
system to determine which is the most productive.
A Grounded Model
The concepts and relationships detailed in this article are based upon established principles from a variety of
disciplines. The basic elements of a system: resources, activity and results, are described throughout the literature on
systems theory and are sometimes referred to simply as an input/output model.
Accountants and financial people, who often speak of return on investment, have known the relationship between
resources and results: (ROI). ROI is where returns are results and investments are costs, and in this way, are able to
determine the health of a business enterprise (activity).
Physicists and engineers recognized long ago that in order to describe the performance of various phenomena and
systems, they had to be described in terms of dynamic relationships between two variables. For example, miles per
gallon or feet per second.
100%
Delivery Cost per Participant
Avera
ge C
om
pete
ncy L
ev
el A
ch
ieved
95%
90%
85%
$150 $125 $100 $75
System B
System A
System A: 85%, $100 System B: 95%, $125
The Journal of Human Resource and Adult Learning * November 2006 7
Thomas Gilbert, a founding father of the human performance field, developed the First Leisurely Theorem, which
says that worth is equal to value divided by costs. In other words, activity adds value to resources (cost) and results in
something of greater worth.
Peter Drucker pointed out in the 1970s that effectiveness was doing right things, while efficiency was doing
things right. In the field of education, these concepts were expanded upon by Ivor Davies and applied to decision-
making concerning instructional methods.
CONCLUDING THOUGHTS
Because the Productivity Model is grounded in general systems theory, it can be used for a wide range of design
and evaluation activities. The author has not only used the Productivity Model to analyze various training systems, but
has used to the Productivity Model to analyze production processes, as a starting point for group discussions that
focused on improving processes ranging from sales, to product development, to distribution and logistics. In the
classroom, the Productivity Model has proved useful as a teaching tool by providing students with a framework on
which to arrange problems solving concepts that in turn help to facilitate the development of their critical thinking skills.
REFERENCES
Davies, Ivor K. (1981). Instructional Technique. New York: McGraw-Hill Book Company.
Drucker, Peter F. (1974). Management Tasks, Responsibilities, Practices. New York: Harper & Row Publishers.
Gilbert, Thomas F. (1978). Human Competence: Engineering Worthy Performance. New York: McGraw-Hill Book Company.
Kirkpatrick, Donald (1998). Evaluating Training Programs (2nd ed.). San Francisco: Berrett-Koehler Publishers, Inc.
Simpson, J.A. and Weiner, E.S.C. (1989) Oxford English Dictionary (2nd ed., Vols. 1-20). New York: Oxford University Press.
Phillips, Jack J. (1997). Handbook of Training Evaluation and Measurement Methods (3rd ed.). Houston: Gulf Publishing Company.
Rothwell, William J. and Kazanas, H. C. (2003). Mastering the Instructional Design Process (3rd ed.). San Francisco: Pfeiffer.