determining the productivity of instructional systems

7

Click here to load reader

Upload: vutuong

Post on 04-Jan-2017

214 views

Category:

Documents


1 download

TRANSCRIPT

Page 1: Determining the Productivity of Instructional Systems

The Journal of Human Resource and Adult Learning * November 2006 1

Determining the Productivity of Instructional Systems

Dr. Philip McGee, Program Coordinator of the Masters in Human Resource Development,

Clemson University, South Carolina

ABSTRACT

The Productivity Model described in this article is applied in such a way as to determine the productivity of

instructional systems by describing a simple to use strategy for assessing and evaluating instructional systems. Data

generated by the model enables instructional designers to determine not only the effectiveness and efficiency of any

instructional system, but also its overall productivity. Data generated by the Productivity Model may be used not only to

assess an instructional system, but may also be used as an evaluation tool enabling instructional designers to pin-point

where and when to take corrective actions.

INTRODUCTION

If you enter the keywords “instructional design” + “productivity” into any Internet search engine you will

generate a number of links to online papers and sites extolling the benefits and advantages of using instructional system

principles and techniques to design and develop training programs and materials that produce predictable learning

outcomes. The same holds true for searching online libraries and databases of journal articles.

However, when one examines the literature related to evaluating training systems, more often than not, one is

referred to the work of Donald Kirkpatrick (1998) and Jack Phillips (1997). Unfortunately, neither Kirkpatrick’s Four

Levels of Evaluation nor Phillips’ work determining the ROI of training systems always provide the type of information

needed by instructional designers to evaluate and improve the instructional systems for which they are responsible.

This is not to say the topic of instructional design and productivity has not been addressed in the past. Ivor Davies

wrote on the issue of designing instructional systems that are both effective and efficient in his book Instructional

Technique (1981). Davies based his writings on the definitions developed by Peter Drucker (1974) who wrote that

“Efficiency is concerned with doing things right.” While, “Effectiveness is doing right things.”

Now, thirty years later, William Rothwell and H. C. Kazanas in their book Mastering the Instructional Design

Process (2003) state that “These terms have no universally accepted definitions.” They continue their discussion of

effectiveness and efficiency by quoting Drucker, as did Davies, and then press on to discuss that instructional design is

guided by a model of human performance.

It is at this point that I would like to offer a new evaluation model using the concepts of effectiveness and

efficiency, and to explore how this model can be used as a strategy for improving the productivity of instructional

systems.

A MODEL FOR DETERMINING THE PRODUCTIVITY OF INSTRUCTIONAL SYSTEMS

The Oxford English Dictionary (1989) states that productivity is the rate of output per unit of input or output/input =

productivity. As with calculating the mean for a set of scores, valuable insights can be lost when data is reduced to a

single measure. It is with this concern in mind that I would like to describe the Productivity Model and how it can be

applied by instructional designers and trainers. Let us begin with the assumption that any system can be described as

being composed of three basic elements as illustrated below by Figure 1.

Page 2: Determining the Productivity of Instructional Systems

The Journal of Human Resource and Adult Learning * November 2006 2

Figure 1. Elements common to all systems

First, all systems produce results. Granted, the results produced by any given system may at times be difficult to

predict. Secondly, to achieve results, something must happen; something must be done. In other words, activity must

take place. Third, in order to have activity, resources must be used. These three elements of a system hold true whether

the system is mechanical, electrical, biological, financial, social or educational.

The next step to understanding this model is to examine the dynamic relationships that exist between the elements

of a system. We will start with the relationship between results and activity.

Effectiveness

It can be said that something is effective only when an activity produces a predetermined result. For example,

when an instructional designer develops a program of instruction, the designer should first strive to create an

instructional system that produces not only the desired results, but also a system that produces the desired results with a

high degree of predictability. It is only when a system does what it is supposed to, with a high degree of predictability,

that we able to say that it is effective.

Figure 2. The Effectiveness Dimension

Effectiveness is concerned with "how well" something works i.e., produces predetermined desired results. This

concern for "how well" is the basis for the concept we call quality. As designers of systems, we must strive first for

effectiveness. For without it, there is little reason to proceed with the design of any system.

Efficiency

Efficiency is the dynamic relationship that exists between resources and an activity. Efficiency is concerned with

quantity and asks the question, "How much?" However, this is a dangerous question to ask. Dangerous in that if we cut

resources too much, we run the risk of producing poor results.

Figure 3. The Efficiency Dimension

Resources + Activity = Results

Resources + Activity = Results

Effectiveness Dimension Qualitative Measures Concern for Quality

Asks: How Well?

Resources + Activity = Results

Efficiency Dimension Quantitative Measures

Concern for Quality Asks: How much?

Page 3: Determining the Productivity of Instructional Systems

The Journal of Human Resource and Adult Learning * November 2006 3

This is not to say that we should not be concerned with resources and their associated costs. We should, because

within every system there is an optimum balance between resources and activity, and activity and results. This optimum

balance is known as productivity.

Figure 4. Productivity defined as a measure of effectiveness and efficiency

Productivity

We can see this balance between effectiveness and efficiency in the illustration below in Figure 5. System (A),

which is neither effective nor efficient, cannot be said to be productive. System (B), while very effective, cannot be said

to be productive, because it is not efficient. System (C), while efficient, is not effective. Therefore it cannot be

considered productive. Only system (D), which is both effective and efficient, can be said to be productive. In

application, it is possible to set performance standards for any system in terms of effectiveness and efficiency, and to

determine corrective courses of action when evaluative data is generated and graphed.

Figure 5. The Slope of Productivity

Applying the Model to Improving the Productivity of Training Systems

If we are to measure and improve the productivity of a training system, we must focus our attention on two

distinct measures. The first is the quality of the instruction, i.e., what should be taught? And the second is the quantity

Efficiency Dimension Quantitative Measures

Concern for Quality

Asks: How much?

Resources + Activity = Results Productivity

Effectiveness Dimension Qualitative Dimension

Concern for Quality

Asks: How well?

Efficiency Low

Low

High

High

Eff

ecti

ven

ess

Slope of Productivity

A

B

C

D

Page 4: Determining the Productivity of Instructional Systems

The Journal of Human Resource and Adult Learning * November 2006 4

of resources needed to deliver the instruction, i.e., by what means should the curriculum be taught? This

interrelationship is illustrated below in Figure 6.

Figure 6. Curriculum development versus Instructional Tactics

Let me expand on this concept. Instructional quality is the major concern of curriculum development. Curriculum

development answers the question, “What should be taught?” It is during the development of the instructional

curriculum that a performance standard for the curriculum is established and program content is identified.

Instructional quality is often measured by how well participants are able to predictably achieve the objectives of the

program. This data is most often gained through traditional testing procedures. For example, a particular curriculum

may produce results, wherein an average participant achieves a 95% level of competency on the material presented.

Figure 7. Average competency level achieved versus delivery cost per participant

Instructional quantity, on the other hand, falls into the domain of instructional tactics, which is the process of

determining and selecting the most efficient method and media for delivering a program of instruction (curriculum).

The goal of instructional tactics is to answer the question, “By what means should the curriculum be taught?” Again, a

performance standard must be established by which to measure this dimension. Common standards are money, time,

instructional staff, equipment required, i.e., instructional resources.

Instructional Tactics Asks: By what means should the

curriculum be taught?

Resources + Activity = Results Productivity

Curriculum Development

Asks: What should be taught?

100%

Delivery Cost per Participant

Avera

ge C

om

pete

ncy L

ev

el

Ach

iev

ed

95%

90%

85%

$150

$125

$100

$75

Instructional System

Page 5: Determining the Productivity of Instructional Systems

The Journal of Human Resource and Adult Learning * November 2006 5

In order to determine the productivity of an instructional system, we must consider both the results produced by

the curriculum and the instructional resources required to deliver the curriculum. Keep in mind that productivity is a

ratio or composite measure of both the effectiveness and efficiency of a system. For example, in Figure 7 above we

have an instructional system in which 95% of the participants achieve competency using a media/method to deliver the

training which has a per participant cost of $125. By simply plotting the data using an X-Y graph and noting where the

data points are located in relation to the Slope of Productivity, it is possible to determine the effectiveness, efficiency,

and productivity for any given system.

Taking Corrective Action

The power of this evaluation strategy is that it enables trainers and instructional developers to identify where they

should take corrective action. Using Figure 8, it can be seen that to improve the instructional system, attention should be

focused upon decreasing the cost of delivering the training.

Figure 8. Determining corrective action using the Productivity Model

In other situations, given different data, trainers and designers may want to improve their curriculums. This brings us to

the following:

Factors that Influence the Effectiveness of an Instructional System (Curriculum Development)

- Needs Assessments

- Assessment of Learners

- Analysis of Work Settings

- Job, Task, or Content Analysis

- Statements of Performance Objectives

- Performance Measurements (test items)

- Sequence Performance Objectives

Factors that Influence the Efficiency of an Instructional System (Instructional Tactics)

- Instructional Techniques

- Designs for Instructional Materials

100%

Delivery Cost per Participant

Avera

ge C

om

pete

ncy L

ev

el A

ch

ieved

95%

90%

85%

$150 $125 $100 $75

Instructional System

Page 6: Determining the Productivity of Instructional Systems

The Journal of Human Resource and Adult Learning * November 2006 6

- Media

- Delivery mode / medium

- Instructional Resources, i.e., money, time, instructional staff, or required equipment.

Comparing Two or More Instructional Systems

Trainers and instructional developers are often asked to compare and contrast two or more instructional systems

and to make a recommendation. Using the Productivity Model, this becomes a simple task as illustrated below.

Let us say that we have two instructional systems that deliver the same information and skills. 85% of the trainees,

who use Instructional System A, achieve competency for an average delivery cost of $100 per participant. However,

95% of the trainees who use Instructional System B achieve competency using media and methods which have a per

participant cost of $125. To determine which of the two instructional systems is the most productive we need only to

graph out the two instructional systems as illustrated in Figure 9.

Figure 9. Comparing two or more instructional systems using the Productivity Model

Because Instructional System B falls higher and closer to the line we call the "Slope of Productivity," we can determine

it is the most productive. It should be noted that using this strategy, it is possible to compare any number instructional

system to determine which is the most productive.

A Grounded Model

The concepts and relationships detailed in this article are based upon established principles from a variety of

disciplines. The basic elements of a system: resources, activity and results, are described throughout the literature on

systems theory and are sometimes referred to simply as an input/output model.

Accountants and financial people, who often speak of return on investment, have known the relationship between

resources and results: (ROI). ROI is where returns are results and investments are costs, and in this way, are able to

determine the health of a business enterprise (activity).

Physicists and engineers recognized long ago that in order to describe the performance of various phenomena and

systems, they had to be described in terms of dynamic relationships between two variables. For example, miles per

gallon or feet per second.

100%

Delivery Cost per Participant

Avera

ge C

om

pete

ncy L

ev

el A

ch

ieved

95%

90%

85%

$150 $125 $100 $75

System B

System A

System A: 85%, $100 System B: 95%, $125

Page 7: Determining the Productivity of Instructional Systems

The Journal of Human Resource and Adult Learning * November 2006 7

Thomas Gilbert, a founding father of the human performance field, developed the First Leisurely Theorem, which

says that worth is equal to value divided by costs. In other words, activity adds value to resources (cost) and results in

something of greater worth.

Peter Drucker pointed out in the 1970s that effectiveness was doing right things, while efficiency was doing

things right. In the field of education, these concepts were expanded upon by Ivor Davies and applied to decision-

making concerning instructional methods.

CONCLUDING THOUGHTS

Because the Productivity Model is grounded in general systems theory, it can be used for a wide range of design

and evaluation activities. The author has not only used the Productivity Model to analyze various training systems, but

has used to the Productivity Model to analyze production processes, as a starting point for group discussions that

focused on improving processes ranging from sales, to product development, to distribution and logistics. In the

classroom, the Productivity Model has proved useful as a teaching tool by providing students with a framework on

which to arrange problems solving concepts that in turn help to facilitate the development of their critical thinking skills.

REFERENCES

Davies, Ivor K. (1981). Instructional Technique. New York: McGraw-Hill Book Company.

Drucker, Peter F. (1974). Management Tasks, Responsibilities, Practices. New York: Harper & Row Publishers.

Gilbert, Thomas F. (1978). Human Competence: Engineering Worthy Performance. New York: McGraw-Hill Book Company.

Kirkpatrick, Donald (1998). Evaluating Training Programs (2nd ed.). San Francisco: Berrett-Koehler Publishers, Inc.

Simpson, J.A. and Weiner, E.S.C. (1989) Oxford English Dictionary (2nd ed., Vols. 1-20). New York: Oxford University Press.

Phillips, Jack J. (1997). Handbook of Training Evaluation and Measurement Methods (3rd ed.). Houston: Gulf Publishing Company.

Rothwell, William J. and Kazanas, H. C. (2003). Mastering the Instructional Design Process (3rd ed.). San Francisco: Pfeiffer.