lecture 8 evaluation- design issues

8/3/2019 Lecture 8 Evaluation- Design Issues

http://slidepdf.com/reader/full/lecture-8-evaluation-design-issues 1/14

Department of Business and ManagementUniversiti Brunei Darussalam

Training Evaluation- Design

Issues

Basic Designs

± Post-test only

± Pre-rest/Post-test

Complex Designs

± Post-test with control group

± Pre-test/Post-test with control group

± Time Series ± Solomon 4 group




Basic Designs

Post-test Only

± occurs when training is provided and then a post-test is

given

X T2

X ± training

T2 ± post-test evaluation




Basic Designs - Post-test Only

(cont¶d)

Goals of evaluation:

± to determine if change has taken place

± to determine if a level of competence has been reached

Post-test only design is sufficient if the

evaluation is for the second goal

± e.g. a training which is developed to equip employees

with certain legal requirements





(cont¶d)

Post-testing can be turned into a pre-test/post-test

design by utilizing data from the following sources:

± needs analysis

± archival data, e.g. performance appraisal

measures of quality

An equivalent group could also be chosen and given

the same post-test and the evaluation becomes

post-test only with control group





(cont¶d)

Post-test only design is problematic for

assessing change because change could be

caused by a number of other competing

courses




Basic Designs- Pre-test/Post-

test

The design can be represented as:T1 X T2

T1 ± Pre-testX ± Training

T2 ± Post-test

Main criticism is that it is without a control

group ² difficult to determine the impact of training on the changes that occur




Basic Designs- Pre-test/Post-

test (cont¶d)

A method of dealing with the lack of acontrol group is known as InternalReferencing Strategy (IRS)

IRS includes using relevant and non-relevant test questions in the pre- and post-test

± relevant : questions that deal with training contents

± non-relevant : questions with aspects not trained (these

questions serve as control)




Complex Designs

Two factors to consider in developing a

sound evaluation design:

± control group: a group of similar employees who do not

receive the training

± random assignment: the assignment of employees to

either the control group or training group by chance




Complex Designs ± Post-test

Only withControl Group

This design can be represented by the followingTrainee Group X T2

Control Group T2

Differences in test scores between the groups will

provide evidence of the effectiveness of the training Weakness: there is no pre-test score to indicateequivalence of the groups before training

Representative sampling: matching of employeeson specific variables e.g. age, educationalbackground, experience etc. may reduce theweakness




Complex Designs- Pre-

test/Post-test withControl

Group It is represented by the following

expression:Trainee Group T1 X T2

Control Group T1 T2

Provides indication of equivalence between

the two groups (through T1)

Post-test differences will provide evidence

for training effectiveness




Complex Designs- Time Series

Design

The expression for this design is:Trainee Group T1 T2 T3 T4 X T5 T6 T7 T8

It uses a series of measurements before and

after training

The design can be made more powerful by

using a control groupTrainee Group T1 T2 T3 T4 X T5 T6 T7 T8

Control Group T1 T2 T3 T4 T5 T6 T7 T8




Complex Designs- Multiple

Baseline Design

Expression:

Trainee Group A T1 T2 T3 X T4 T5 T6 T7 T8 T9 T10 ..

Trainee Group B T1 T2 T3 T4 T5 X T6 T7 T8 T9 T10 ..

Trainee Group C T1 T2 T3 T4 T5 T6 T7 X T8 T9 T10 ..

Trainee Group D T1 T2 T3 T4 T5 T6 T7 T8 T9 X T10 ..

Multiple measures are taken (as in time series) but

each group is trained at different times

If each group improves after receiving training, it can be said that the changes are due to training




Complex Designs- Solomon 4

Group Design

Expression:Group 1 T1 X T2

Group 2 T3 T4

Group 3 X T5

Group 4 T6

Members of the groups should be based onrepresentative sampling

If T2 > T1 and T4, and if T5 > T6, and if T5 > T3 theninference about the effects of training can be made

with confidence

If T6 = T1 and T3 (history and maturation are not thecause of improvement)




Evaluation± TheChoice of

Design

The true impacts of training depends on the validity of evaluation results

The more complex the design, the more valid the results

When multiple measures are not possible, the following designmay be useful:

Trainee Group A T1 X T2

Trainee Group B T1 X T2

Trainee Group C T1 X T2

Trainee Group D T1 X T2

lecture 8 evaluation- design issues

Documents