1 2007 national human services training evaluation symposium fasih ahmed, ph.d

1

2007 National Human Services Training

Evaluation Symposium

Fasih Ahmed, Ph.D.

2

Reaction to Training: A Few Basic Questions about the

Nature of the Construct

Is reaction to training a multi-dimensional construct with distinct components or a generalized holistic perception of training experience?

If reaction to training is a multi-dimensional concept, how the various dimensions relate to each other and how they collectively formulate the overall reaction to training?

Are the various dimensions identifiable (and measurable) as separate ‘valid’ constructs with sufficient specificity for application in evaluative analysis of training programs?

3

Some Possible Approaches and Assumptions

Most trainee satisfaction surveys assume reaction to training as a multi-faceted concept and measure satisfaction/dissatisfaction with such aspects of training as the curriculum, the trainer, and training delivery arrangements/facilities, etc.

Statistical techniques can be used to delineate and examine specific dimension of reaction to training. These can range from such sophisticated techniques as factor and logistic analyses to simple cross tabulations, depending on the level of measurement and tenability of assumptions about the underlying distributions of the specific variables being measured.

The purpose of this analysis is to determine the nature of relationship between the various dimensions of reaction, and how they individually relate to the overall reaction to training experience. Question: are individual dimensions independent of each

other, or related in some causal model, or have interactive effects on each other?

Question: What is the nature of relationships between individual dimensions with the overall reaction to training?

4

Methodology & Data: The Case of North Carolina

NC uses the Participant Satisfaction Form (PSF) to survey reaction to training and its components: curriculum (five questions); trainer(s) (six questions); training delivery arrangements (three questions), in addition to a few attitudinal questions and a question on satisfaction with the overall training experience.

Almost all questions have four response categories (strongly agree, agree, disagree, strongly disagree) which. when combined, yield dichotomous variables with two outcomes: satisfied-dissatisfied.

Our focus of this analysis are only two components: reaction to curriculum and reaction to trainer. This analysis is based on data collected over three years: 2004,2005, and 2006. Data for each year is analyzed separately to discern any patterns.

In order to aggregate data, a mean score for each component is computed from the cluster of questions pertaining to that component, which yields an interval scale continuum of satisfaction-dissatisfaction. Distribution of mean scores for reaction to trainer and reaction to curriculum are shown below.

8

Limitations of Analysis

Given the extremely skewed distribution of mean scores, our statistical analysis will have to be limited to non-parametric approaches, which do not depend on assumptions of normal, or approximately normal, distributions.

Mindful of the fact that the responses were originally measured as categorical variables, we have transformed the mean scores into two categories: satisfied and dissatisfied, for each of the component.

Since the scores of 3 and 4 in the original measurement indicated satisfaction, all mean scores of less than 3 were categorized as indicative of dissatisfaction, which typically includes the bottom 10% to 27% of all mean scores.

Given the need to aggregate the data as categorical variables, cross tabulation is the statistical techniques we have used for the following analysis.

Simple distribution of satisfaction/dissatisfaction with curriculum and trainer for the three years under consideration are shown below.

9

Mean Score Curriculum 2004 Frequency Percent

Satisfied 5223 72.8

Dissatisfied 1951 27.2

Total 7174 100.0

Mean Score Trainer 2004 Frequency Percent

Satisfied 10380 82.8


Total 12530 100.0


Satisfied 6834 76.3


Total 8962 100.0


Satisfied 12676 80.8


Total 15696 100.0


Satisfied 6442 89.3


Total 7212 100.0


Satisfied 6379 88.9


Total 7179 100.0

10

Are Reactions to Curriculum and Trainer Independent Dimensions?

Curriculum Trainer Total

Dissatisfied Dissatisfied Satisfied

% within curriculum

% within trainer

44.5 55.5

70.4 18.3

100

Satisfied

% within curriculum

% within trainer

7.0 93.0

29.6 81.7

100

Total 100 100

2004

2005



% within curriculum

% within trainer

43,7 56.3

33.7 6.6

100

Satisfied

% within curriculum

% within trainer

9.8 90.2

66.3 93.4

100

Total 100 100



% within curriculum

% within trainer

42.0 58.0

40.3 7.0

100

Satisfied

% within curriculum

% within trainer

7.5 92.5

59.8 93.0

100

Total 100 100

2006

11

Nature of Relationship Between Reaction to Curriculum and Reaction to Trainer2004



% within curriculum

% within trainer

44.5 55.5

70.4 18.3

100

Satisfied

% within curriculum

% within trainer

7.0 93.0

29.6 81.7

100

Total 100 100

Assumption I: Reaction to curriculum is the IV and reaction to trainer is the DV.

A. Condition of dissatisfaction with trainer:44.5% of those dissatisfied with curriculum are also dissatisfied with trainer.7.0 % of those satisfied with curriculum are dissatisfied with trainer.Indication: dissatisfaction with trainer is more likely to be related to dissatisfaction with

curriculum. % difference: 44.5-7.0 = 37.5.

B. Condition of satisfaction with trainer:55.5 % of those dissatisfied with curriculum are satisfied with trainer.93% of those satisfied with curriculum are also satisfied with trainer. Indication: satisfaction with trainer is more likely to be related to satisfaction with curriculum. %

difference: 93.0-55.5 = 37.5.

Assumption II: Reaction to trainer is the IV and reaction to curriculum is DV.

C. Condition of dissatisfaction with curriculum:70.4% of those dissatisfied with trainer are dissatisfied with curriculum.18.3% of those satisfied with trainer are dissatisfied with curriculum.

Indication: dissatisfaction with curriculum is more likely to be related to dissatisfaction with trainer. % difference: 70.4-18.3 = 52.10.

D. Condition of satisfaction with curriculum:29.6% of those dissatisfied with trainer are satisfied with curriculum.81.7 of those satisfied with trainer are satisfied with curriculum. Indication: satisfaction with curriculum is more likely to be related to satisfaction with trainer. %

difference: 81.7-29.6 = 52.10.

Tentative Conclusion: Comparing the % differences under the two assumptions, Assumption II seems more tenable.

12



A. Condition of dissatisfaction with trainer:43.7% of those dissatisfied with curriculum are also dissatisfied with trainer.9.8 % of those satisfied with curriculum are dissatisfied with trainer.Indication: dissatisfaction with trainer is more likely to be related to dissatisfaction with

curriculum. % difference: 43.7-9.8 = 33..9.

B. Condition of satisfaction with trainer:56.3 % of those dissatisfied with curriculum are satisfied with trainer.90.2% of those satisfied with curriculum are also satisfied with trainer. Indication: satisfaction with trainer is more likely to be related to satisfaction with curriculum.

% difference: 90.2-56.3 = 33.9.


C. Condition of dissatisfaction with curriculum:33.7% of those dissatisfied with trainer are dissatisfied with curriculum.6.6% of those satisfied with trainer are dissatisfied with curriculum.

Indication: dissatisfaction with curriculum is not more likely to be related to dissatisfaction with trainer. % difference: 33.7-6.6 = 27.1.

D. Condition of satisfaction with curriculum:66.3% of those dissatisfied with trainer are satisfied with curriculum.93.4% of those satisfied with trainer are satisfied with curriculum. Indication: satisfaction with curriculum is not more likely to be related to satisfaction with

trainer. % difference: 93.4-66.3 = 27.1.

Tentative Conclusion: Comparing the % differences under the two assumptions, Assumption II seems more plausible, but only slightly so.



% within curriculum

% within trainer

43,7 56.3

33.7 6.6

100

Satisfied

% within curriculum

% within trainer

9.8 90.2

66.3 93.4

100

Total 100 100

13



A. Condition of dissatisfaction with trainer:42.0% of those dissatisfied with curriculum are also dissatisfied with trainer.7.5 % of those satisfied with curriculum are dissatisfied with trainer.Indication: dissatisfaction with trainer is more likely to be related to dissatisfaction with curriculum. % difference: 42.0-7.5 = 34.5.

B. Condition of satisfaction with trainer:58.0 % of those dissatisfied with curriculum are satisfied with trainer.92.5% of those satisfied with curriculum are also satisfied with trainer. Indication: satisfaction with trainer is more likely to be related to satisfaction with curriculum. % difference: 92.5-58.0 = 34.5.


C. Condition of dissatisfaction with curriculum:40.3% of those dissatisfied with trainer are dissatisfied with curriculum.7% of those satisfied with trainer are dissatisfied with curriculum.Indication: dissatisfaction with curriculum is more likely to be re lated to dissatisfaction with trainer. % difference: 40.3-7.0 = 33.3.

D. Condition of satisfaction with curriculum:59.8% of those dissatisfied with trainer are satisfied with curriculum.93.0% of those satisfied with trainer are satisfied with curriculum. Indication: satisfaction with curriculum is more likely to be related to satisfaction with trainer. % difference: 93.0-59.8 = 33.2.

Tentative Conclusion: Comparing the % differences under the two assumptions, both Assumption I and Assumption II seem equally tenable.



% within curriculum

% within trainer

42.0 58.0

40.3 7.0

100

Satisfied

% within curriculum

% within trainer

7.5 92.5

59.8 93.0

100

Total 100 100

14

Interaction Between Overall Training Experience and Reaction to Trainer 2004

Overall Experience Trainer Total


% within oa exp.

% within trainer

57.7 42.3

44.6 6.7

100

Satisfied

% within oa exp.

% within trainer

10.9 89.1

55.4 93.3

100

Total 100 100

Overall Experience Curriculum Total


% within oa exp.

% within curriculum

59.3 40.7

28.9 7.3

100

Satisfied

% within oa exp.

% within curriculum

22.2 77.8

71.1 92.7

100

Total 100 100

Interaction Between Overall Training Experience and Reaction to Curriculum 2004

PSF Data for 2004

There is positive association between overall experience of training and reaction to trainer as well as reaction to curriculum.

The effect of overall experience on reaction to trainer is a little stronger than that on reaction to curriculum, suggesting an interactive effect between overall experience, reaction to curriculum, and reaction to trainer.

One should, however, be mindful that perception of overall experience is based on only one question.

15




% within oa exp.

% within trainer

51.2 48.8

32.2 4.7

100

Satisfied

% within oa exp.

% within trainer

9.8 90.2

67.8 95.3

100

Total 100 100




% within oa exp.

% within curriculum

63.0 37.0

31.1 5.7

100

Satisfied

% within oa exp.

% within curriculum

18.5 81.5

68.9 94.3

100

Total 100 100

PSF Data for 2005

There is positive association between overall experience of training and reaction to trainer as well as reaction to curriculum.

The effect of overall experience on satisfaction with trainer is slightly more than the other way round.

The effect of reaction to curriculum on perception of overall experience is more than effect of overall experience on curriculum.

These pattern suggest an interactive effect between overall experience, satisfaction with trainer, and reaction to curriculum.

16




% within oa exp.

% within trainer

45.1 54.9

33.5 5.1

100

Satisfied

% within oa exp.

% within trainer

8.0 92.0

66.5 94.9

100

Total 100 100




% within oa exp.

% within curriculum

62.6 37.4

48.9 3.4

100

Satisfied

% within oa exp.

% within curriculum

5.8 94.2

51.1 96.6

100

Total 100 100

PSF Data for 2006

There is significant association between overall experience of training and reaction to trainer, the effect of overall experience on reaction to trainer being slightly more than the converse.

There is a significant positive effect of overall experience of training on

reaction to curriculum , the effect of overall experience on reaction to curriculum being slightly more than the converse.

The pattern of associations suggests interactive relationships between the overall experience of training, reaction to curriculum and reaction to trainer.

17

Inconclusive Conclusions

The three years data does not yield any consistent patternsto allow definitive answers to the questions raised initially. However, some of the interesting, though unexpected findings are:

Given the patterns of relationships between reaction to curriculum, reaction to trainer, and the overall experience with training, there is no clear evidence that specific dimensions of reaction to training are independent components of a theoretical construct.

When examining relationships between reaction to curriculum and reaction to trainers under different assumptions of a causal model, there only indications, not evidence, that satisfaction with trainer tends to play more important role in determining reaction to curriculum, relative to perception of overall experience of training.

Although analysis of reaction to training may serve some limited evaluative purpose, its utilization as a decision-making tool for management of training warrants due caution.

1 2007 national human services training evaluation symposium fasih ahmed, ph.d

Documents