DESIGNING AN EFFECTIVE TRAINING EVALUATION SYSTEM FOR TRAINING

PROGRAMMES IN SARAWAK TIMBER INDUSTRY DEVELOPMENT CORPORATION : AN ACTION

RESEARCH STUDY

JtJIATA AKAW

Kota Samarahnn 1998

ACKNOWLEDGEMENT

In The Name Of Allah The Most BeneJficent and The Most Mercvid

Alhamdulillah, the writer is most gratefill for His Blessing and Mercifulness and with the strength showered, the writer was able to complete this thesis in fulfillment of the Master of Science Degree award.

My highest appreciation and thank you to my Supervisor, Ms Angela Anthonysamy, lecturer at the Faculty of Cognitive Science and Human Development, for all the assistance and guidance provided durixlg the preparation of this thesis. The same note of appreciation is extended to both the external and internal examiners; Dr. Maheswari Kandasamy from the Institute Aminuddin B&1 and Assoc. Prof. Dr. Peter Songan from UNIMAS.

Thank you to the State Government for the scholarship award lo enable the writer to further her studies at the post-graduate level. Thank you to the Management of Sarawak Timber Industry Development Corporation, Kuching, Sarawak for their approval to further my studies as well as to conduct the study in STIDC. My heartfelt gratitude to my peers, colleagues and subordinates especially my action research group members: Mr. Charles Aseng Tenggoi from the Centre for Modern Management, Mr. Affendi Sapiic, Mx. Dahrawi Muhammad, Ms. Kasunla Haron, Ms Suraya Hj. Mohd, Ali, Ms Ruth Sibat @ Ntir Mithali Abdullah and Ms Dayang Masmolydiza A.H. Dawi from STIDC. My thank you also extends to those who were involved in the study, either directly or indirectly.

My appreciation also goes to the Dean of Faculty Cognitive Science and Human Development, Prof Dr. Razali Arof, the Coordinators of the Twinning Programme UNMAS-CMM, all the lecturers involved and fellow coursemates for their continuous guidance and encouragement. This appreciation is also extended to the management and staff of the Gibaran Institute of Management, Australia.

Lastly, the writer wish to note her heartfeft gratitude and thank you to her beloved husband, Bujang and daughter, Dewi Kltartini, and all members of her family who provided their support. encouragement, sacrifices and understanding during her studies and especially during the preparation and completion of this thesis,

May Allah Bless and Reward your kindness and our efforts.

TABLE OF CONTENT

ACKNOWLEDGEMENT

TABLE OF CONTENT

LIST OF TABLES

LIST OF FIGURES

LIST OF ABBREVIATIONS

ABSTRACT

Pages

ii

iii

vi

vi

vii

... Vl l l

Introduction 1 Background Of The Study 2 Problem Statement 4 Objectives Of Study 4 Significance Of The Study 5 Definitions 6 Scope and Limitations Of Study 9

L I T E R A T E RW'IEW Introduction The Training Cycle Definitions Of Evaluation Purposes Of Tra~ning Evduatlon The Frainework Of Evaluation

Evaluation model Evaluatlon design Evaluatlon Instrumentsfstrategles Human resources roles in an evaluation

system Environment support Survey And Studies

Conclusion

cwAp'rER 3 RESLIAKCI I IWSI'I IODOL~OGY Introd~ict~on Purpose CIS Study SltefI3a~kg1 ounUT11en~dtic Concern 1,lterature lieview On Action Resedrch (AK)

Orlgins ol'ac~on reseiirch Uef'inltion of actloll research A ~ t l o n reseal ch process

Chara~tcr lstics of actloil research Act1011 Research In liclahon 'fo Actiotl Learning

Ihc actlon learnlng concept Ihfterences and slinllarit~es between actlor1

resem~h and actlor1 learning Just~ficat~on Of Action Keseilrch Method In 'This

Studv Ihe d~t ion ~esearch process Action I esearch cycle Charaderrst~cs of action research

Conclusion

CHhP'WIZ 4 FINIIINGS Introcfiiction Puq)osc 01' Studv S ~ o p e Of Study tlesearch Qtiest~ons Action Research Cycles h d Processes First A ~ t ~ o r l Rc5earcl.l Cycle

Entry Into the organi/at~oii I~orindt~on of the study group First d~t lon r e x d r ~ h illect~ug Sc~orld a~t ion rcsedrcli rrieetlng 1 h11d and Ibu~t l i meet~ngr b I Lth dl~d S I X ~ ~ I mcctlngs S ~ L crrtll nlcetlng I'lghth rneetlng

Second Act1011 12e\ecrrch Cycle 1 esting ol tile cvaludioii strategies '~nd

technl y ues Sricfiizg Quebtlonnalrc aiid anCllysls Indiv~dual actloll plan Ciro~rp problem-solving prolect

Evaluation Of A ~ t ~ o n liesearch Gloup Tr~anguitttion 01 l ' i ~ Action Kesedrch Process And

The Study Conclus~on

C W T E R 5 SUMMARY, CONCLUSIONS AND RECOWN13A'I 'IONS Introduction Sumrnary

13ackground To Probieln Research Methodology Research Questions Scope and Limitations OF The Study

Discussron 'I he dct~on research process Results based on research cpestions

Conctusions Reco~mendatlons Recommendi~tions I;or Further Studies

B l B L I O W H Y

APPENDlCES A-L

LIST OF TABLES

Tables

4.1 Status Of Projection Of COSEC Training I11 STIDC

LIST OF FIGURES

Pages

53

The Concepttlal Framework Of Training And Training 5 Evaluation

The Systematic Training Cycle 11

Criteria For Evaluation Of Completed Training Occ~lring 15 At Four Levels

The Spiral Of Action Research Cycles 28

The Kolb Learning Cycle 3 5

Action Research Cycles In Designing A Training Evaluation 40 System

Single Measurement DesigniPost Training Design 58

Individual And Team-based Inlplementation Strategies 64 Foe Evaluating COSEC

Analysis Of Eval~~ation Questionnaire 67

First And Second Action Research Cycles 77

Training Evaluation Model 80

LIST OF ABBREVIATIONS

AR Action Research

COSEC Core Skills For Effectiveness and Change

CMM Centre for Modern Management

CMD Chief Minister's Department

HRDD Humail Resource Developmeilt Division

HRD Human Resource Developmeilt

STIDC Sarawak Timber Industry Development Corporation

ABSTRACT

and yet there is

... Vlll

Kajian fni krtujuan untufz merekabentuk sistem penifaiatr latihen derrgan menggunakan program tatihan, Ketamgritan ,4sas Unruk Kebenkesanan dun Petwbahan (TASXU), di Perhadamn Kemajuun Perusahaan Kayu Samwak (PUSAK4) se-i w a n kes. Qbjecfij ktjian ini adaiuk untuk nterekabenhrk sutu srstm p g terancang dun temtur se* kaedah d m strategi penilaian yang sesuai untuk menilai kgberkesanan program TRSKU, Program ini #elah dikenalpasfi sebagai pmgwm latikan ufamu untuk unggotu hmpt5lan sokangan dun operasi ofek Bakagian P m b a n g w a ~ ~ Suntbw jl4anrrsia Ebegeri di Jbatan Kehau Menten'. Stlmber- stftnbm b~rbentuk kmmgan dun nzasa yang teiafr ditabttrkan ke atas progran ini csrkrp bunyak Ctapi makIumat t e p r i n d beyhabung keberkesanan progrum tersebut belum dapnt dipemlehi. fni disebabkun belum wjud sistent pmilaiun yang tetancang dun berkesan lrntrtk digunakan di PUSAKA. Perlazedahan lcsfian yang digurtdksrn dikennli sebagai "action research". Safah satu daripada dapatan W a n ini isrlah kcrjien ini manampilkan sebslas daripada ciri-ciri "action mwrch". Sistem yang dif.tasifkan adafuh berrrsasbn model k b i k Entpat-Peringkaf Kirkpatrick. Entpur peti&kat ini iulah Reuksi: Pevrbela,uran: Penthahan s,&: dun f&i. Kaedak penilaian yang digrrt~akatz lit~tuli n~mguntpul makl~intat herhubung keherkesannn latihun iahh melatui borang penilaian laFihan dun kaeduh pelan tindakun. Srmtegi yang digunukm ialah meiului projek penyelesaian masulah secara berlcumpulan dun juga ntelnlz~i pelart t i n d h individu, Kgi'a~i ini mmrSaputi bahmyc? elemerr-ehzm yang mmim6ufkan persekitumn yang menggulukkan peralihan pembelajam ftasil daripada tafihatr yang diterima ke tempat bekerju tidak krrmrrg penting and nresfi diambilkiru dulum mewujudkan satu sistem penilaian yang berkesan. Dapatan tcajian ialah peranan penyeliu pelatih dun pihufr pengurusun penting untuk tnemastikan pmses peralikun pembelajumn tersebut herjaya. Pela filt mesti $&us dm komifed sebefurn proses pemfielajuran dapat bedab. fni ukun membawu k p d a pembahaa sikap sepcrti dijangka dun seferusnyu kesm posit$ kzpada o~ani.pasi. Kujian mendapati bahuwa pam pelafih berpe?zdupaf progrant latih~n ini bergtinu dan pmkfikal dun tidak menghadapi masalah untuk aplikasi di ternpat bekerja. Walau bagain~unupun, berdasarkan maklurnbalas daripadu penggrtnctan pelan tindakan individz~, para pefatih rnasih menghadapi masrrlah mendup f sokottgu~i sertu bimhingan daripuda penyefia m r e b . Dapnta~ kajian ialah perkiredahan kqiiafi dapu f digunakan untxtk pembanganatz organisasi. Setain itu, unttuk membentukkun sfstem pe~zilaiatt y a ~ g berkesan, pihak pengumsan per& nsengaz - faktor-faktor yang boleh ntmjndkan parsekituran yung dupat ntenyokmg sistm ber Sisfem penilaian in1 hukan sakcqu menilai keberkesanan larihan ke atas se.~ebuah or$ tetapi secura lungsung sistenr tersebut napat menggalakkan pemlihan hasil latihu~r ke beketja.

CHAPTER 1

Introduction

achievement of ce of human resource

human resource devel

Introduction

the uther productive functions in the organization and is considered as an investment

2

Introduction

een an increasing purpose of training activities.

I, The HRD department is Add accountable for its activity, not fo

3

Introduction

Probfcm Statement

programme is provided by the C M ,

1. What are the

objectives of this stu

4

Introduction

achieve these goals.

The conceptual framework of this study looks at ' . ng evaluation as depicted below in Figure I , 1 : -

CIRU

Level 3 - Behaviour

-

Fimre I . 1: The Conceptual Emmework 01 Training And Training Evaluation

5

In the context o€ this study, the training activi as addressed was

6

Introduction

finding out how a course or any

ure how the trainees react to rs, come materials, and so on;

raining programme.

modifications to

programme I

Core Skills For Effectiveness anti ChanPe training ;)rotyamme.

d for skilled and

empluyees due to

The Core Skills For Effectiveness and fits in to the concept of “core skills” which wo

raw& State Civil

ves of COSEC

7

Introduction

a total number of 202 em

x

Introduction

resources; and to develop the Bumiputera small-medium scale industries in the timber processing industry.

ropriate to assess the staff. The study was

t ST31DC’s head

measure it, The

from the C.M.M.

9

CHAPTE

Literature Review

evaluation system w

Literature Review

Brinkerhoff & Gill (1994) also agrees that the training process should be suffused with assessment measures. Measurement data are needed to identify employees' deficiencies and required improvement; to establish training objectives; to improve learning events; to assess acquisition of new skills, knowledge and attitude; and last but not least to obtain feedback on the trainees' mastery of new abilities.

Brinkerhoff & Gill (1994) uses a four-part process approach to training for performance improvement. Their basic assumption is that the end objectives of training is not to produce learning in new knowledge, skills and attitude. The benefits from training is only when new learning is maintained and applied on the job. The newly learned knowledge, skills and attitudes only provide the capacity to change and add value to organizational products and services. If learning is not reinforced into improved performance, then training is ineffective and failed.

New learning that is not reinforced can easily disappear (Nadler and Nadler, 1989). This reinforcement can be in any form, such as providing the opporlunity for the trainee to use the new learning as soon as possible after returning to the job. Another reinforcing factor is the supervisor's expectations to see improved performance. On another level, reinforcement can be in the form of increased pay or a supporting environment. Although it is easy to reinforce training, it is often overlooked (Nadler and Nadler).

The effectiveness of training is not only concerned with the learning that has taken place. The transformation of learning to add value to organizational effectiveness requires that the learning experience be endured and applied over time to enhance job performance (Brinkerhoff & Gill, 1994). The transfer of training or learning to the job can be induced through follow-up assessments of trainees' job performance, encouraging trainees to use the acquired new skills, knowledge and valued attitudes.

According to Shapiro (1995), training is effective :

.... to the degree that it enables each performer to produce results on the job using the knowledge and skills taught. When this occurs, the organization gets a returns on its investment. If the performer continues to use the skills we can say that the investment has grown in value (p. 3).

A common method of evaluation is a post-course questionnaire which is usually filled by the trainee immediately at the end of the training programme. Generally, its purpose is to evaluate the trainee's level of satisfaction with the programme. Often a high level of satisfaction suggests the successfulness of the

12

Literature Review

programme. The disadvantage of this method is the "powefil halo effect" that occurs whereby the trainee expresses his or her enjoyment of the programme and food, and so on. It however, does not examine the learning that has been taught (Rossi and Freeman, 1993).

Definitions Of Evaluation

In discussing the attempts to measure training, various terms are interchangeably used and these include evaluation, assessment and validation. The Glossary of Training Terms defines evaluation as:

The assessment of the total value of a training system, training course or programme in social as well as financial terms. Evaluation differs from validation in that it attempts to measure the overall cost-benefit of the course or programme and not just the achievement of the laid down objectives. The tenn is also used in the general judgmental sense of the contilzuous monitoring of a programme or of the training functiort as a whole pramley, 1986, p. 2).

Goldstein (1993) defined evaluation as "the systematic collection of descriptive and judgmental information necessary to make effective training decisions refated to the selection, adoption, value, and modification of various activities" (p. 147). Bee and Bee (1994) quoted Hamblin's definition (1974) as a more usefill and practical one which is "any attempt to obtain information (feedback) on the effects of a training programme, and to assess the value of training in the light of that information'' @. 174).

Based on the definitions of training evaluation, we can summarize that evaluation basically as a systematic process of activities to collect valuable information on the effectiveness of training. This study focused on Hamblin's definition of evaluation whereby the shtdy attempted to collect information on the impact of the COSEC training programme on the trainees' at their work place. The impact or the training should be able to provide the value or returns gained from training in ternzs of improved performance and thus, resulting in business productivity.

Purposes Of Training Evaluation

Evaluation is conducted to meet several purposes. Brarnley (1986) has grouped these purposes into five types: feedback, control, interventions, research and power games evaluations. Feedback evaluation is ibr the purpose of providing

13

Literature Review

evidence of the quality of the design and delivery of training activities and this is usually trainer-centred. Trainers need to conduct this evaluation to collect information related to the learning and that the gap between the identified deficiellt and desired peribrmance has narrowed.

The second purpose is co~ltrol evaluation which relates the training policy and practice to organi~ational goals. This form of evaluation is to measure the effectiveness of the HRD function. The third purpose IS for intervention evaluation which is as an intervention strategy. It relates to evaluation issues such as managerial and supervisory support for training and its resultant impact. It can lead managers and supervisors to re-appraise their responsibility in ensuring transferability of training.

Research evaluation sceks to add to knowledge of training principles and practices. It can also result in improvements to the existing evaluation methods and techniques The fifth purpose of evaluation is as power games evaluation. Evaluation can be used, or misused, to play power games. The data collected will bc manipulated to assist in organizational conflicts.

Reeves (1994) on the other hand, simply outlined that the purposes of evaluation are to understand:- (a) the strengths and weaknesses of a current training programme; (b) the impact of training on individuals; (c) the impact of training on the organizations; and (d) who benefit the most and the least from the training event. Reeves' outline of evaluation purposes reflects the purpose of this study.

The Framework For Evaluation

The discussion on the framework of evaluation here looks at the evaluation model; its design; the evaluation insirumetlts employed; and the external factors related to evaluation such as the human resources roles in an evaluation system and environment supporl.

Evaluation model.

There are several accepted evaluation models wh~ch establish different categories or levels of evaluation, each with different reasons. Thls review will only discussed the evaluation model that the study has adopted as a basis to design an evaluation system to assess the COSEC training programme.

The model referred to is the classic and most-used reference model which is the Kirkpatrick model (Kirkpatrick, 1994). The Kirkpatrick model focus on training evaluation at four levels. The four levels represent a sequence of ways to evaluate programmes and each level is equally important. As the evaluator progress from one level to the next, the more difficult and time-consuming the process becomes but the more valuable the information collected. None of the levels should be by-passed (Kirkpatrick). The four-leveled model is depicted in Figure 2.2.

Fimre 2.2 : Criteria for evaluation of completed training occurring at four levels. From The Winning Trainer (p.403), by J.E. Eitington, 1996, Houston : Gulf Publishing

Company.

In the model, the first level or reaction level measures how the trainees react to the training event. Kirkpatrick calls it "a measure of customer satisfaction" (p.21). He said that the purpose of getting a positive reaction is not only to market the programme but also to induce learning. Although a positive reaction may not guarantee learning but a negative reaction would certainly reduce the learning capacity. In the learning level, Kirkpatrick defined learning as the "extent to which the participants change attitudes, improve knowledge, and/or increase skill as a result of attending the programme" @. 22). In his model, learning has take11 place when either attitudes are changed; knowledge is increased; or skill is improved.

Level IV

Impact/Results (was there a measurable impact on the organizationl)

In measuring behaviour change, Kirkpatrick defined behaviour as the extent to which change in behaviour has occurred due to the training attended. He does not advise evaluators to by-pass the first two levels in order to measure behavioural change.

Level III

Participant Application (was there a change in job performance?)

Level I

Participant's ReactionsReeling (was the training programme liked1 satisfactory?)

Level I1

Participant Learning (was there an obvious change in performance in classroom?)