employers and coaching evaluation - icf - search...

Employers and Coaching EvaluationAlison Carter, Ph.D., Helen Wolfe and Maire Kerrin, Ph.D.

This article first appeared in the International Journal of Coaching in Organizations, 2005, 3(4), 63-72. It can only be reprinted and distributed with prior written permission from Professional Coaching

Publications, Inc. (PCPI). Email John Lazar at [email protected] for such permission.

ISSN 1553-3735

2005

© Copyright 2005 PCPI. All rights reserved worldwide.

Journal information:

www.ijco.info

Purchases:www.pcpionline.com

63

Employers and Coaching Evaluation

Alison Carter, Ph.D., Helen Wolfe and Maire Kerrin, Ph.D.

Many UK corporate purchasers perceive coaching as a widely accepted method for the development of executives’ talents insupport of corporate objectives. There is, however, perceived to be a paucity of evidence about whether coaching is an effectivetool for improving individual and organisational performance despite its widespread purchase and use in practice. As far asemployers are concerned, advice and guidance are not readily available to them on how to evaluate the coaching which theyare providing within their organisations. Using three case study illustrations of how leading UK organisations are evaluatingtheir coaching provision, this article explores issues in evaluating coaching in an organisational context. It also identifieswhat factors seem to help and hinder good evaluation practice in a workplace context. The article draws on findings arisingfrom a 2003-04 study of coaching evaluation practices funded by the UK-based IES Employee and Management DevelopmentResearch Network of 57 large employers. Commentary is drawn from the authors’ own experiences: all as evaluators and oneas an executive coach.

Existing Research on WhetherCoaching is an Effective Tool

There is limited evidence on the impact of coaching as atool, and some of the published research studies in thisarea have serious limitations in their design andanalysis of the data collected. However, there are someinsightful papers and in the area of executive coachinga number of these studies have been identified that focuson the efficacy of executive coaching. In summary thesestudies provide some limited evidence that executivecoaching is effective for increasing performance1, isviewed favourably by executives2 and has the potentialto facilitate developmental change3. However, these andother studies4 all vary as to what they were assessing interms of effectiveness, and all use different approachesand methods for evaluation. Some had ‘quasi-experimental’ approaches to gathering information oncoaching; others were more informal in their approach.As such, they provide an indication as to the potentialfor coaching to be effective in certain areas, but apartfrom one study they offer limited guidance on how toevaluate coaching within organisations.

Why Do We NeedMore Research on Coaching Evaluation?

Timing of Evaluation and Availability of InformationTwo practical barriers to evaluation were highlightedby UK employers that they claimed made coachingevaluation feel different than their traditional trainingevaluation. First is the timing of the evaluation and howit affects the information gathered on the effectivenessof the coaching. Evaluation directly after the coachingexperience may not be appropriate as the benefits may

only accrue later when the coach or person coachedreflects fully on what the relationship has brought them.Although this is true also of other forms of training,whereby you may not get to use your new skills until alater date, it is more likely that you will be able toanticipate or recognise where training will help. Withcoaching, the outcomes of what is learnt or discussedmay not be immediately obvious to either the coach orthose coached, unless again they are linked into a widerprogramme (e.g., leadership development, culturechange). Also, as the defined success criteria may notbe agreed upon until the coaching process has begun, itis difficult to measure pre- and post-coachingintervention to assess the impact. Second, informationwith which to evaluate may not be readily accessible ina coaching situation (e.g., due to the confidential natureof the coaching context, etc.).

Evaluation ModelsAny discussion of evaluation within a learning anddevelopment context does not go far before models suchas Kirkpatrick’s5 are introduced. In a search of thepublished academic research literature, one significantreported study of coaching evaluation within anemploying organisation that had used any systematicapproach based on a model of evaluation was given byMcGovern and colleagues6. They applied Kirkpatrick’smodel systematically to evaluate coaching at all fourlevels (reaction, learning, behavioural change andorganisational outcomes). They also included a level 5measure of return on investment (ROI) based on thework of Jack Phillips7. In addition, they developed amethodology for assessing the factors contributing tocoaching effectiveness and ineffectiveness and foundthat the key factor to the effectiveness was the quality ofthe relationship between the coach and the executive.

© C

op

yrig

ht 2

00

5 P

CP

I. A

ll rig

hts

re

serv

ed

wo

rldw

ide

64

International Journal of Coaching in Organizations

We have encountered some employers who struggle tosystematically evaluate traditional forms of training anddevelopment, let alone adapt these arrangements forevaluation coaching programmes. So why are weworrying about coaching evaluation? According toprevious IES research on training evaluation8, mostevaluation of management development takes place atthe ‘reaction level’ of Kirkpatrick’s model of evaluationwithout much effort placed at the learning orbehavioural change levels. If this is the norm forevaluation for other forms of training and development,should we settle for this level of approach to coachingevaluation as well? We say no. In practice, and fromdiscussing the issues with IES Research Networkmember companies, there seems to be a need todemonstrate the contribution of coaching in more detail,at the levels of learning gained, behaviours changed, oroutcomes achieved either individually ororganisationally.

According to a survey of large UK based organisationsin 20029, 92 per cent of companies contract externalsuppliers called ‘executive coaches’, with 65 per cent ofpublic sector organisations following suit. Amongrespondents in both the public and private sectorspredictions were that would continue to increase overthe next five years. Despite this widespread adoptionof coaching in the UK, coaching is still regarded as arather new, unproven and less established form ofdevelopment in comparison to classroom based courses.Because of its ‘newness’ coaching is more likely to bescrutinised in terms of its impact on development thantraditional forms of development that are wellestablished in organisations. More traditional forms ofdevelopment may have limited evidence of impact onindividual and organisational behaviour change, butthey are more accepted and less likely to be challengedin terms of their value. An attempt at addressing keyevaluation issues in coaching is therefore becomingimportant in establishing its credibility as a successfulintervention. If we take existing models such asKirkpatrick and try to apply them to coaching, what arethe key issues to emerge?

Defining OutcomesEvaluating coaching presents a number of problems fortraditional training evaluation models, including thatof Kirkpatrick. Coaching is often conducted less formallythan training interventions, based on one-to-onesessions which may not be related to any specificprogramme. A particular issue here is that coachingmay start without formal aims, objectives or outcomeswritten by the organisation that would almost certainlybe central to any formal training course they planned tocommission. It may be that the coaches hired are alsounable or unwilling to have the up-front conversationabout what organisational results are intended from

the coaching intervention. This lack of clear aims andobjectives at the start of the process causes realdifficulties for evaluation of the process. Definingoutcome measures of an intervention in developmentand training is the cornerstone to most evaluationmodels as it provides a basis for conducting yourevaluation. The lack of such common practice perhapsis the single biggest difference between the evaluationof coaching and that of other approaches.

The credibility of coaching depends on being able tohelp achieve set outcomes for the coaching process. Yetas coaches know, the nature of coaching means that itoperates at a very individual level where differentpeople will get different things out of coaching. Wheretraditional approaches to training aim to have a levelplaying field by the end of the session, where everyoneknows the same thing to the same level, coaching isindividually focused. This has a major impact onevaluation in that no set standard measurement criteriafor coaching outcomes can be set to cover all coachinginterventions across an organisation. However, this doesnot mean that no evaluation at the organisation levelcan be taken as criteria can be set within a broaderframework.

Corporate purchasers of coaching services told us thatthat they find coaching sessions which are not linkedto any wider development programmes or feedbackexercises the most difficult to evaluate. This is becausethey do not appear to link to any particular individualoutcome or organisational contribution. Where coachingis purchased as one element of a more traditionalintervention (e.g., a leadership development programmeor cultural change programmes) that are being formallyevaluated, outcomes will have been articulated for theprogramme and there may be some way of linkingevaluation to these outcomes. While the effect on theindividual through self-report may be positive, thestrategic value for organisations as well as individualsneeds to be addressed.

Published accounts of the executive coaching process,10

and best practice guidance for coaches, suggest anumber of common processes involved in coaching,including setting goals or reaching a sharedunderstanding of what will be discussed. Some of thecommonly agreed features of executive coaching orbusiness performance coaching are that it is goalspecific, action- and performance-oriented, andobjective. These elements all lend themselves to ameasurement framework and hence evaluation at thelevel of each individual being coached. However, thesegoals and objectives often emerge as part of the process,rather than existing at the beginning as part of thecourse. Once goals are established within the coachingprocess, these then lead to defined success criteria that

An Open Forum (2005/Issue Four)

65

could be linked to evaluation measures. The key to howthis might operate in practice would be to decide: Whatare these elements or success criteria and outcomes?Which of these elements are measurable and when? Bywhom? And what is the overall value to both theindividual and the organisation?

So if it is reasonable to assume that professional coachesare already encouraging their coachees to measuresuccess at the individual level as a matter of course,employing organisations need to be clear about theirinterests. Is it that they want to be inside the ‘loop’ andbe party to the confidential information individuals aregenerating with their coaches about their individuallevel successes? Coaches and individuals mightjustifiably resist this. Or should employers rely on thegood performance of the coaches they hire to ensureindividual success is articulated and instead spend theirevaluation time and effort on identifying organisationlevel success?

In our discussions with UK organisations, we found alack of clarity about the nature and implication of thischoice. It seems to us that these two options are verydifferent. Training and development functions areincreasingly organising large scale coaching-basedprogrammes linked to leadership development orculture change initiatives, sometimes involving multiplecoaching providers. In these circumstances we believethe organisational effort would be better spent in aproper definition of organisational success criteria,organisational outcomes and organisation levelmeasures. It is in this area that employers feel evaluationresearch is failing them. There may be plenty of advicefor coaches on how to evaluate at the individual level,but where are the examples for employers of evaluationpractice at the organisation level? And where is theadvice to employers on models and how to apply them?In short, they perceive there to be missing pieces to theevaluation research jigsaw.

Evaluating the ‘Bigger Picture’Research on evaluation of mentoring suggests thatdetermining the contribution of mentoring to achievingthe objectives of broader organisation-wide initiativesis also a problem11. Determining and distinguishing theeffects of coaching are perceived by corporatepurchasers as difficult enough; then to link these effectsto the achievement of specific objectives in a broadercontext is even more difficult. Many other factorsinfluence progress towards objectives. (See the O’Neillarticle in 2005 Issue 1 of this journal for how to teaseout other causal factors.) Although no direct causal linkcan be claimed, it is perfectly possible to clarify theexpected logical links between coaching and theachievement of objectives within the broader initiatives.For example, relating outcomes to the broader initiative

or as McGovern and colleagues12 have done bydemonstrating the chain of impact. However, thequestion still remains, how do you evaluate thecoaching element, in particular, from the organisationalperspective rather than from that of the individual orcoaches? This leads us to consider from whoseperspective the evaluation is taking place.

Different Perspectives on Evaluating OutcomesThe objectives and aims of the evaluation, particularlyin terms of which outcomes are measured, will also differaccording to the various stakeholders involved in theprocess. For example, the organisation may want anyevaluation to demonstrate the impact of the coaches onwhatever initiative they have been brought in toinfluence (e.g., leadership, cultural change).Individuals, on the other hand, may have a different setof criteria with which they evaluate the success of thecoaching and may be related to those agreed upon inthe coaching session, rather than the broaderorganisational aims. While these criteria may often becomplementary and occasionally the same they maynot always be the same, and so any evaluation strategyneeds to be explicit as to whose criteria and outcomes itis measuring. Where success criteria are different, bothperspectives are to be considered. The same principleof multiple stakeholder perspectives may be given foran external coaching company who may want todemonstrate the influence of their coaches on a clientorganisation initiative, in addition to achievingindividual outcomes and success criteria. They wouldneed to employ evaluation tools that evaluate from bothperspectives.

Qualities of the Coach and Coaching RelationshipA further significant issue is the contribution of thequality and abilities of the coach to the outcomes.Researchers and practitioners alike agree that themoderating effect of the quality and skills of the coachis perceived to be more important in coaching than therole of the trainer in a traditional training event. This isdue to the suggestion that the effectiveness of anycoaching intervention is primarily determined by theability of the coach – what was it about them and thequality of the coaching relationship that contributed tothe changes and outcomes observed? Therefore,employers seeking a comprehensive evaluation strategywould need to consider mechanisms which evaluatenot only the changes in outcome variables, or ‘successcriteria’ as a result of coaching, but also assess the natureof the relationship that helped to bring about thosechanges. Such a comprehensive approach would allowthe organisation to identify underlying factors ormechanisms that influenced the results. Not allemployers may choose to take on this additionalperspectives, although we would recommend it sinceany common or corporate’ blockages’ getting in the way

66


to achieving desired results, once identified, can beremoved by the organisation.

Case Studies of Evaluation PracticeDespite the problems and difficulties previouslyoutlined, our IES Research Network study of coachingevaluation practices by employers13 did come acrossmany organisational examples. Although very few ofthese examples were overtly based on an evidence-basedmodel of evaluation, we nevertheless consider them tobe a useful contribution. They were planned activitiesdesigned to assist the organisation in makingjudgements about how the coaching was going and tomake decisions about future coaching provision in thelight of these judgements. If evaluation is essentiallyabout putting a value on things – and we believe it is –then these companies were engaging in coachingevaluation.Since employers seem to believe theacademic research literature is not providing enoughnew insights in this area, the contribution of reflectivepractitioners in organisations should perhaps bepromoted.

In this section we share with you three differentexamples of applied evaluation research, and ourthoughts on their potential for wider application.Weencountered three basic categories of evaluation in thepractitioner literature and in practice underpinningwhat organisations were doing: they were seeking toprove something, improve something or learnsomething. Our three chosen case study examples, onefrom each category, are not presented as exemplars ofgood practice, but rather as examples of ‘home-grown’practical approaches used by organisations in theabsence (as they would describe it) of clear advice fromresearchers and coaches about how they should bedoing it.

Case Study 1: Financial Services Provider14

In June 2003 the Head of Training and Recruitment at amajor UK financial services provider was consideringsupplementing existing training programmes with one-to-one follow-up coaching sessions after training hadbeen delivered. As well as helping to transfer newlylearnt skills into the workplace, it was also hoped themove would support the creation of a coaching culturethroughout the organisation. Before proceeding thecompany decided the best approach was to quantifythe likely effectiveness by running what they called a‘test pilot’. We would describe it as a research project.The performance of the sales force was selected as thebest output measure for a small-scale test pilot. Theselling skills training course was targeted as the bestinput activity to research. The sales performance of 24financial planning managers was followed over a six-month period. For each planner, sales figures from thethree months prior to the training were compared to the

three months following the training course. Twelvemanagers attended the course, six of these wereassigned an external coach for three months followingthe course; the other six had no external coaching butdid receive the usual follow-up support from their linemanager. The remaining 12 managers acted as a ‘controlgroup’ as they did not attend the training course andtherefore had no follow-up support from a coach or theirmanagers.

The dramatic results led the organisation to value theirtraining programme, as well as emphasise howimportant some kind of effective follow-up regime is toembed learning. For the control group, sales grew byfour per cent. From this the company concluded thathad it done no training or coaching at all other internalor external factors such as market conditions meantbusiness would have increased by four per cent anyway.For the training-only group, sales increased by eightper cent, double the increase of the control group. Forthe group who received training plus coaching salesincreased by 27 per cent.

Case Study 2: Local Councils in England15

Achieving improvements in public services is acornerstone of current central government policy inEngland.Service improvement initiatives areunderpinned by a number of infrastructure projectsincluding building HR capacity and leadershipcapability in local councils. A pilot HR coachingprogramme was designed by one of the authors todetermine whether coaching might contribute toimprovements in strategic thinking, and contributionto corporate leadership, among senior HR professionals.Twenty-four HR participants were recruited from 24councils across the Northwest and eastern regions ofEngland. The pilot programme comprised a series ofone-to-one coaching sessions supplemented by twoseminars during the period February – September 2004.A comprehensive approach to evaluation was taken.Data collection methods used included: mid-point in-depth telephone interviews with all participants; endof programme impact questionnaires for participants,their coaches and their organisation sponsors (typicallythe CEO); separate focus groups for participants andcoaches/programme organisers; and telephoneinterviews with coaches. These generated a substantialvolume of information regarding the experiences andimpact of the programme.

Of particular interest was the opportunity to explorediffering perceptions of behavioural change. In order toassess any potential differences between the three groups– participants, organisation sponsors and coaches – amatched triad methodology was adopted. This involvedisolating from the impact questionnaire data only the


67

ratings data that we had from all three groups. Therewere ten participants that fell into this category whoseresponses we examined in parallel with those of theirindividual sponsors and also their coach. Tenparticipants represent 42 per cent of the total participantpopulation and 50 per cent of the participant samplereturning impact questionnaires. The results have to beinterpreted with caution due to the small sample size,but they are interesting nonetheless.

The end-of-pilot impact questionnaires had soughtinformation relating the perceived and actual impact ofthe coaching along three key dimensions of behaviouralchanges:

· Personal effectiveness· HR function leadership· Strategic thinking and corporate leadership.

Within each of these areas a series of statements waspresented to all three groups who were asked to ratetheir level of agreement along a standard five-pointLikert scale. For each thematic grouping of statements,factor analysis was then performed to ascertain theextent to which the items used were in fact assessingthe same concept. Distinct differences in the reliabilityof conceptual clusters can be seen between the threegroups, highlighting that different groups of individualsare likely to hold divergent views of the same issue.

Along all impact dimensions, and for all items, positivemean responses were found for all of the evaluationgroups. This indicates the required changes inbehaviour were thought to have taken place. However,participants consistently rated the extent of changemuch higher than their sponsors did – on 15 out of 19assessment items. Sponsors did rate the impact of thecoaching positively along all of the items, but to a lesserdegree than their participants.

The ten participants who were selected for this furtheranalysis reported higher levels of engagement in thecoaching process than the other participants whosesponsors did not complete the evaluation. The meanscores were at a level higher than for the entireparticipant population group. Perhaps this suggeststhat higher levels of organisational support, andproactive sponsor engagement, lead to higher levels ofperceived impact from coaching. The implication forcoaching is that we can be confident that engagementon all sides, not merely from participant and coach, isvital to the overall success of the programme. Withinthe HR coaching pilot programme, there wereparticipants who reported they were unable to find orname a sponsor who was supporting their applicationand development in this process. Where this was the

case, participants also tended to comment (in theinterviews and focus groups) that they felt more isolatedin their development process, and subsequently mayalso have supplied lower levels of agreement with theimpact assessment statements. Coaching may be highlyindividualised, and focused on personal development,yet these findings would support the view that for thegreatest levels of behavioural change, support withinthe organisation is vital.

Case Study 3: Public Healthcare Providers in Wales16

Following changes to the regional political landscapein Wales following the birth of the Welsh Assembly inApril 2003, the Centre for Health Leadership in Walesidentified that the time was right to broker a developmentprogramme for CEOs and Directors across the publichealthcare sector. The programme aimed to kick-startthe creation of a more innovative organisation culturethat valued new ideas, new thinking and managed risk-taking. Twelve coaches with a good knowledge of thesector, selected from five different external coachingcompanies, provided four one-to-one coaching sessions.Thirty-two CEOs and Directors participated as‘coachees.’

An independent institute was asked to conductevaluation on behalf of all the programme sponsors andpartners. The evaluation approach focussed on twoquestions:

· Has the coaching intervention taken place in the wayintended?· What can the organisations learn from the intervention?

A variety of evaluation methods were used, many ofwhich were based on ‘self-report’ by coachees. First, allcoachees were asked to provide some ‘baseline data’ atthe start. This included information about the biggestchallenges they faced in their role, biggest workplaceissues, and which networks they were already involvedin.

After each coaching session every coachee was askedto complete a score sheet to generate data useful forquality assurance purposes. Aspects scored includedtiming, quality of coach/coachee relationship andcontent. Coachees were also asked to identify the mosthelpful element in each session. After their final session,all coachees were asked to complete a questionnaire onmore general issues such as motivation to joiningprogramme; criteria they used in selecting which coachto work with; models/inputs particularly helpful; andwhat they have done differently as a result. Six of the 32coachees were also interviewed face-to-face to allowsome of the issues to be explored in more detail.

68


The perceptions of coaches were also collected. Theywere asked at the outset what issues they imaginedmight be key. The idea was to go back to the coachesagain after the final session to compare these with theactual issues arising. Finally coach/coachee pairs wereasked to write a ‘learning vignette’ covering what theywere working on and how successful the coachingrelationship had proved.

The findings revealed that the opportunity offered bythe coaching had definitely been perceived and utilisedas a management learning tool, rather than a personaldevelopment one. Coaching as a learning method wasalso viewed positively. Coachees valued the individualcare and attention, and the acquisition of easy-to-applymodels. The programme was also thought particularlytimely in terms of inter-professional learning and had apositive effect on the personal morale and motivationof coachees.

The evaluation also raised some questions. Theimplications of using ‘known’ coaches – who wereformerly senior employees within the sector – mighthave sent mixed messages to coachees if the programmewas about retention. Coachee comments from theinterviews suggest that using ‘role models’ who haveleft an organisation can be seen as something of aparadox.

Factors Helping orHindering Evaluation Practice

During our discussions with employers we formed theview that coaching evaluation is an activity that almostall corporate purchasers agree is important but onewhich is often forgotten in the perpetual rush to getthings done. Where it was taking place it rarely felt likethe neutral and objective exercise that practitioners feelresearchers describe and promote.

Case Study 1 was a small-scale example of intent toprove whether an increase in sales, a key organisationalperformance measure, occurred when coaching wasintroduced to support sales staff after their attendanceon formal training courses. It is an interesting examplebecause the organisation did not attempt to evaluatethe likely impact for any other staff group, althoughcoaching was being considered for many groups, norin any other context than post-course support. Theapproach was for the organisation to focus its energywhere they believed it was easiest to measure possibleorganisational impact. The idea was then to use thesefindings as ‘reasonable evidence’, though not the sameas claiming ‘proof’, that there might be benefit for othergroups or contexts. This avoided unnecessaryadditional expense and the complexity of measuringelsewhere where it might be less straightforward toidentify an appropriate organisational measure of

impact. It is also interesting because the evaluation wasdesigned by the organisation in partnership with theexternally sourced coaches.

The in-house research project in case study 1 stoppedshort of a cost-benefit analysis or return on investmentformula. When we enquired why, we were told therewas no need because everyone could clearly see thatthe increase in sales significantly exceeded the cost ofthe training and coaching.

As researchers we are well aware that problems of‘causality’ can be viewed as a big barrier to evaluation,especially for those employers that might be seeking to‘prove’ something. There is no logical causation betweenimproved business results and the fact that there hasbeen coaching17 and such links are interpretative. Manyvariables affect business results including linemanagement actions and competition in the market. Justbecause sales went up by 27 per cent in Case study 1does not mean it was because of the coaching. Indeedone could argue that it may have gone up by more hadthere been no coaching.

It is interesting to find that the employer in case study 1did not worry as much about causality as researcherstend to. The search for ‘proof’ to the Head of Trainingand Recruitment and her Board of Directors is a test of‘reasonableness’ of the chain of impact being claimed.A scientific level of proof was not required. The use of acontrol group, comparing financial planning managerswho received coaching with those who had not, wascertainly a good idea in understanding the issues ofcausation and we would commend the organisationfor this step. Interpretation was still needed though.For instance, one could argue that the group who werenot offered coaching or training and the group notoffered add-on coaching may have felt less favourablytreated and might have had lower levels of motivationthan the coached group.

This, one could argue, may explain the poorerperformance of some groups. So whilst employers inthe UK seem to agree that claiming business results isnot proof in a strictly scientific sense, companies canremove the weight of the ‘causality’ barrier by adoptingthe pragmatic attitude of the employer in case study 1.The key question for applied evaluation research is thusredefined as ‘do we believe the coaching helped achievethe business outcome?’

Case Study 2 is an example of the most common intentwe encountered: to improve a flagship developmentcoaching programme utilising other learning methodsbut predominantly one-to-one coaching. This confirmsthe findings of another study that found that feedback


69

from participants was the most common approach toevaluation by employers.1818

Case study 2 involved an in-depth evaluation by twoof the authors using multiple data collection methodsto identify behavioural changes that occurred inindividual leaders involved in a pilot coachingprogramme. The idea was to use this information toassess whether the desired behavioural changes werebeing realised. It was also expected that any necessaryincremental improvements to the programme would beidentified, and these might increase the extent ofbehavioural change for individual leaders embarkingon future cohorts of the programme.

Instead of evaluating direct business outcomes, thecoaching in case study 2 was measured throughsubjective judgements on whether participants hadimproved their effectiveness. This is what we asresearchers would call ‘perceptual evaluation.’ It is, ofcourse, perfectly possible for in-house practitioners tocarry out this kind of measurement in a standardisedway and you do not have to hire external researchers.Structured questionnaires can be administered toparticipants before and after the coaching to comparetheir pre-and post- coaching perception of their owneffectiveness. 360-degree feedback questionnaires canalso bring into the evaluation process the perceptionsof their line managers, customers, suppliers, peers andown staff.

UK employers we contacted had mixed views aboutperceptual coaching evaluation. On the one hand theyfound it much easier to conceptualise than measuringbusiness outcomes because there were fewermethodological barriers or hindrances to overcome. Onecould still argue about causality, but if all parties agreedthat the required behaviours were being displayed, asin case study 2, employers told us that evidence that thedesired outcome had been achieved made it irrelevantas to what extent the coaching was responsible. Butwhilst perceptual evaluation was easier toconceptualise, on the other hand employers perceivedit to be much harder to implement. Insistence fromparticipants and coaches that such data should betreated as confidential hindered attempts to get a goodresponse rate, and casts doubts on the generalisabilityof the results.

In Case Study 3 we would categorise the primary intentas being to learn about the potential of coaching as atool for organisational learning and culture change. Wewould describe this example as being primarily aboutcollecting reactions to the coaching. The organisationin case study 3 hired an external evaluator to explorethe issues arising for individual leaders from

participating in a coaching programme. However, anin-house team could just as easily collect and explorereactions for itself using questionnaires or shorttelephone interviews or focus group interviews.Response rates from ‘reactionnaires’, as questionnairesto assess participants’ reactions are sometimes known,can be improved considerably by asking coaches or linemanagers to distribute and collect the questionnaires.Sometimes forms are even completed as part of the finalcoaching session in the presence of the coach. However,our experience suggests the data may not then be asreliable as coachees may feel protective of their coach orunder pressure to give positive responses.

Concluding Messages forCoaches and Corporate Purchasers

Corporate PurchasersIn this article we have explored issues in evaluatingcoaching in an organisational setting. We have arguedthat there is a missing piece of the jigsaw as far asemployers are concerned in that not enough advice andguidance are available to them. We perceive that thekey issue for coaching evaluation remains the same atboth organisational and individual level, i.e., that thecrux of any evaluation is defining the criteria formeasurement in terms of success or outcomes. Towardsthe beginning of an engagement, there may be difficultiesin a corporate purchaser getting too involved in this atthe individual level because of the nature of coachinginterventions and that providing much greaterencouragement and support for coaches and coachesto do a better job on evaluation may be a more pragmaticway forward at the individual level. However, webelieve that there are some clear improvements thatcorporate purchasers can make to their coachingevaluation practice at the organisational level whichare likely to prove more effective that the currentwidespread use of routine ‘reactionnaires’. We suggestthat corporate purchasers giving consideration to fiveaspects will help.

Put aside your usual training evaluation model.Firstly, corporate purchasers from HR or trainingfunctions need to understand that the traditionalmodels of training evaluation they may use elsewheremay not apply very well with coaching interventions.This is because much of these traditional trainingapproaches are based on the premise that objectives areset prior to training, usually via a training needsanalysis. Traditional training evaluation is then linkedto measuring whether these objectives have been met.In coaching, training needs analysis is often carriedout within the coaching process from which aims andobjectives emerge. Therefore it is more appropriate forindividual level evaluation of behaviour change andimpact of coaches to be assessed after this point. Thus

70


corporate purchasers may do better (for the purposes ofevaluation) to conceptualise coaching interventions asless like training programmes and more like clinicaland counselling consultations where, following aperiod of diagnosis and problem generation, plans foraction are defined with appropriate measurement ofsuccess and rules of exit for the counsellor.

Talk to your coaches. Having urged caution in applyingtraditional training evaluation models, we still believethere are lessons to be learned at this individual levelby organisations. Specifically, we suggest that corporatepurchasers enter into an early dialogue with prospectivecoaches to make clear their expectation that onceindividual level objectives emerge, some form of agreedevaluation process is put in place. You can also clarifyany data collection you expect the coaches you hire tocarry out for you, when, in what format and by whom itwill be seen. This is also the time to agree in advancehow any confidentiality issues will be handled by theparties. Such conversations must also be held with thecoachee so that there is a shared understanding of whatremains between coach and coachee, and what may bediscussed (and at what level of detail) with a manageror HR.

Develop standardised evaluation tools and guidanceabout methodology. Whenever you use tools, there willbe an accompanying methodology for itsimplementation. This will require co-operation and co-ordination by the organisation. Contractingconversations are where these issues can mosteffectively be broached, discussed, and a course of actionagreed on.

We suggest that organisations be far more proactive inoffering guidance before coaching sessions commenceto help both coaches and organisations in selectingappropriate evaluation tools to match objectives thatare set. These tools need not be cumbersome oflaborious. Making tools available, together withguidance on careful selection of methods at an earlystage will provide useful data for the individual, coachand organisation further down the track. Certainly itwill be a significant improvement on the ad hocapproaches to evaluation that many organisationscurrently engage in. Through this combination of earlydialogue together with planning ahead by setting outmethods and tools corporate purchasers feel areappropriate to use in evaluation in their setting,organisations should be able to identify much betterwhether they are achieving good practice andconsistency across their coaching provision.

Broaden your view of ‘success’. Take a leaf out of thecase study examples in this article by not just focussingon outcomes per se when it comes to coaching

evaluation at the organisational level. If you can link toquantifiable financial or sales targets, that’s great. Ifnot, consider focussing on organisational barriers orenablers to successful transfer and change (and henceoutcomes). The transfer climate within organisationsand the opportunity for use are well researchedorganisational level areas that predict whether peopleuse newly acquired learning and can be a useful proxyfor understanding why outcomes and successes haveor have not occurred. Doing this should alloworganisations to make sense of any information theyget about outcomes as well as generating feedback onthings that can actually be changed.

Access a range of viewpoints. We suggest morecorporate purchasers consider the potential value ofmore comprehensive evaluation packages. Differentialviewpoints clearly exist and more than the view of justthe coachees are needed for a fuller view of successcriteria and achievement. Coaching is often viewedfavourably by those being coached. This rings true tomany corporate purchasers who may get very positiveanecdotal reports from enthusiastic individuals andleave it at that. Be warned: you may get an overly rosypicture if you only ask those who have been coached.Value and success will look very different whenobserved by different audiences. Whilst your two mainpriority audiences may be the organisation as a wholeand an individual who has been coached, don’t forgetthe coaches, an individual’s boss, the programmesponsor or budget-holder, the colleagues or teammembers of coachees and the coaching schemeorganisers/managers.

CoachesOur primary focus in this article has been on corporatepurchasers. We recognise that many coaches areroutinely encouraging their coachees to define andmeasure success as a matter of course and, we hope,many will welcome our call for organisations to becomemore ‘intelligent’ evaluators of coaching outcomes.Other coaches may feel that evaluation should remainpart of the coach-coachee relationship and will besuspicious of an organisation’s motives. It is not for usto tell other coaches what they should be doing. Wesimply believe that in an increasingly crowdedmarketplace for coaching suppliers, good coaches mayneed to do more to make sure they stand out. One wayto enhance their professional credibility may be bydemonstrating a willingness and ability to contributeto a company’s thinking about its organisational levelcoaching evaluation strategy. For coaches wanting togo down this route we suggest two actions.Be willing to engage in different evaluation processes.We urge all coaches to engage fully in evaluationprocesses and not just leave it to individual coaches


71

and their companies. One implication of this is thatcoaches will also need to be willing to use differentevaluation tools for the different organisations theyoperate within. Coaches will therefore need to be lessattached to their preference for their usual evaluationmethods. By doing this, they can be more responsive tocorporate purchasers’ increasing need to ‘own’ orjointly own (with coaches and coaches) some aspects ofthe evaluation process.

Be pro-active about evaluation conversations. We hopethat coaches will become braver about having up-frontcontracting conversations with prospective corporatepurchasers which use and push the language ofoutcomes and success criteria. We would also hope morecoaches will encourage their coaches to engage withtheir managers or HR departments about at least someof their objectives. In that way, organisations can bereassured about the existence of some line-of-sightbetween individual objectives and organisationalobjectives.

_______________________________

Alison Carter, Ph.D. and Helen Wolfe

Email: [email protected]: www.employment-studies.co.uk

Dr Alison Carter is Principal Research Fellow at theInstitute for Employment Studies (IES) in the UK, whereHelen Wolfe is a Research Officer. Alison also headsup the IES HR Coaching Service, is a founding Directorof the European Mentoring and Coaching Council(EMCC) and Co-Director of the UK’s Employee andManagement Development Research Network.

Máire Kerrin, Ph.D.

Email: [email protected]

Dr Máire Kerrin is Director of OrganisationalPostgraduate Programmes in Psychology at CityUniversity, London. Máire has a particular interest intraining evaluation methodologies, supply chainpartnering, and knowledge management.

________________________

Endnotes

1 Joy McGovern, Michael Lindemann, Monica Vergara, StaceyMurphy, Linda Barker and Rodney Warrenfeltz. “Maximisingthe impact of executive coaching behavior change,organization outcomes and return on investment”, TheManchester Review, US, 2001, Vol. 6, No 1; Gerald Olivero,Denise Bane and Richard Kopelman. “Executive coaching as atransfer of training tool: Effects on productivity in a publicagency”, Public Personnel Management, 1997, Vol. 26, No 4,who investigated the outcomes of executive coaching in a USpublic sector agency.

2 Cheriece Genger. “Coaching: Theory and practice”,unpublished masters thesis, University of San Francisco,California, 1997, investigated the effectiveness of executivecoaching practices through quantitative and qualitativemethods.

3 Otto Laske. “An integrated model of development coaching”,Consulting Psychology Journal: Practice and Research, US, 1999,Vol 51, who explored the transformative effects of executivecoaching on an executive’s professional agenda.

4 William Judge and Jeffrey Cowell, who surveyed currentexecutive coaching practices, “The brave new world of executivecoaching” Business Horizons, 1997, Vol. 40 Issue 4; DouglasHall, Karen Otazo and George Holenbeck, “Behind closeddoors: What really happens in executive coaching”,Organizational Dynamics, 1999, Vol. 27, who interviewed bothexecutives and coaches regarding executive coaching practice,effectiveness and future directions; and S Foster and J Lendle.“Eye movement sensitisation and reprocessing: four casestudies of a new tool for executive coaching and restoringemployee performance after setbacks”, Consulting PsychologyJournal: Practice and Research, US, 1996, Vol. 48.

5 Donald Kirkpatrick, “Evaluating training programs: evidencevs. proof”Training and Development Journal, 1977, Vol. 31;Donald Kirkpatrick. “Four Steps to Measuring TrainingEffectiveness”, Personnel Administrator, US, 1983, Vol. 28, No11.

6 Joy McGovern, Michael Lindemann, Monica Vergara, StaceyMurph, Linda Barker and Rodney Warrenfeltz, “Maximisingthe impact of executive coaching behavior change,organization outcomes and return on investment”, TheManchester Review, US, 2001, Vol 6, No 1.

7 Jack Phillips. Return on investment in training and performanceimprovement programs, Houston, TX: Gulf PublishingCompany, 1997.

72


8 Penny Tamkin, Jane Yarnall and Máire Kerrin, “Kirkpatrickand beyond: A review of models of training evaluation”, IESResearch Report No 392, UK, 2002.

9 CIPD Annual training and development survey, CharteredInstitute for Personnel and Development, UK, 2004.

10 Alison Carter, “Executive coaching: Inspiring performance atwork”, IES Research Report No 379, UK, 2001.

11 James Wilson and Nancy Elman, “Organizational benefits ofmentoring”, Academy of Management Executive, 1990, Vol 4, No4.

12 Joy McGovern, Michael Lindemann, Monica Vergara, StaceyMurph, Linda Barker and Rodney Warrenfeltz, “Maximisingthe impact of executive coaching behavior change,organization outcomes and return on investment”, TheManchester Review, US, 2001, Vol 6, No 1.

13 Alison Carter, Máire Kerrin and Helen Wolfe, “Methods forevaluating coaching”, IES Research Report No 410, UK, 2006(forthcoming).

14 Jan Sherwood, “Does coaching actually work?”, InternationalJournal of Mentoring and Coaching, European Mentoring andCoaching Council (EMCC), 2004, Vol 3, Winter.

15 Government in the UK is not federal, but some services areprovided by local authorities rather than from the center.

16 The National Health Service (NHS) provides the vast bulk ofall healthcare provision in the UK, paid for from generaltaxation. This situation is quite unlike that in the US.

17 Alison Carter, “Practical methods for evaluating coaching”,Training Journal, UK, January 2004.

18 CIPD, A guide to coaching and buying coaching services,Chartered Institute for Personnel and Development, UK, 2004.

The International Journal of Coaching in Organizations (IJCO) is the signature publication of

Professional Coaching Publications, Inc. (PCPI). In addition to this internationally acclaimed

journal, PCPI publishes books on topics of interest to those in the coaching community, whether

practitioner, decision maker, or end user. You can count on PCPI, Inc. to provide content that

pushes the envelope — bringing theory, research and application together in ways that inform,

engage and provoke. Visit the PCPI website, www.pcpionline.com, to view and purchase our

growing line of products.

If you have administrative questions, please refer them to our IJCO Office Manager, at

[email protected]. For advertising, marketing and operations inquiries, please refer

them to John Lazar, IJCO Co-Executive Editor, at [email protected]. Please submit unsolicited

manuscripts for peer review consideration to the IJCO office manager at [email protected].

Visit Both Our Sites at Your Convenience

Journal information:www.ijco.info

Purchases:www.pcpionline.com

Resource Center for Professional Coaching in Organizations

employers and coaching evaluation - icf - search...

Documents