ARTICLE IN PRESS

Evaluation and Program Planning 31 (2008) 259– 265

Contents lists available at ScienceDirect

Evaluation and Program Planning

0149-71

doi:10.1

� Tel.:

E-m

journal homepage: www.elsevier.com/locate/evalprogplan

Revisiting a theory of negotiation: The utility of Markiewicz (2005) proposedsix principles

Diane McDonald �

Centre for Program Evaluation, University of Melbourne, Victoria 3010, Australia

a r t i c l e i n f o

Article history:

Received 10 May 2007

Received in revised form

2 February 2008

Accepted 31 March 2008

Keywords:

Stakeholder decision making

Empowerment

Dealing with conflict

89/$ - see front matter & 2008 Elsevier Ltd. A

016/j.evalprogplan.2008.03.003

+613 9787 3003; fax: +613 8344 8490.

ail address: demc@unimelb.edu.au

a b s t r a c t

People invited to participate in an evaluation process will inevitably come from a variety of personal

backgrounds and hold different views based on their own lived experience. However, evaluators are in a

privileged position because they have access to information from a wide range of sources and can play

an important role in helping stakeholders to hear and appreciate one another’s opinions and ideas.

Indeed, in some cases a difference in perspective can be utilised by an evaluator to engage key

stakeholders in fruitful discussion that can add value to the evaluation outcome. In other instances the

evaluator finds that the task of facilitating positive interaction between multiple stakeholders is just ‘an

uphill battle’ and so conflict, rather than consensus, occurs as the evaluation findings emerge and are

debated.

As noted by Owen [(2006) Program evaluation: Forms and approaches (3rd ed.). St. Leonards, NSW:

Allen & Unwin] and other eminent evaluators before him [Fetterman, D. M. (1996). Empowerment

evaluation: An introduction to theory and practice. In D. M. Fetterman, S. J. Kaftarian, & A. Wandersman

(Eds.), Empowerment evaluation: Knowledge and tools for self-assessment and accountability (pp. 3–46).

Thousand Oaks, CA: Sage Publications; Patton, M. Q. (1997). Utilization-focused evaluation (3rd ed.).

Thousand Oaks, CA: Sage Publications; Stake, R. A. (1983). Stakeholder influence in the evaluation of

cities-in-schools. New Directions for Program Evaluation, 17, 15–30], conflict in an evaluation process is

not unexpected. The challenge is for evaluators to facilitate dialogue between people who hold strongly

opposing views, with the aim of helping them to achieve a common understanding of the best way

forward. However, this does not imply that consensus will be reached [Guba, E. G., & Lincoln, Y. S.

(1989). Fourth generation evaluation. Newbury Park, CA: Sage]. What is essential is that the evaluator

assists the various stakeholders to recognise and accept their differences and be willing to move on.

But the problem is that evaluators are not necessarily equipped with the technical or personal skills

required for effective negotiation. In addition, the time and effort that are required to undertake this

mediating role are often not sufficiently understood by those who commission a review. With such

issues in mind Markiewicz, A. [(2005). A balancing act: Resolving multiple stakeholder interests in

program evaluation. Evaluation Journal of Australasia, 4(1–2), 13–21] has proposed six principles upon

which to build a case for negotiation to be integrated into the evaluation process. This paper critiques

each of these principles in the context of an evaluation undertaken of a youth program. In doing so it

challenges the view that stakeholder consensus is always possible if program improvement is to be

achieved. This has led to some refinement and further extension of the proposed theory of negotiation

that is seen to be instrumental to the role of an evaluator.

& 2008 Elsevier Ltd. All rights reserved.

1. Structure of this paper

This article commences with a description of an evaluation taskassociated with a youth program that was at a point of crisis interms of its future viability. A brief overview of the evaluationprocess is then provided. This shows how the evaluator used an

ll rights reserved.

inductive approach to build on an initial evaluation plan negotiatedup-front with key stakeholders, to create a ‘grounded’ programtheory. The paper goes on to describe how the review processcontinued to emerge as the study progressed and moved moretowards what Patton (1996) describes as a ‘user-focused approach’.The final stage of this case study involved facilitating the youngpeople to describe their program model and led to the practicalapplication of the review findings in program development.

Having summarised the background and main elements of theevaluation, the author then discusses the case study experience in

ARTICLE IN PRESS

D. McDonald / Evaluation and Program Planning 31 (2008) 259–265260

light of Markiewicz principles, providing evidence to support orqualify each one. Finally some conclusions are drawn which,although limited to the one example, aim to extend the proposedtheory and stimulate others to document their own experiences ofdealing with divergent stakeholder interests.

2. Context of the evaluation

As a teenager in the late 1960s, I joined the local branch of aninternational youth movement which was led by young peopleand had a strong orientation towards social justice. It was herethat over the next few years I learnt about workers’ rights, stroveto address the exploitation of migrant women, protested againstthe unjust treatment of Aboriginal people and other disadvan-taged groups within the Australian society and campaigned forfreedom and equality in countries such as South Africa, Chile andEast Timor.

Before long I became a youth leader in this movement andrepresented Australia at national and international meetings.There I met other young people with similar beliefs. Together westruggled with conviction, passion and unwavering energy to‘change the world’. The experience of these years and of thepeople I met left an indelible mark on me as I grew older.

At some stage in the mid 1970s, my life moved on and beforelong I lost touch with the new generation of young people nowbeing recruited into the youth movement. But, like many of mycounterparts, I retained a firm commitment to the basic principlesand approaches that I had learned in those formative years. Thisincluded a strong belief in the value of self-reflection, experientiallearning and social action to effect lasting change. Over the yearsthese beliefs underpinned all aspects of my life and work.

3. A challenging task

By the beginning of the new millennium, I found myselfworking both as an evaluator at a University and as a privateconsultant. In 2005, I was asked to review the effectiveness ofyouth-led projects being implemented across six Australian Statesby the current members of the same organisation I had joinedwhen I was 18. This evaluation was commissioned by the Board ofthe funding body which financed these projects, all formermembers of the youth movement. A subsidiary objective of thestudy was to broaden the Board’s knowledge about the organisa-tion’s present-day situation.

I accepted the offer enthusiastically, but with some trepidation,since I had heard that the current membership of the youthmovement that I had known well and to which I owed so muchwas having great difficulty surviving in today’s world. I did notwant to be the one who sounded its death knell.

As is often the case, I found that my first task was to assistthose who had drafted the Terms of Reference for the review toclarify exactly what they wanted the evaluation to address. Likeother evaluators before me, I soon discovered that this was noeasy task (Owen, 2006) since the 10 Board members (four ofwhom were national youth leaders presently involved in themovement) had divergent and even contradictory opinions aboutthe purpose of the study. Not only this, but there werefundamental differences in relation to how the older membersof the Board perceived their role and what they believed theyoung people should be doing with the funds they were given.

There were two major concerns for me. One was the negativeperception that some of these older Board members conveyedabout the value of the existing youth program. The other was thenervousness of the younger Board members to articulate their

views within this forum. A complicating factor was that thefunding body had been established with the sole purpose ofmanaging investment funds secured by the youth movement inthe 1950s and had decided to use the interest solely for thepurpose of supporting the organisation’s ongoing development.Moreover, each Board member had been deeply affected person-ally by their own experience in the youth movement and thisaccentuated their individual convictions about what they shouldbe doing for the Australian youth of today.

Knowing that conflict between evaluation stakeholders is notuncommon at some point in an evaluation process, and that therole of the evaluator often includes awareness-raising andfacilitation between people with strongly opposed points of view(Owen, 2006), I supported the formation of an Evaluation SteeringGroup for the study comprising a sub-committee of both olderand younger Board representatives. This, I thought, would providea useful forum for everyone to hear one another’s differentperspectives and to come to a clearer, and hopefully, a commonunderstanding of the most suitable future direction for the review.

However, following a series of discussions with this group, andsubsequently with the Board as a whole, aimed at reachingagreement about the purpose and scope of the evaluation, I wasgiven a revised Terms of Reference which bore little resemblanceto my advice about how the review should be focused. Never-theless, I was encouraged by the broad acceptance of my idea forinvolving the two youth representatives from the Board in theevaluation process, to facilitate liaison with the State CoordinationTeams, assist with data collection and interpretation and helpidentify appropriate strategies for organisational and programdevelopment. So, after careful consideration of the potentialbenefits and risks of being associated with this study (both for theyouth movement and for my own personal and professionalreputation!), I decided to ‘take the bull by the horns’ and acceptthis evaluation challenge!

4. The evaluation process

My decision to involve the two youth leaders in the EvaluationTeam was based on my strong belief that for the findings to beuseful and used, it was vital that the views of young people beheard. It was also based on well-documented evidence about thebenefits of using more collaborative and empowering evaluationapproaches which can enhance utilisation (Patton, 1996, 1997)and a commitment to change (Preskill, 1996).

The evaluation process, which began as a three-monthindependent review of a number of funded projects, turned intoa more expansive piece of work which spanned a period of twoyears. It drew on some of the key features of several complemen-tary, interactive evaluation approaches such as participatoryevaluation (Cousins & Whitmore, 1998), fourth-generation eva-luation (Guba & Lincoln, 1989) action research (Kemmis, 1985),appreciative enquiry (Cooperrider, 1995) and empowermentevaluation (Fetterman, 1996). However, in reality it could not beclassified strictly as any one of these. Instead, it could be called a‘combination approach’—and in some respects could be said tohave incorporated a ‘seat of the pants approach’, which developedin response to the emerging context in which the study wasconducted.

Overall, the evaluation comprised three key components:conduct of the initial project review; provision of advice to boththe Board and the youth leaders about how to implement thereview recommendations; and facilitation of a developmentalprocess which identified a social vision and established a newstrategic direction for the organisation. All of this involved closeassociation with the two national youth leaders and periodic

ARTICLE IN PRESS

D. McDonald / Evaluation and Program Planning 31 (2008) 259–265 261

liaison with other members of the Board throughout the two-yearperiod.

In order to complement my efforts, I sought the support of anevaluation colleague who had no prior involvement in the youthmovement, on the Evaluation Team. Apart from his role inconducting some aspects of the project review and helping tofacilitate the social vision workshop, he provided a usefulsounding board for me to discuss the findings as they emerged.I considered his input invaluable in ensuring objectivity, just incase I became inflicted with the dreaded ‘Forrest Gump Complex’

(The Age, 2005)1 and tried to use the evaluation and thesubsequent program development phase to try to recreate some-thing from my own past!

Before beginning to collect the data, the Evaluation Team metto plan our ‘course of action’. This meeting provided a crucialopportunity for me and my evaluation colleague to learn moreabout the current context of the youth movement in the variousStates and the difficulties faced by the National Youth Executive intrying to coordinate and foster organizational growth. On thebasis of this feedback we agreed that the younger members of ourteam would make initial contact with the State Coordination Teamrepresentatives to explain the background to the evaluation andassure them of its appreciative and developmental focus. In doingso, they would encourage them to see the study as a learningopportunity which would increase the effectiveness of their futureefforts.

We also agreed, following my own experience of using anevaluation technique called the Success Case Method (Brinkerhoff,2003), to incorporate some aspects of this approach into the datacollection process and to invite the State-based teams to selectwhat they thought was the ‘most successful’ project in which theyhad been involved in the last two years and the project that theyconsidered ‘least successful’. The chief value of this strategy wasthat it would provide a simple and non-threatening way for us tofind out what worked and what did not in different contextsand to identify the key elements associated with success or thelack of it.

The field visits, which were undertaken between May andSeptember 2005, proved to be illuminating. Each one wasconducted over one or two days either by my evaluation colleagueor me, working alongside one of the younger members of ourteam. Although obviously a little nervous at first, CoordinationTeam members were very open in their discussions with us aboutwhat they felt they had achieved and the challenges that they hadfaced. Clearly, the presence of one of the national leaders at thesemeetings helped to facilitate our communication and to surfacethe key issues that needed to be addressed.

5. Encouraging change based on the findings

The project review situated the findings within the broadercontext being experienced by young people in today’s world. Itarticulated and affirmed the hard work being undertaken by asmall band of youth leaders in various parts of the country anduncovered some of the major obstacles that they faced and whatcaused them. It also pinpointed key areas where change wasrequired if the organisation was going to survive and grow. This

1 This reference, taken from a Melbourne-based daily newspaper, refers to the

fictitious movie character, Forrest Gump, who lived in the USA and never really

grew up. In the movie, we are given a chance to revisit key historical events that

occurred in the USA in the 1960s and 1970s, through the eyes of Gump, who had a

profound effect on everyone he met. This analogy refers to the author’s awareness

of the need to remain objective and not influence the outcomes of the review,

based on her own lived experience.

was documented and presented to the Board in the form of a DraftEvaluation Report. The report emphasised the pressing need forthe young people to review the structure and strategic direction oftheir organisation. It also highlighted the need for the Board andyouth leaders together to determine how former members(including those involved in the funding body) could help theyoung people to achieve their goals most effectively.

A few weeks after the report was circulated, the EvaluationTeam met the full Board to discuss the findings. At this meeting itwas clear that Board members were extremely grateful for thedetailed information provided and were united in their commit-ment that something had to be done, urgently, to ensure theorganisation’s ongoing viability. However, it was evident that theylacked accord in terms of how to respond optimally to the reviewrecommendations. Although apparently pleased at being enligh-tened about the current situation, I got the impression that someBoard members would have preferred that the evaluation processwould have allowed them to have more control in shaping theorganisation’s future. Others were emphatic that they wanted tohear what members of the youth movement (both current andpast) throughout Australia thought about the review findings andsuggested actions.

In the end it was agreed that I should finalise the report withminor amendments, and that it would be circulated widely toboth present and former members of the movement, with theintention of seeking feedback from a larger audience about thebest way forward. In the meantime no further annual fundingcommitments were to be made. This initial outcome was, for me,most unsatisfactory, since the underlying implication was thatunless the present-day members rallied to express their ideasabout the future direction of their organisation, then the views ofpast members might well take precedence.

Over the next few months while we awaited feedback on thereport, I stayed in touch with the two national youth leaders. Theywere keen to plan for the future but lacked any funding guaranteeto do so. Consequently, they asked my advice about how toconfront the immediate challenge presented by this stalemate andinvited me to address a national meeting of youth leaders.Participants at this forum openly discussed the issues raised bythe review and I encouraged them to be proactive in determiningthe most appropriate way to respond to the recommendations. Asa result, they developed a plan of action and forwarded it to thePresident of the Board.

At the same time I was aware that, although not included in myTerms of Reference, the review had clearly exposed the fact thatthe funding body lacked unity of direction and that this washaving a negative impact on the youth movement itself. Thus, thereview had taken on an organisational focus. I was invited toattend a second meeting to discuss the review findings. This time,those present comprised some of the older Board members and anumber of other people who had been part of the youthmovement in the 1950s, when it was at its peak. The groupexpressed deep concern about the organisation’s fragility andsought advice from those National Executive members who werepresent about what positive contribution they could make.

This discussion made it clear that what the young peoplewanted most was the active support of a committed group ofadults to help them to fulfil the organisation’s higher-levelobjectives in ways that were appropriate to today’s youth. Therewas also unanimous agreement within the group to encourage theyoung people to identify a common social vision that wouldinform their organisational structure and planning. Some of theparticipants backed this up by volunteering to take on specificsupport roles identified by the young people themselves. Follow-ing the meeting several written responses to the Board wereprepared that included suggestions of this nature.

ARTICLE IN PRESS

D. McDonald / Evaluation and Program Planning 31 (2008) 259–265262

The momentum developed from the discussions which tookplace at these two meetings led to a decision by the Board tosupport the young people’s plan of action. A few months later myevaluation colleague and I were invited to facilitate the socialvision workshop that was recommended in our review. As a result,youth leaders from around the country made a clear statementabout their primary goal and how they would work together toachieve it. This was further developed in State-based meetingsand ratified at a National Council comprising a broad cross-sectionof members. This was a significant step since, in the past,State branches had displayed little unity in their approach toprogramming.

The following extract from an e-mail sent to me by one of thenational youth leaders after this National Council expressessomething of what the young people saw as the outcome of thisevaluation activity, which initially they had feared:

There was a sense amongst all there that we are moving in anew and focused way, and a lot of energy was generated. (Now)we have a holistic plan that moves us in a direction of strengthand unityy I believe none of this would have been possiblewithout some clear recommendations from your report andyour personal commitment to help us begin to realise our idealworld and develop strategies on how to get there. I amdelighted that the role of (the funding body) has changed toreally support the development of the (youth movement) inAustralia (National President, 2006).

The courage shown by the young people in voicing their viewsabout their future direction so strongly had ramifications forthe Board as well. At its next Annual General Meeting some of theformer youth movement members who had met to discuss thereport and had expressed their desire to support the current youthleadership, nominated for Board positions. Several of these peoplewere elected. The personnel changes that resulted have helped torevitalise the funding body and have led to a real sense ofoptimism, cooperation and trust between older and youngerBoard members. This is reflected in a second e-mail extract, sentto me this time by the new Board President:

Since (the AGM)y we have made some good progress inestablishing a working relationship with the present (membersof the) movement and I feel excited about the direction theyare takingy Their National Councily was a very impressivegathering (President of the Board, 2006).

Moreover, this situation has also led to the realisation amongstthe Board members that it is now time to reassess their ownstrategic direction. When I look back, I feel somewhat surprisedthat a fairly straightforward project review has resulted in suchsignificant changes both for the youth organisation and itsfunding body.

6. Discussion of Markiewicz theory in light of the case studyexperience

The instrumental role played by an evaluator in helping toresolve the different interests of multiple stakeholders is wellknown (Owen, 2006). However, this responsibility, which cansometimes consume a considerable amount of time and energy, isgenerally not recognised by those who commission an evaluation.Particular technical and personal skills are required to undertakethese negotiation tasks effectively, but evaluators often lack theformal training required (Worthen, 1996). With this in mind,Markiewicz (2005) has proposed six principles which build a casefor negotiation being integrated into the evaluation process in

order to achieve stakeholder consensus regarding methodologyand outcomes. These principles are critiqued below in light of myexperience in the youth movement study.

6.1. Principle No. 1: recognition of the inherently political nature of

evaluation

Markiewicz recognises that evaluators operate in a politicisedcontext and this makes it imperative that they possess strong‘people skills’ to:

communicate processes and findings, facilitate conflict resolu-tion strategies, and engage stakeholders throughout theprocess of evaluation.

Whilst agreeing with the need for an evaluator to carry outthese tasks, I found that this role may sometimes also include aless conciliatory dimension. It was vital in the study describedabove that the evaluation expose the constraints being imposedon the development of the youth movement by the negativedynamic that was operating within the funding body Board itself.This was no easy task, particularly as the funding body hadcommissioned the review.

However, for the study to be effective and for the youthmovement to have new life, some hard decisions had to be made.The evaluation brought things to a head. In the end the focus ofthe evaluator’s skill was not so much to do with helping thedifferent stakeholder groups to find common ground, as onempowering the young people to speak out and to takeresponsibility for the future direction of their movement. Thisincluded encouraging the youth leaders to articulate an appro-priate role for former members of the youth movement insupporting them to achieve their objectives.

It was apparent that some Board members believed that, inorder for the youth movement to survive, a group of older, moreexperienced people needed to take greater control of theorganisation’s future. Such a strategy was completely alien tothe views of the young people. Because of the momentumgenerated by the evaluation, the Board members who soughtcontrol were eventually replaced by other people who wereprepared to commit time and energy to supporting the newdirection proposed by the youth leaders. Without this confronta-tion, it is very likely that the organisation would have sunk to evengreater depths and may even have folded.

6.2. Principle No. 2: valuing the contribution of multiple stakeholders

Markiewicz maintains that by collecting a variety of viewsfrom different stakeholders, the evaluator is adhering to socialjustice principles and is more likely to find truth and enhanceutilisation. My experience in the youth movement evaluationreinforces the value of a multiple stakeholder approach. Althoughthe collection of information from a variety of people can belabour intensive, lead to an overwhelming quantity of data thatmust be sorted and analysed and mean that the evaluator has todecide how to deal with conflicting feedback; the pluralism ofexperiences and insights obtained is likely to enhance the qualityof the findings and stakeholder commitment to implementing therecommendations.

In the aforementioned study, it was vital that the EvaluationTeam understood the perspective of the various Board membersand the underlying rationale for their differing views. It was alsocritical that the voices of young people at different levels of theorganisation were heard. It was important, too, that feedback wascollected from other stakeholder groups, such as project bene-ficiaries and former members and supporters of the youth

ARTICLE IN PRESS

D. McDonald / Evaluation and Program Planning 31 (2008) 259–265 263

movement. However, encouraging and valuing the contributionsof a wide range of stakeholders does not mean that an evaluatormust accept all of the opinions that are expressed equally. Valuejudgements need to be made based on the combined, accumu-lated evidence.

Furthermore, Markiewicz recognises that in order to beeffective, evaluators need an ability to handle the pressuresassociated with differing stakeholder perspectives and agenda.This is true, but they also need to make decisions, and not all thesedecisions will satisfy every stakeholder. In the case of the youthmovement evaluation, analysis, reflection and further discussionof the information provided by various respondents helped theEvaluation Team to build a more holistic picture of the currentreality, recognise the challenges being faced and create a possibleway forward. Beyond this, the evaluation process includedopportunities for the main stakeholder groups to hear oneanother’s perspective and to be engaged jointly in discussionsabout the organisation’s future.

But in the end, it was not possible or appropriate for me, as theevaluator, to keep control of the dynamic that grew from thevarious discussions that followed the release of my report. Atsome point, I had to let go and allow those who had a stake in theevaluation, and its outcome, to take responsibility for acting onthe findings, despite the fact that some people might disagreewith the new direction being pursued.

6.3. Principle No. 3: assessment of stakeholder positions and

planning the evaluation

The third principle proposed by Markiewicz is that anevaluator should identify the agenda of different stakeholders asearly as possible and facilitate the formation of an evaluationsteering committee or advisory group to deal with a pluralism ofviews. She also supports the idea of an evaluator sharing his/herexpertise with stakeholders, where possible involving them in theplanning process.

The benefits of establishing an advisory group for theevaluation and of engaging various stakeholders in the evaluationprocess are readily understood (Owen, 2006). However, myexperience in this current review highlighted some of thedifficulties associated with implementing such forums andengaging stakeholder involvement in evaluation planning andimplementation.

The youth movement evaluation provided a range of possibi-lities for involving the two main stakeholder groups (i.e. membersof the funding body Board and youth leaders). An Advisory Groupcomprising representatives from both parties was establishedearly on. Unfortunately this group was not easy to maintain sincethe participants lived in different parts of Australia and metinfrequently, due to busy schedules. It was also apparent that thepower differential between the two groups made it difficult forthe youth and Board representatives to gain some appreciation ofone another’s viewpoints.

Nevertheless, the involvement of the two national youthleaders in the Evaluation Team was most beneficial. This helpedto engage the rank and file youth in both the evaluation processand the subsequent developmental phase. It also provided anopportunity for these two young people to hear, first hand,the major concerns that faced their State-based counterpartswhen trying to implement their projects. Importantly, thecombination of external observations and internal insights led toa richer analysis and more focused and realistic strategies forimprovement.

Furthermore, this strategy allowed trust to build between theexternal evaluators and the youth leaders. It led to frank

discussions about the critical issues that were identified in thereview and to constructive debate from which some harddecisions about the future were taken. Perhaps most significantwas the fact that the participation of the two national youthleaders in the Evaluation Team placed them in a position ofstrength, rather than to them feeling that they were mere‘subjects’ of investigation, and the strong relationship that grewwithin the Evaluation Team led to an ongoing partnershipbetween the evaluators and youth leaders in working togetherto find an appropriate way to implement the recommendations.

However, it was virtually impossible to involve the full Boardin anything more than the initial planning meetings and thediscussions about the review findings and associated recommen-dations. This was partly due to the fact that these people led verybusy lives. It was also influenced by the fact that the Board lackeda clear direction and unity of purpose. What I did manage to dowas to participate in follow-up discussions with some individualBoard members about how they might lend positive support tothe new direction proposed by the youth leaders based onconsultation with their members. Some might say this smacksof undue bias. But I saw it as a necessary contribution and wouldwillingly have had similar discussions with the other Boardmembers had the opportunity been forthcoming.

6.4. Principle No. 4: the evaluator as an active player within the

stakeholder community

Next, Markiewicz purports that in order to act as a negotiator,the evaluator needs stakeholder acceptance. This, she believes,can be gained by actively participating in various stakeholderforums and by demonstrating skills, knowledge and personalattributes such as flexibility and responsiveness. She warns thatevaluators need to take care not to become too close tostakeholders and be co-opted by them, but rather stressed theneed for objectivity, as well as the avoidance of bias and misuse ofevaluative information. Therefore, in her view:

The evaluator needs to forge a position as an active playerwithin these forums, and assuage concerns that such involve-ment is a threat to objectivity and neutrality.

The main challenge that I faced in this regard was to gaincredibility in the eyes of all of the Board members since they metso infrequently. I endeavoured to manage this by maintainingfairly close contact with the Board representative who coordi-nated the Evaluation Advisory Group. My strategy was to provideher with regular feedback which I hoped would be disseminatedto the other Board members. In retrospect, this approach was notalways completely effective and I can see that a more intensiveeffort was required on my part to discuss the findings with her,explore the possible implications for change as they emerged andencourage her to share this information with other members ofthe Board.

In my relationship with the national youth leaders whobecome members of the Evaluation Team, I was ever consciousof the need to listen carefully to what they had to say. At the sametime I worked hard at trying to maintain my objectivity by seekinginput from a range of sources to inform my assessment of thesituation and to identify the most suitable strategies for change.The co-option of my evaluation colleague who had no priorinvolvement in the youth movement also provided a good checkagainst possible bias.

Where possible I utilised opportunities to be actively involvedwith various stakeholder groups by attending meetings andforums with young people and seeking feedback from formermembers of the youth movement via individual and group

ARTICLE IN PRESS

D. McDonald / Evaluation and Program Planning 31 (2008) 259–265264

discussions. This approach helped me to gain legitimacy in theeyes of most stakeholders, except perhaps, in the final analysis,with a minority of Board members who were disenchanted withthe new organisational direction and the changed relationshipbetween the youth leaders and the funding body. In some ways Isee this as the price an evaluator has to pay for weighing up theinformation gathered and making judgements based on theevidence obtained, to inform decision making. In making suchjudgements the evaluator runs the risk of disaffiliating anystakeholders who are unable to accept the professional advicegiven. However, the alternative is to endeavour to retaincredibility at the expense of truth.

6.5. Principle No. 5: skills of the evaluator as negotiator responding

to conflict

The penultimate principle proposed by Markiewicz relates tothe role of the evaluator in facilitating multiple stakeholders todiscuss and resolve their conflicting viewpoints. This, she says,requires well-developed personal skills including communication,empathy and assertiveness, and an ability to guide people todiscuss their differences openly, respect one another’s opinionsand come to productive solutions.

I agree that all these competencies are essential, but in myexperience, even for an experienced evaluator with all these skills,fulfilling this role effectively is more easily said than done! Asindicated above, in order to have any chance of achieving this, theevaluator needs to utilise all available opportunities to bringstakeholders together. In the case of the youth movementevaluation this was complicated by a number of factors: inparticular, distance and a long history of un-productive relation-ships between younger and older Board members, and by a lack ofclear direction compounded by disagreements between Boardmembers themselves.

In reality what was required was a willingness to seek out,listen to and respect the different perspectives raised by thevarious stakeholder groups, accompanied by an ability to conveythe chief concerns expressed by different parties to one another.On the rare occasions when the key stakeholders met, my role alsoincluded the capacity to facilitate, in order that the variousparticipants could express their views freely to one another and toattempt to reach some common understanding about the nextstep.

6.6. Principle No. 6: skill development in managing conflict with

multiple stakeholders

Markiewicz recognises that evaluators do not necessarily have,and may need to develop, the personal skills required for effectivenegotiation. She also considers that there would be value inevaluators supporting one another by sharing their experiences indealing with conflicting stakeholder views.

I concur with these views. My own feeling is that evaluatorscan easily become caught up in the technical and methodologicalissues associated with data collection and pay insufficientattention to the importance of social interaction with stake-holders. How often, for example, is the necessity for this discussedwith those who commission an evaluation or stipulated as a lineitem in an evaluation plan or budget? Demanding timeframes tocomplete evaluation projects with limited funds often necessitatethat an evaluator focuses on producing the findings, rather thanon taking up opportunities as they present themselves to interactwith stakeholders.

In the particular evaluation discussed, the fact that I had apersonal background of working in this youth movement and a

commitment to providing a service, over and above what I wasbeing paid to do, meant that I was inclined to give extra time andenergy to my role. I made myself available to liaise withstakeholders by email and phone and attended the occasionalmeeting, in addition to the structured evaluative activities thatwere included in the plan. On reflection, I consider these lessformal interactions critical in helping me develop my under-standing of the key issues and how these could be addressed mostappropriately. But in reality, this is not always possible.

The involvement of my evaluation colleague on the EvaluationTeam provided me with a ‘built-in sounding board’ to air anyconcerns that I had in dealing with different stakeholders.Likewise, I provided the same support for him. This helped usboth to reflect on how we should approach further interactionsand meetings. However, evaluators do not often have the luxuryafforded by joint work. In these situations some form of regularpeer review would be advantageous in providing both immediatesupport and longer-term skills development to enhance thenegotiation skills of evaluation practitioners.

7. Conclusion

Evaluators play a significant role in facilitating interactionbetween different stakeholder groups. This includes the use ofhigh-level negotiation skills to assist those engaged in theevaluation process to understand and come to terms with diversepoints of view. In the ideal world the evaluator would act as amediator, encouraging multiple stakeholders to find the middleground and to cooperate with one another with respect to theevaluand that is under review.

The importance of this role is generally not understood bythose who commission an evaluation project, and evaluatorsare not necessarily equipped with effective negotiation skills.Markiewicz has identified some strategic principles needing to beheeded by practitioners when planning an evaluation, in order toachieve stakeholder consensus regarding evaluation methodologyand outcomes. The emergent theory proposed by these principleshas been tested within the context of a recent evaluation in whichstakeholder conflict was evident.

Although broadly in agreement with the need for an evaluatorto assist multiple stakeholders to appreciate each other’s differentperspectives and reach a common view about the most appro-priate way forward, this current investigation has found thatconsensus is not always possible or desirable. There are timeswhen the evaluation process leads to quite the opposite, withcritical differences of opinion coming to ‘a head’.

What is required of an evaluator at such times is as follows:

�

An ability to introduce proactive strategies thereby ensuringthat all stakeholders (particularly those in less powerfulpositions) can express their views openly; � The courage to make clear judgements, backed by solid

evidence, with respect to the evaluand, even though theconclusions drawn might not be to the liking of somestakeholders and might lead to hard decisions being made;

� The capacity to facilitate ‘difficult’ discussions between the

various stakeholder groups about the findings and how toimplement the recommendations most effectively; and

� The willingness to respond to stakeholder requests concerning

evaluation follow-up, together with the astuteness to standback at an appropriate point and not try to control the post-evaluation stakeholder dynamic.

These points extend the proposed theory of negotiation, andground them in reality. The desired role of an evaluator as

ARTICLE IN PRESS

D. McDonald / Evaluation and Program Planning 31 (2008) 259–265 265

mediator and negotiator requires a particular skill set whichincludes communication, empathy and assertiveness, and anability to encourage stakeholders to listen to, and respect, oneanother and to make joint decisions. While it is not likely that anyevaluator would savour the opportunity to use an evaluationprocess to unearth conflict, the political context in which anevaluation takes place means that this is likely to happen at somepoint. When this occurs the evaluator needs the confidence andskills to support stakeholders to deal with the situation appro-priately in order to achieve the best possible outcome withrespect to the evaluation task.

8. Post script

Throughout this article I have referred to ‘the stakeholders’ asif I were not one of them. However, the significance for me of myown previous involvement in this particular youth movementmotivated me to bring a passion and a commitment to theevaluation role that meant that I was more than a dispassionate,uninvolved outsider.

As noted earlier in this paper, as a young person involved in thismovement, I learned the value of self-reflection, experientiallearning and social action for change. That has influenced my adultlife and I would like the youth of today to discover somethingsimilar within their own lives. For that reason, a basic thrust in myintervention as a contracted evaluator has been to empower theyouth of today, to help them find their way ahead by identifying ashared social vision and working out ways to implement it.

As an evaluator I was aware of certain differences of opinionbetween older and younger Board members, and amongst adultBoard members themselves. I endeavoured to facilitate dialoguewhere this was possible but, due to inadequate opportunities forface-to-face sharing and debate, I realised that this evaluationprocess could not resolve the central issue successfully without itcoming to ‘a head’. This was namely, whether the role of adultBoard members should be one of guidance or of support for theyouth leaders of today.

My own experience, corroborated by the analysis of feedbackobtained through the evaluation led to recommendations that theBoard support the youth in their search for a way ahead. Whileclear in my own mind with regard to the validity of this optionand what it meant in terms of my involvement with various

stakeholders, especially the youth leaders, I also recognised thatthis is not a black and white option. Guidance can drift intosupport and vice versa.

In that sense, I recognise the goodwill and efforts of allinvolved in this evaluation project. Each individual has been animportant part of a process aimed at finding a way forward fromwhat appeared to be a seemingly hopeless situation. One year onfrom the social vision workshop, feedback from the youth leadersindicates that this process has been positive. It has helped them toarticulate their direction, organise nationally and begin to takesteps to put their vision into practice. They are now confident intheir capacity to be part of the movement to effect social changein favour of a more just and caring society. This has been mirroredby some positive changes on the Board which is now in theprocess of confirming its own raison d’etre.

References

Brinkerhoff, R. O. (2003). The success case method: Find out quickly what’s workingand what’s not. San Francisco, CA: Berrett-Koehler.

Cooperrider, D. L. (1995). Introduction to appreciative inquiry. In W. French,& C. Bell (Eds.), Organizational development (5th ed). Englewood Cliffs, NJ:Prentice-Hall.

Cousins, J. B., & Whitmore, E. (1998). Framing participatory evaluation. NewDirections for Evaluation, 80, 5–23.

Fetterman, D. M. (1996). Empowerment evaluation: An introduction to theory andpractice. In D. M. Fetterman, S. J. Kaftarian, & A. Wandersman (Eds.),Empowerment evaluation: Knowledge and tools for self-assessment and account-ability (pp. 3–46). Thousand Oaks, CA: Sage Publications.

Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation evaluation. Newbury Park, CA:Sage.

Kemmis, S. (1985). The action research planner. Geelong: Deakin UniversityPress.

Markiewicz, A. (2005). A balancing act: Resolving multiple stakeholder interests inprogram evaluation. Evaluation Journal of Australasia, 4(1–2), 13–21.

Owen, J. M. (2006). Program evaluation: Forms and approaches (3rd ed.). StLeonards, NSW: Allen & Unwin.

Patton, M. Q. (1996). A world larger than formative and summative. EvaluationPractice, 17(2), 131–144.

Patton, M. Q. (1997). Utilization-focused evaluation (3rd ed). Thousand Oaks, CA:Sage.

Preskill, H. (1996). From evaluation to evaluative inquiry for organizationallearning. Paper presented at the annual conference of the American EvaluationAssociation, Atlanta, GA.

Stake, R. A. (1983). Stakeholder influence in the evaluation of cities-in-schools. NewDirections for Program Evaluation, 17, 15–30.

The Age (2005). 27th August, Fairfax, Melbourne.Worthen, B. (1996). Mechanisms for making the program evaluation standards

meaningful to graduate education. Paper presented at the annual meeting of theAmerican Evaluation Association, November 1996, Atlanta, GA.