stories for change: a new model of evaluation for ... · stories for change: a new model of...
TRANSCRIPT
STORIES FOR CHANGE:
A NEW MODEL OF EVALUATION FOR AGRICULTURAL
EXTENSION PROJECTS IN AUSTRALIA
Jessica Jane Dart
Submitted in total fulfilment of the requirements of the degree of Doctor of
Philosophy
July 2000
Institute of Land and Food Resources, the University of Melbourne
i
ABSTRACT
A model of evaluation was investigated and developed to help meet the evaluation needs of
agricultural extension agencies in Australia. The model was based on Davies’ (1996) Most
Significant Change (MSC) model; a participatory evaluation approach that involves the generation
of stories and their interpretation by project stakeholders. This model had not previously been
implemented in Australia.
Over the last decade agricultural extension institutions have been undergoing structural and
philosophical changes. These changes have brought about an increased demand for evaluation that
can deal with participatory, group-based extension, and can identify and value diverse impacts.
There is also an increased demand for internal evaluation that can deal with multiple audiences,
promote organisational learning and communication, as well as meet needs for accountability. To
develop an evaluation approach capable of addressing these needs, a novel evaluation model was
selected, modified, implemented and empirically tested.
A case evaluation of the MSC approach was conducted across a large dairy extension project
operating in four regions of Victoria. After implementing the evaluation, a meta-evaluation was
conducted to determine the extent to which the MSC process contributed to project improvements,
met needs and represented ‘good’ evaluation. It was found that it met the articulated needs of the
stakeholders of the case project and combined well with their existing evaluation practices. This
was ultimately verified by their commitment to continue with the MSC process after the 12 month
trial period. Project staff perceived that the MSC process helped stakeholders to make sense of
project impact and to understand each other’s values, increased staff morale and helped draw staff,
farmers and other collaborators more centrally into the evaluation process. However, there were
regional differences in the perception of benefits achieved. An examination of these differences
provided information about the sort of project contexts in which benefits are likely to be gained
from the MSC model.
In considering the evaluation needs of extension more broadly, the MSC model was found to offer
processes capable of accommodating the new genre of participatory extension projects. Where
projects are likely to have diverse outcomes that are not all pre-determined, the MSC model could
ii
play an important role in searching for impact. The MSC model can also accommodate a wide
range of participants in the evaluation process. However, the model does not provide indication of
the spread of adoption of technologies across the farming population, so does not fully meet the
demand by purchasers for greater accountability. It was designed to capture ‘remarkable events’
rather than the average experience of farmers. While the MSC model satisfied some of the premises
for good evaluation, it was found to have inherent biases and weaknesses, including an inability to
capture unintended outcomes unless these are deemed ‘significant’ by those involved in the process.
It was suggested that the MSC model should be strategically combined with other evaluation
approaches selected to offset its inherent bias, and to meet the full range of project needs for
evaluation.
The MSC model was found to offer an important contribution to the ‘basket of choices’ for
extension evaluation, but like other evaluation models, it does not provide an overarching solution
to the myriad of evaluation demands being felt by extension agencies in Australia. It is suggested
that those evaluating extension projects should first come to understand the evaluation needs of
their project stakeholders, and then develop a carefully crafted collection of evaluation approaches
to meet these requirements. For some projects, the MSC model may provide an important new
component of their evaluation strategy. In particular, large, process-orientated projects, with high
levels of farmer involvement, could gain much from this model.
iii
This is to certify that
(i) the thesis comprises only my original work except where indicated in the preface,
(ii) due acknowledgment has been made in the text to all other material used,
(iii) the thesis is less than 100,00 words in length, exclusive of tables, maps, bibliographies and
appendices.
Jessica Jane Dart
iv
PREVIOUS PUBLICATION OF MATERIAL FROM THESIS
Earlier versions of the review of evaluation in agricultural extension presented in Chapter 4 were
published in the following documents and conference papers:
Dart, J.J., Petheram R.J., & Straw, W. (1998), Review of Evaluation in Agricultural
Extension, Rural Industries and Research Development Corporation, publication number
98/136, Canberra.
Dart, J.J., Bardsley B., & Petheram R.J., (1997), ‘Forms and approaches of evaluation in
agricultural extension’, Proceedings of the Australasian Evaluation Society (AES)
International Conference, Adelaide, AES, 210-225.
Dart, J.J., Petheram R.J., & Straw, W. (1997), ‘Forms of evaluation – for agricultural
extension’, Proceedings of the 2nd Australasia Pacific Extension Conference (APEN),
Albury, NSW, APEN, 2, 408-471.
Earlier versions of the description concerning the implementation of the MSC model, presented in
Chapter 7, were published in the following papers:
Dart, J.J. with Dysdale G., Cole, D., Saddington, M. (in press), ‘The most significant
change approach for monitoring an Australian extension project’, PLA Notes: Participatory
Learning and Action, International Institute for Environment and Development, London.
Dart, J.J. (1999), ‘A Story Approach for monitoring change in an agricultural extension
project’, proceedings of the Association for Qualitative Research (AQR), International
Conference, Melbourne, AQR, world wide web:
http://www.latrobe.edu.au/www/aqr/offer/papers/JDart.htm
Dart, J.J. (1999), ‘The tale behind the performance story approach’, Evaluation News and
Comment, 8, no.1, 12-13.
v
ACKNOWLEDGEMENTS
Firstly, I would like to acknowledge all the effort and help that has been provided by my
supervisory panel. A big ‘thank you’ to Dr Patricia Rogers for stepping in during the final year and
providing excellent guidance and expertise on program evaluation and also for encouragement and
support. Many thanks to Dr John Petheram, who has been involved with this research from the very
beginning, and who has been a great help. I would also like to acknowledge important contributions
from Dr Ruth Beilin and Dr Barrie Bardsely.
I received valuable and important help from many staff of the Victorian Department of Natural
Resources and Environment. Thanks to Bron McDonald for being my mentor throughout the three
years, and for sharing her expansive networks with me. I am also very grateful to all the staff and
committees members of the Target 10 Dairy Extension Project, who took the risk of participating in
this somewhat unconventional research. I would especially like to thank John Boomsma, Mark
Walton, David Cole and Geoff Drysdale for their help and support in implementing the Most
Significant Change evaluation process. I am also grateful to all the storytellers for contributing their
stories to the evaluation process.
I would like to acknowledge several people for sharing with me their wisdom and advice
concerning program evaluation. Thanks to Rick Davies for his pioneering work with the MSC
model, and for insightful conversations. The staff from the Centre for Program Evaluation at the
University of Melbourne have also been a great help, especially Dr Gerald Elsworth. Thanks also
to Jerome Winston from RMIT University for his inspirational ideas concerning program
evaluation.
Finally, I am thankful for the help of my friends and family for being patient and accepting. Thanks
to Scott for help with the technical aspects of the Internet questionnaire, and thanks to Carolyn for
putting up with me and sharing an office space through these three years. Thanks to Mick Maguire
for helping me with the production of the booklet Target 10 Evaluation Stories.
vi
ACRONYMS
CBA Cost benefit analysis
CCBD Christian Commission for Development in Bangladesh
CEC Central Executive Committee
CSIRO Commonwealth Scientific and Industrial Research Organisation
DPI Department of Primary Industries (Queensland)
DRDC Dairy Research and Development Corporation
FGE Fourth Generation Evaluation (Guba and Lincoln, 1989)
IIED International Institute for Environment and Development
KASA Knowledge, attitudes, skills and aspirations
LWRRDC Land and Water Resources Research and Development Corporation
M&E Monitoring and evaluation
MIA Murrumbidgee Irrigation Area
MSC Most Significant Change model (Davies, 1996)
NPM New public management
NRE Natural Resources and Environment
NSCP National Soil Conservation Program
NUD.IST Non-numerical Unstructured Data Indexing Searching and Theorizing
OECD Organisation for Economic Development and Cooperation
ORID Objective, Reflective, Interpretive, Decisional
PAR Participatory Action Research
PEMS Pig Enterprise Management Services
PM&E Participatory Monitoring and Evaluation
PRA Participatory Rural Appraisal
PPRDP People’s Participatory Rural Development Program
RIRC Rural Industry Research Corporations
RIRDC Rural Industry Research and Development Corporation
RRA Rapid Rural Appraisal
UFE Utilization Focused Evaluation (Patton, 1997)
VFSG Variable Farming Systems Group
vii
CONTENTS
CHAPTER 1: INTRODUCTION
1.1 Problem Statement 1
1.2 Aim 2
1.3 Research Objectives 3
1.4 Research Approach 3
1.5 Road Map for Thesis 5
1.6 Overview of Chapters 7
CHAPTER 2: A CONCEPTUAL FRAMEWORK FOR GOOD PROGRAM EVALUATION
2.1 Introduction 10
2.2 What is Program Evaluation? 11
2.3 What is ‘Good’ Program Evaluation? 18
2.4 Six Normative Approaches to Evaluation 19
2.5 Bricolage and Mixed-Approach Designs 31
2.6 A Normative Framework for Evaluation 34
2.7 Conclusions 37
CHAPTER 3: EVALUATION NEEDS IN AGRICULTURAL EXTENSION
3.1 Introduction 38
3.2 A Changing World 39
3.3 Changing Philosophy of Agricultural Extension 40
3.4 Organisational Change and Extension Evaluation 52
3.5 A Case Study of the Agriculture Division in the Victorian Department of Natural Resources
and Environment 57
3.6 Conclusion 65
CHAPTER 4: REVIEW AND DESCRIPTION OF CURRENT PRACTICE OF EVALUATION IN
AGRICULTURAL EXTENSION
4.1 Introduction 67
4.2 Background to the Review 68
viii
4.3 Approach Taken 70
4.4 Description of the Evaluation Studies 73
4.5 Case Studies of Different Forms of Evaluation in Agricultural Extension 75
4.6 Summary of the Evaluation Samples 97
4.7 Examining the Evaluation Studies Against the Meta-evaluation Questions 99
4.8 Conclusions 105
CHAPTER 5: CONTENDING MODELS OF PROGRAM EVALUATION FOR AGRICULTURAL
EXTENSION
5.1 Introduction 107
5.2 Criteria for Choosing a Model to Pilot 108
5.3 Contending Models of Program Evaluation to Test 110
5.4 Utilization Focused Evaluation (UFE) 111
5.5 Realistic Evaluation 115
5.6 Fourth Generation Evaluation (FGE) 122
5.7 Most Significant Change (MSC) Model 127
5.8 Selection of Evaluation Model to be Tested 133
5.9 Considering all Four Models 138
5.10 Conclusions 139
CHAPTER 6: METHODOLOGY FOR THE CASE EVALUATION
6.1 Introduction 140
6.2 The Overarching Research Design 141
6.3 Rationale for Choice of Project 143
6.4 Participatory Action Research Component 144
6.5 Meta-evaluation 148
6.6 Ethical Considerations 160
CHAPTER 7: CASE EVALUATION OF THE MSC MODEL AS IMPLEMENTED ACROSS THE TARGET 10
PROJECT
7.1 Introduction 161
7.2 Description of the Project 162
7.3 Implementation of the Approach 168
7.4 Analysis of Outputs 181
7.5 Problems During the Implementation 198
7.6 Conclusions Concerning the Implementation of the MSC Process 200
ix
CHAPTER 8: THE IMPACT OF THE MSC PROCESS ON THE TARGET 10 DAIRY EXTENSION PROJECT
8.1 Introduction 201
8.2 Method of Testing the CMO Configurations for the MSC Process 202
8.3 Different Contexts 206
8.4 Findings 209
8.5 Summary of Testing Model 238
8.6 Impact of the MSC Process on the Project 239
8.7 Conclusions 243
CHAPTER 9: THE MOST SIGNIFICANT CHANGE MODEL FOR EVALUATION IN AUSTRALIAN
AGRICULTURAL EXTENSION
9.1 Introduction 244
9.2 Applying the Meta-Evaluation Questions 245
9.3 Is the Evaluation Relevant to the Context and Purpose in Hand? 246
9.4 To What Extent Can the MSC Evaluation Contribute to Projects Better Able to Meet Needs? 250
9.5 Is the Evaluation Model Guided by Program Theory? 251
9.6 Does the Evaluation Model Make an Attempt to Account for Different Program Outcomes? 254
9.7 Are the Evaluation Processes Socially and Politically Just? 255
9.8 Is there an appropriate ratio of Costs and Benefits for the Project from the Process
(and Outcomes) of the Evaluation? 257
9.9 To what Extent are the Design and Data Collection/ Analysis of the Evaluation Valid? 257
9.10 Does the Evaluation Attempt to Judge the Merit and Worth of Projects? 260
9.11 Summary: Extent to which the MSC Model adds to the ‘Basket of Choices’ for Extension
in Australia 262
9.12 Improving the Contribution of the MSC Model 262
CHAPTER 10: CONCLUSIONS 265
REFERENCES 269
APPENDICES 284
x
Figures
Figure 1’Road Map’ of layout of thesis and research.....................................................................................................6
Figure 2 Outcomes and intermediate outcomes of six different normative approaches to evaluation..........................30
Figure 3 The process of modifying an evaluation bricolage for a mega-project..........................................................33
Figure 4 Example of an evaluation bricolage for an hypothetical program, showing three domains ..........................73
Figure 5 Distribution of the evaluation studies according to Owen’s five forms .........................................................74
Figure 6 Graph of net present value at different adoption rates....................................................................................79
Figure 7 Distribution of evaluation studies according to the level of investigation of Bennett’s Hierarchy ..............102
Figure 8 Level of participation in evaluation by farmer clients in the studies............................................................104
Figure 9 The realist evaluation cycle..........................................................................................................................116
Figure 10 Diagram to represent monthly flow of stories in MSC model for a hypothetical project ..........................129
Figure 11 A systems diagram to illustrate case-study design including concurrent action research and meta-
evaluation components .................................................................................................................................142
Figure 12 An evaluation theory model showing how the MSC process was thought to bring about
improvements in the project .........................................................................................................................155
Figure 13 The service delivery model of the Target 10 Dairy Extension Project.......................................................163
Figure 14 Items contained in the proforma for collection of stories...........................................................................170
Figure 15 Proposed main steps for implementation of MSC process with Target 10 Project ....................................176
xi
Tables
Table 1 Owen’s five forms of program evaluation........................................................................................... 72
Table 2 Bennett’s Hierarchy............................................................................................................................. 85
Table 3 Summary of main characteristics of the sample of evaluation studies ................................................ 97
Table 4 Results of the Delphi process used to establish domains of change .................................................. 172
Table 5 Questions asked to help describe the stories...................................................................................... 182
Table 6 Number of stories collected each round ............................................................................................ 183
Table 7 Distribution of stories per region....................................................................................................... 183
Table 8 Distribution of the story tellers.......................................................................................................... 184
Table 9 Project ‘programs’ mentioned in the story ........................................................................................ 184
Table 10 Description of themes that have been the subject of multiple stories.............................................. 185
Table 11 Level of Bennett’s Hierarchy .......................................................................................................... 186
Table 12 Grouping of stories by theme using Bennett’s Hierarchy................................................................ 187
Table 13 Characteristics of stories selected by the Central Executive Committee (n=24) ............................. 190
Table 14 Conjectured context, mechanism, outcome configurations for the MSC process ........................... 204
Table 15 Staff attending their regional committee meetings during the trial ................................................. 206
Table 16 Attendance at regional committee meetings during trial ................................................................. 206
Table 17 Number of stories contributed by farmers and other non-staff committee members per region ..... 206
Table 18 Extent to which meetings in the different regions provided a context conducive to discussion
and exchange of views.......................................................................................................................... 208
Table 19 Respondents’ descriptions of the stories.......................................................................................... 210
Table 20 Number of stories read in booklet against number of story sessions attended ................................ 211
Table 21 Extent to which respondents remembered the stories...................................................................... 212
Table 22 Staff perception of the extent to which the MSC process helped farmer committees to ‘steer’
the project against meeting context....................................................................................................... 214
Table 23 Staff perception of morale boosting effect of the MSC process against meeting context ............... 218
Table 24 Staff perceptions of the extent to which the MSC process affected staff understanding of project
impact against meeting context............................................................................................................. 222
Table 25 Extent to which the MSC process affected staff understanding of outcomes valued by the CEC
against meeting context......................................................................................................................... 224
Table 26 Extent to which staff felt that the MSC process had affected their understanding of the outcomes
valued by the purchasers meeting context ............................................................................................ 225
Table 27 Staff perception of the extent to which the MSC process affected the creation of a more shared
vision against meeting context.............................................................................................................. 228
Table 28 Extent to which staff used the stories .............................................................................................. 233
xii
Table 29 Extent to which staff used the stories against meeting context ....................................................... 233
Text boxes
Text box 1 Segment of discussion at a regional committee meeting during story selection ......................................179
Text box 2 Examples of comments from the purchasers concerning the stories ........................................................191
Text box 3 Story example 1........................................................................................................................................193
Text box 4 Story example 2........................................................................................................................................194
Text box 5 Story example 3........................................................................................................................................195
Text box 6 Story example 4........................................................................................................................................196
Text box 7 Extract of discussion between two extension staff from the South West Region, taken from a
focus group discussion between regional champions.........................................................................................218
Text box 8 Comments concerning the competitive aspect of the MSC process.........................................................219
Text box 9 Comments associated with the benefits of discussing the stories.............................................................227
Text box 10 Comments concerning the value of the MSC process for creating dialogue with purchasers................227
Text box 11 Comments associated with the value of stories for evaluation reporting ...............................................235
Text box 12 Comments referring to how stories have affected change at a strategic level ........................................236
Text box 13 Informant views of the extent to which the farmer committee members ‘represent’ the wider
dairy farming community .................................................................................................................................256
1
CHAPTER 1: INTRODUCTION
1.1 Problem Statement
In Australia agricultural extension is undergoing philosophical change. This change is influencing
how agricultural extension is conceptualised, how it is conducted, and consequently how it should
be evaluated. The science-push paradigm, that has conventionally underpinned extension, is being
partially replaced by new approaches influenced by the emergent farmer-first paradigm. This has
resulted in a movement towards participatory, group-based and process-oriented forms of extension.
These new models of extension require models of evaluation that can deal with diversity in
implementation, participation of stakeholders within evaluation, and the need to provide
accountability for projects that begin with loosely defined outcomes.
The 1990s have also seen considerable structural and policy reform across the organisations that
deliver extension. For example, as a result of restructuring and increased collaboration, large
projects have emerged which are bigger and more complex in structure than in the past. A second
example of change is that in most states of Australia, the purchaser-provider model has been
introduced for the provision of publicly-funded extension. In simplistic terms, the purchaser is
responsible for investing resources wisely, and demonstrating the value of the investment. The
provider’s role is to provide the service contracted and to assist with demonstrating the value of the
service. With the purchaser-provider split, accountability and communication have become crucial.
These are but two of a plethora of recent changes that pose considerable challenges for the
evaluation of extension projects.
Since the introduction of these changes in agricultural extension, the demand for evaluation appears
to have increased both in terms of the diversity of approaches sought, and the number of
stakeholders requesting evaluation information. Extension project managers are being asked to
provide evidence as to whether both tangible and intangible outcomes have been achieved, to
describe the expected and unexpected impacts caused by interventions and to determine the extent
to which the farmer-clients perceive that their needs have been addressed. These managers are
required to produce plans of how they intend to conduct evaluation, yet there are no clear guidelines
2
on how this should be done. At the same time, there is a lack of knowledge among extension agents
of evaluation approaches able to address these new demands. An increased demand for outcome-
focused evaluation is particularly problematic for projects that comprise participatory group-based
models of extension. These projects tend to produce diverse, context-specific outcomes, which are
difficult to aggregate in order to provide information to evaluate the project as a whole.
The literature of agricultural extension reveals very little information on approaches that can meet
these needs (Dart et al., 1998). There is a serious gap in the knowledge required to effectively
evaluate these large, complex projects. Yet, there is an extensive literature concerning evaluation in
the fields of education, health promotion and international aid and development. There appears to
be considerable scope for transfer of ideas and learning about evaluation between the different
disciplines. Madaus et al., (1983) point out that there is a need to educate evaluators in the
availability of new techniques; a need to try out and report the results of using these new
techniques; and to develop additional methods. New approaches need to be developed, or adapted
from other disciplines, to meet the evaluation needs of agricultural extension. The emphasis should
be on making the approaches fit the needs of extension agents and their organisations, and the
farmers and public whose needs they aim to meet.
1.2 Aim
The aim of this thesis is to develop a model of evaluation that has the potential, in combination with
other approaches, to address the current challenges associated with evaluating the new genre of
agricultural extension projects in Australia.
3
1.3 Research Objectives
The research objectives are to:
• examine the evaluation needs and current practice of program evaluation in agricultural
extension projects in Australia
• identify suitable models of evaluation for agricultural extension, that have not been
implemented in Australia previously
• select, implement, modify and evaluate one novel model of evaluation across an Australian
extension project
• develop an improved and context-specific model of evaluation that has potential to bring about
improved agricultural extension projects which better meet stakeholders needs
• provide a coherent strategy to combine this approach with others, in order to address the
challenges posed by changes occurring in extension organisations.
1.4 Research Approach
In order to develop or adapt evaluation approaches to meet the needs of agricultural extension, a
pragmatic stance was taken with regard to evaluation design. The current genre of large-scale
extension projects has various evaluation needs that will only be met through the application of a
range of complementary evaluation approaches and tools. The overall approach to evaluation design
advocated in this thesis can be described as ‘bricolage’; a collective term that has been used to
describe qualitative research (Denzin and Lincoln, 1998) and cultural studies (Nelson et al., 1992):
The bricoleur produces a bricolage, that is a pieced-together close-knit set of practices that provide
solutions to a problem in a concrete situation. The solution (bricolage) which is the results of the
bricoleur’s method is an emergent construction (Weinstein and Weinstein, 1991:161 cited by Denzin
and Lincoln, 1998: 3).
In this case the bricolage is the situation-specific solution to a whole range of evaluation needs and
requests that exist for one large project. The practices pieced together comprise different
approaches and methods of evaluation. In building this evaluation bricolage, therefore, it is
necessary to understand the evaluation needs. This understanding is best gained through an analysis
of the evaluation requirements of the specific organisation in conjunction with an analysis of their
4
existing evaluation practises.
In this thesis, evaluation needs were determined at two levels and both contributed to the selection
and modification of a novel evaluation model:
• evaluation needs and current evaluation practice were explored at a generic level for
agricultural extension projects in Australia
• the specific evaluation needs and current practice were identified for one large extension
project.
A novel model of evaluation was then selected on the grounds that, (a) it could help address the
evaluation needs and complement the existing portfolio of evaluation practice of the case project
and (b) it could address some of the challenges in evaluating large extension projects in general.
The empirical research consisted of implementing a single evaluation study that was to form one
part of the evaluation strategy of the Target 10 Dairy Extension Project. The selected model was
implemented and later was itself evaluated. It was evaluated for its ability to meet evaluation needs
at the project level, and to meet the challenges posed for extension evaluation at a national level.
Implementing, modifying and evaluating the model
In conducting the research for the case study implementation of a novel evaluation model, two main
research approaches were employed; Participatory Action Research (PAR) and a theory-guided
approach to meta-evaluation. Key differences between the two research components were the
degree of participation of project stakeholders and the purpose for which the information was
collected. The Target 10 Project stakeholders conducted the PAR to meet the needs of the project,
while I conducted and directed the meta-evaluation to meet the requirements of this doctoral thesis.
Participatory action research (PAR) pursues both action and research outcomes. Its purpose is to
bring about planned change in such a way that there is also a contribution to the understanding of
the system that is being changed (Cunningham, 1993). In this thesis the PAR component involved a
12-month trial of a specific model of evaluation. The trial was conducted with the staff and
committee members of the Target 10 Dairy Extension Project in Victoria. These project
stakeholders collected and interpreted the data, which formed part of the evaluation, while I
5
initiated the trial and acted as facilitator of the process. Staff and committee members modified the
MSC process as it was implemented to meet regional needs and local context. I collected data
concerning the effectiveness of the model during the implementation that contributed both to
immediate modifications and to the meta-evaluation.
Meta-evaluation is the evaluation of an evaluation (Scriven, 1991). Theory-guided meta-evaluation
is an approach to ‘evaluating evaluation’ which involves constructing causal models of how the
evaluation is understood to improve the project (Rogers, 1996). The meta-evaluation was inductive
and iterative in design and involved two interconnected parts: inductive theory building to
understand how the evaluation process was working; and data collection to determine the extent to
which the process was effective, and in which situations and why. This data consisted of field notes,
meeting notes, semi-structured interviews and an on-line questionnaire administered to project staff.
1.5 Road Map for Thesis
The research for this thesis was conducted in an iterative manner; as one set of data was collected
and interpreted, this informed the design of the subsequent phase of data collection. As a
consequence of the iterative design, several methods were used at different stages of the research
cycle. Figure 1 represents a ‘road map’ of the thesis in which the colour coding represents a
distinction between chapters that focus on the macro national level (purple), the micro project level
(blue), or both levels (pink).
6
Figure 1 ‘Road Map’ of layout of thesis and research
Evaluation of the MSC evaluation of the Target 10 extension project (Chapter 8)
What was the impact of the MSC evaluation on the project? How well did it meet the evaluation needs of the project?
Evaluation of the MSC approach for its potential to meet needs of agricultural extension projects nationally (Chapter 9)
How good was it? How well did it address the challenges identified?
Description of modification and implementation of approach across an extension project (Chapter 7)
What happened when the MSC approach was
implemented?
Method (Chapter 6)
How can the evaluation approach be tested?
Develop a normative framework for what entails good evaluation based on global literature (Chapter 2)
What is good program evaluation?
Review current evaluation practice at a national level to identify gaps (Chapter 4)
What extension evaluation is occurring nationally?
Determine the challenges particular to context of Australian Agricultural Extension (Chapter 3)
What are the evaluation needs of extension organisations at the national level?
AIM: Develop a model of evaluation that has potential to help address the current challenges associated with evaluating the new genre of agricultural extension projects in Australia
Choose a novel approach to evaluation - consider needs and gaps at the project and national level (Chapter 5)
Which approach to test?
Conclusions (Chapter 10)
What are the implications for the evaluation of agricultural extension programs in Australia?
7
1.6 Overview of Chapters
Chapter 2 addresses the question: What is good evaluation? I argue that no single normative model
for evaluation can encompass the diverse and multi-faceted range of tasks required of evaluation in
large projects. After presenting an overview of the different facets of program evaluation, a series of
premises for ‘good’ evaluation are presented to guide the development of a new model of
evaluation for agricultural extension. A key premise is that evaluation should lead to programs that
better meet needs. A further premise is that the best way of conducting an evaluation depends on the
purpose of the evaluation and the program context. From these premises, a series of meta-evaluation
questions are developed that will be used in the subsequent chapters to guide the development of a
new evaluation model for extension.
Chapter 3 addresses the question: What are the evaluation needs in this particular context? The
current context and challenges associated with the evaluation of agricultural extension projects in
Australia are examined. The chapter begins with a brief look at changes that are occurring at a
global level and how these changes are reflected in the practice of agricultural extension. A
description is provided of the development of extension models from the transfer of technology
paradigm to new approaches influenced by the farmer-first paradigm. An emphasis is placed on the
move towards more participatory, group-based and process-oriented forms of extension. I suggest
that new models of extension require new models of evaluation that can deal with diversity in
implementation, participation of stakeholders within the evaluation study and an ability to provide
accountability to projects with loosely defined outcomes at the onset. Following this, changes
occurring in organisations that deliver extension are examined. An overview is provided of the
impact of recent changes in public sector management within the Agriculture Division, the
Victorian Department of Natural Resources and Environment. An attempt is made to unravel the
various reforms and to describe the consequent challenges posed for the practice of project
evaluation in agricultural extension.
Chapter 4 addresses the question: What evaluation is already occurring? The findings of a review
of evaluation in agricultural extension in Australia are presented in which 50 evaluation documents
were analysed using Owen’s (1993) meta-model that is based on distinct forms or purposes of
evaluation. Nine case studies are presented that illustrate the range of forms and approaches in
practice. Where possible, the evaluation studies are examined against the meta-evaluation criteria
laid out in Chapter 2. I found that there is very little monitoring of projects, and a tendency to stick
8
to a handful of familiar methods and approaches. Most of the studies were limited to a summative
assessment of project impact, and there was a pre-dominance of ‘black-box’ approaches. I argue
that while there is a range of forms and approaches to evaluation in practice, the suite of approaches
that are currently used is inadequate to meet all challenges and demands for extension evaluation in
Australia. In particular, no evaluation study is found that facilitated on-going communication
between multiple project stakeholders.
Chapter 5 addresses the question: Which evaluation model to test? Four contemporary models of
program evaluation are examined in terms of their ability to address both the macro needs of
extension projects nationally and the micro needs of the case study project across which it was to be
tested. The models examined are Patton’s (1997) ‘Utilization Focused Evaluation’, Pawson and
Tilley’s (1997) ‘Realistic Evaluation’, Guba and Lincoln’s (1989) ‘Fourth Generation Evaluation’,
and Davies’ (1996) ‘Most Significant Change’ model. All four models are found to have much to
offer extension evaluation at a national level. Davies’ (1996) Most Significant Change (MSC)
model is selected for application, primarily as it appeared to complement the existing evaluation
practice of the Target 10 Project and meet the evaluation needs of the project. The MSC model
employs novel methodology based on the collection and selection of stories of significant change. I
suggest that the use of stories could offer an effective way of creating dialogue and ‘sensemaking’
between the multiple stakeholders of large extension projects, which was one of the evaluation
needs of the case study project.
Chapter 6 addresses the question How to test the evaluation model? The research approach and
methodology employed for the case study evaluation and meta-evaluation are described and
justified. The research approach is based on a combination of Participatory Action Research and
non-participatory theory testing.
Chapter 7 addresses the question: What happened when the MSC model was implemented? The case
study is presented, comprising a description of the Target 10 Project and the implementation of a
12-month trial of the MSC model. The problems and issues that arose while implementing the
model are discussed. The output of the process included 134 stories, and feedback from the review
of these stories during 15 meetings of project stakeholders.
Chapter 8 addresses the questions: What impact did the MSC model have on the Target 10 Project
and how well does it meet the evaluation needs? To answer these questions, the findings of a
9
theory-guided meta-evaluation of the MSC evaluation of the Target 10 Extension Project are
offered. Firstly, a model is presented of how the process was thought to bring about improvements
to the project. From this model nine conjectured outcomes are identified which included: increased
participation, increased knowledge, increased staff morale, and operational and strategic changes to
extension practice. Secondly, empirical evidence is used to test whether the outcomes stipulated in
the model had been achieved, in what contexts and why. Significant regional differences are found
in the extent to which respondents perceived the outcomes to have been achieved. I suggest that
these regional differences were associated, amongst other things, with the structure and nature of
participation in the meetings at which the stories were reviewed. Despite regional differences in
magnitude of success, the MSC process was able to meet many of the current evaluation needs of
the Target 10 stakeholders. The MSC process also appears to complement the project’s existing
evaluation practice.
Chapter 9 addresses the question: How good is the MSC model, and how well can it address the
challenges facing large extension projects? To answer this question the MSC model is examined
against the key meta-evaluation questions developed in Chapter 2. I found that the MSC model goes
a long way to meeting the evaluation needs at the national level. In terms of leading to projects
being better able to meet needs, this will be somewhat dependent on the individual project context. I
suggest that this model may be particularly appropriate for large extension projects with diversity in
outcomes, with a high degree of farmer participation in project planning and decision-making. On
further examination against the meta-evaluation questions, the model satisfies some of the premises
for good evaluation, but also has inherent biases and weaknesses. This emphasises the need to
combine the MSC model with complementary evaluation approaches, to help meet all evaluation
needs and also to offset the bias.
Chapter 10 addresses the question: What are the implications for the evaluation of agricultural
extension programs in Australia? The chapter provides a conclusion to the previous chapter and
concludes the overall thesis.
10
CHAPTER 2
A CONCEPTUAL FRAMEWORK FOR GOOD PROGRAM
EVALUATION
In this chapter I argue that no single normative model for program evaluation can encompass
the diverse and multi-faceted range of tasks that evaluation involves. An overview of the
different facets of program evaluation, followed by a series of premises for ‘good’ evaluation
are presented to guide the selection and development of a new model of evaluation for
agricultural extension. A key premise is that evaluation should lead to programs that better
meet stakeholder needs. A further premise is that the best way of conducting an evaluation
depends largely on the evaluation questions and the program context. I suggest that a
bricolage approach to evaluation be employed: after analysis of the program context,
evaluation needs and existing practice, the bricoleur picks and chooses from various
evaluation models and tools and adds a new component to the bricolage. A further step
suggested is that the new evaluation tools themselves should be piloted, evaluated and
modified if necessary.
2.1 Introduction
The development of any new model of program evaluation should be guided by current theory and
practice. To select appropriate theories and practice from the large body of knowledge known as
‘program evaluation’, an appreciation of the current status and developments in this field must be
gained. It is also essential to establish some conception of what ‘good’ evaluation entails.
The aim of this chapter is provide a conceptual framework for program evaluation that will guide
this thesis. In simple terms this chapter addresses the question: What is good program evaluation?
This question is addressed by firstly determining what is implied by ‘program evaluation’, then by
examining different perspectives on ‘good evaluation’. A major source of theory and ideas are the
11
journals and texts entitled ‘program evaluation’ (that mainly relate to the evaluation of education
and health related programs). A sub-set of literature that offers practical suggestions and novel
approaches, is evaluation associated with development studies. This includes evaluation of projects
that are delivered in low-income countries, but often funded by organisations from other countries.
In development studies, program evaluation is commonly referred to as either ‘impact evaluation’
or ‘monitoring and evaluation’.
I begin the chapter by presenting definitions of evaluation and describing the literature of program
evaluation. I trace the historical beginnings of program evaluation through to the state of great
diversity that characterises program evaluation today. In an attempt to make sense of this
burgeoning field, I offer a meta-model that accounts for six broad approaches to evaluation.
Through an examination of these different approaches, I highlight some of the fundamental issues
surrounding the discourse in the field. Following this, I present a theoretical framework for
characterising ‘good’ evaluation, in the form of a series of meta-evaluation questions. I use this
framework to review examples of evaluation practice in Australia (Chapter 4), to choose an
approach to apply (Chapter 5), and finally to evaluate the case study evaluation (Chapter 7).
2.2 What is Program Evaluation?
Program evaluation today is characterised by great diversity: from large-scale, long-term,
international comparative designs costing millions of dollars, to small-scale short evaluations of a
single component in a local agency. Worthen et al., (1997) suggested that over 60 different models
for the conduct of evaluation have been developed. Each model has built-in assumptions.
Evaluation literature is vast and fragmented and appears to be aimed mostly at evaluation theorists
rather than practitioners. There are at least 17 international journals on evaluation. The discipline
has its own societies, books, awards, conferences and standards.
There is no single recognised definition of program evaluation. Historically, the definitions of
evaluation and program evaluation have changed constantly, reflecting the evolving trends in
thinking and practice. Early evaluation texts define evaluation narrowly as the application of social
science methods to measure goal attainment or the systematic investigation of the merit or worth of
an object. More recent definitions include the concept of using the evaluation findings for program
improvement, development, and decision making. For example, Owen (1993:3) defined it as ‘the
process of providing information designed to assist decision-making about the object being
12
evaluated’. The matter of defining evaluation is of considerable importance, as different evaluation
approaches rest on different definitions of evaluation. Before turning to an analysis of contending
issues and definitions for program evaluation, it is worth clarifying that in general program
evaluation does not include the:
• evaluation of student performance, course or curriculum evaluation as in educational evaluation
• personnel or staff evaluation as in Human Resource Management and personnel appraisal
• evaluation of computer programs or software (Sharp and Lindsay, 1992:6).
This thesis refers to both ‘program evaluation’ and ‘project evaluation’. The terms ‘program’ and
‘project’ are used to differentiate between different levels of organisational activity. In some
organisations, a program is considered to be a large-scale intervention comprised of a series of
smaller scale projects, yet in other organisations the opposite meaning is adopted. These terms have
context-specific meaning within organisations. Similarly, the terms ‘program evaluation’ and
‘project evaluation’ refer to organisationally-specific levels of evaluation practice. In international
evaluation literature there is a tendency to refer to ‘program evaluation’, yet in agricultural
extension it is more usual to refer to ‘project evaluation’. The two terms will be used
interchangeably throughout this thesis, and should be regarded as synonymous.
The various terms used in evaluation (models, approaches, method, and tools) can also be
confusing, so an attempt is made here to explain the relationships between various terms and how
they are differentiated in this thesis. Later in the thesis, the terms ‘meta-model’ and ‘forms’ of
evaluation will also be used, and are included in this list.
• Models of evaluation are based on a normative understanding of how evaluation should be
conducted. These are usually associated with known authors who promote their particular
model of evaluation.
• Approaches of evaluation: a collection of similar models or methods of evaluation, or ways of
going about evaluation.
• Methods of evaluation: methods of data collection, retrieval and creation.
• Tools of evaluation: a loose term signifying anything that helps to conduct an evaluation study.
This includes evaluation models, methods of data collection, retrieval and creation, even
methods of data analysis.
• Forms of evaluation: Owen (1993) differentiated between five different types of evaluation,
13
referred to as ‘forms’, based on the purpose of the evaluation study. ‘Forms’ are used in
Chapter 4 to distinguish different evaluation studies in agricultural extension practice.
• Meta-model of evaluation: a framework for describing or categorising various different kinds
of evaluation. The categories of a meta-model may be differentiated by model, approach, forms
or methods.
The historical development of program evaluation
Many popular texts on evaluation begin with an overview of the history of evaluation. Some
describe the history of evaluation in terms of the predominant paradigms (Pawson and Tilley,
1997), others in terms of the predominant actors in evaluation (Shadish et al., 1995). Guba and
Lincoln (1989) describe four generations of evaluation: description, measurement, judgement; and a
‘fourth generation’ model of their own (Fourth Generation Evaluation). The following summary
incorporates some of the prominent features in the history of evaluation, according to Madaus et al.
(1983), Shadish et al. (1995), Guba and Lincoln (1989), Worthen et al. (1997) and Patton (1997).
Wherever possible, an Australian perspective is also provided, based largely on the work of Sharp
and Lindsay (1992).
Evaluation has a long history originating from many diverse disciplinary fields – including
agricultural extension. In many western countries, including Australia, trends in program evaluation
appear to be linked to changes in public policy. The historical development of program evaluation
has not been a smooth one. There has been a tendency towards conflict and short memory, with
regards to the shortfalls of one approach against another. The diversity of backgrounds of the
evaluator-theorists can perhaps also explain some of these conflicts. Despite much effort, it seems
that the quest for the ‘holy-grail model of evaluation’ to satisfy all audiences, from all program
contexts, was in vain.
The beginnings of program evaluation
Between 1930 and 1945, the work of Tyler came to have an enormous effect upon the field of
evaluation. Tyler (1967) coined the term ‘educational evaluation’, which meant assessing the extent
to which valued objectives had been achieved in an institutional program. Evaluation was
conceptualised by Tyler as a comparison of intended outcome with actual outcomes. The currently
popular outcomes-oriented evaluation is clearly not a new concept for program evaluation and this
highlights the circular nature of the debates within this field.
14
Most texts agree that modern program evaluation evolved primarily in the USA and was considered
a ‘semi-professional’ discipline by the 1960s. Although the root of evaluation development lies in
the US, in the 1960s, evaluation began to surface in Australia and later in Europe. The Baume
Report (Baume, 1977) traced the history of health and welfare evaluation in Australia from
Federation to the Whitlam Government (1972), and stated ‘there was almost no formal evaluation
during this time’ (cited by Sharp and Lindsay, 1992, Senate Standing Committee on Social Welfare,
1979a: 19). Despite this claim, there were some who can be identified as active practitioners and
theorists in the 1960s and earlier. They participated in the predominantly educational fields of
agricultural extension and Technical and Further Education (Sharp and Lindsay, 1992). The first
national evaluation conference in Australia was held in 1982.
Influences of government policy
Much of the history of program evaluation appears to be linked with government policy. Shadish et
al. (1995) suggest that in the USA an increased interest in evaluation in the period between 1946-
1960 can be largely attributed to the rapid economic growth after World War II, and in the
interventionist role many Western governments took on social policy during the 1960s. In the USA
during this period, educational services and social programs were expanded in efforts to address
rural poverty and despair in the inner cities (Madaus et al., 1983). The increase in public spending
on programs led to an increase in evaluation activity to determine whether these programs were
working.
Around 1960 the US government invested large sums of money towards programs in education,
income maintenance, housing, and health. These programs are collectively termed ‘Great Society’
programs. Expenditure on these programs was massive and constituted an 18-fold increase in US
public spending between 1950 and 1979 (Bell, 1983). This vast expenditure raised issues that
resulted in evaluation being promoted as a necessary component of social programs. Issues of
accountability for distribution of funds and concerns that program funds were being spent in ways
that caused undesirable results led to an increased demand for evaluation for accountability.
While large-scale programs with an evaluation component became commonplace in the US, this
trend also occurred in Europe and Australia. By the late 1960s in the United States and
internationally, evaluation research had become a ‘growth industry’ (Wall Street Journal, cited by
Rossi et al., 1979). In Australia, under the Whitlam Government of the 1970s, expansion of
15
government activities was seen in many fields. Many commentators regard the Whitlam era as
significant for its emphasis on social welfare programs, and with it among the first attempts to
address the policy and practice of program evaluation across many of the public sector services in
Australia (Sharp and Lindsay, 1992).
A multi-disciplinary enterprise
Along with Tyler’s educational model, the theory and practice of program evaluation had early
roots in diverse disciplines, including social psychology (Lewin, 1948), sociology (Lazerfield and
Rosenburg, 1955) and agricultural extension (Fisher, 1956; Rogers, 1962). Nowadays, evaluation
seems to be practised in almost every field of work, and yet not all of these get a mention in the
main texts on program evaluation. The literature of program evaluation is so vast that subsets exist,
which operate quite independently of one another. In particular, evaluation has long been practised
in the field of development studies: that is projects that are delivered in low-income countries, but
funded by donor organisations. Development projects were evaluated from as early as 1970 and
today there is still a strong tradition of evaluation (Jiggins, 1995).
In Australia, in the sectors of education, health and welfare, program evaluation was not made
mandatory or endorsed as government policy until the late 1980s. However, Sharp and Lindsay
(1992) point out that in the fields of agricultural extension and technical education, program
evaluation became the prescribed, widely implemented policy of the government agencies
concerned as early as 1968. Sharp contends that both in Australia and in US in this period,
agricultural extension came to have a great influence on evaluation practice:
Like the role of agricultural research on the development of statistical methods (Fisher, 1956) and
research on the diffusion of innovation (Rogers, 1962), the research on agricultural extension
provided opportunities which had a formative effect on the early approaches to program evaluation.
Some of the current practitioners of program evaluation in the human services gained their
experience through agricultural extension program evaluation. For example, Michael Quinn Patton
(former President of the American Evaluation Association) was involved for many years in extension
services through the Institute of Agriculture, Forestry and Home Economics at the University of
Minnesota (Sharp and Lindsay, 1992:11).
In 1947, some major projects in Australian agricultural extension were evaluated as part of their
planning and funding process. By 1962, South Australia had introduced a compulsory training
program in program planning and evaluation for agricultural extension officers – and a similar
16
compulsory scheme was introduced in New South Wales by 1968 (Sharp and Lindsay, 1992). In the
Murrumbidgee Irrigation Area (MIA), an agricultural extension project was conducted with a
deliberate research and evaluation strategy by the CSIRO, state and local authorities (Farquhar,
1961). This evaluation was conducted by trained evaluators and it was reported that ‘many lessons
were learned from the careful evaluation of the MIA extension experiments…probably the most
significant was the lesson in agricultural organisation’ (Farquhar 1961:210).
Many texts, many societies, many differences of opinion
Before 1970, limited published literature about program evaluation existed; many evaluations were
carried out by untrained personnel, others by research methodologists who tried to fit their methods
to program evaluation (Guba and Lincoln, 1981). Responding to the need for more sophisticated
evaluation models, many new books and papers were published by 1970, especially in the USA
(Cronbach, 1963; Scriven, 1967). Around 1973, a number of journals, including Evaluation Review
and New Directions in Program Evaluation were launched. In the USA in the 1970s, two
professional evaluation societies were formed: the ‘Evaluation Research Society’ and the
‘Evaluation Network’, later combining to form the ‘American Evaluation Association’. By 1986 the
International ‘Australasian Evaluation Society’ and the ‘Canadian Evaluation Society’ had formed,
and by 1995 new professional evaluation societies had been established to represent Central
America, Europe and Great Britain.
Together these new publications and fora for debate stimulated a number of new evaluation models
in response to the different situations and needs of specific types of evaluation. This body of
information revealed sharp differences in philosophical and methodological preferences, but ‘it also
underscored a fact about which there was much agreement: evaluation is a multidimensional
technical and political enterprise that requires both new conceptualisation and new insights into
when and how existing methods from other fields might be used appropriately’ (Worthen et al.,
1997:39).
Another issue that is highlighted by an historical perspective is the heavy influence of work from
the USA on evaluation theory. Many texts used in Australia are still produced in the USA; the
extent to which these texts are wholly relevant to the Australian context is debatable. The
development of program evaluation in Australia differed somewhat from that in the US, in that
internal evaluation was always more prevalent than in the US. This can perhaps be explained by the
attitude in the US to evaluation as an academic discipline and the existence of the numerous
17
evaluation higher degree courses there. In the US, evaluation tends to be seen as something that is
done by trained external evaluators, qualified with a PhD, whereas in Australia there tends to be a
greater focus on internal evaluation (Winston, J. pers. comm., 1997). There is certainly an emphasis
on internal evaluation in the field of agricultural extension in Australia (Dart et. al., 1998).
Diversity of evaluation models and the paradigm wars
Shadish and Epstein (1987) suggest that, as evaluation matured, its theory took on a special
character that resulted from the interplay among problems encountered by practitioners from
different academic disciplines. In a later text, Shadish et. al. (1995) go on to suggest that:
Evaluation may be the broadest methodological speciality…to inform evaluators about choosing
methods, it needs to discuss philosophy of science, public policy, value theory and theory of use
(Shadish et al., 1995: 31).
In the multi-faceted field of program evaluation, several paradigm wars raged throughout the
second half of the 20th century. Early differences concerned the focus on internal validity
(Campbell, 1969) against a consideration of external validity of evaluation findings (Cronbach,
1963). Another argument concerned the exclusive emphasis on pre-determined outcomes, against a
need to understand the quality of the implementation and the causal process that mediated program
impacts (Sechrest et al., 1979). However, the most cited debate has been between the constructivist
stance (e.g., Guba and Lincoln, 1989) and post-positivist stance (e.g., Campbell, 1969), which
correspond to qualitative and quantitative methods respectively. This particular conflict became
increasingly heated from 1970 until the 1990s. Datta (1994) notes that the ‘dialogues’ of three
successive presidents of the American Evaluation Association (Fetterman, 1992; Lincoln 1991;
Sechrest, 1991) were stringent in their defence of their methodological positions and in their attack
on the position of the other side. Datta suggested that this paradigm war became increasingly
unproductive during the 1980s and 1990s.
There have been attempts by several scholars to make peace in this paradigm war, with the
increased use of multiple or mixed methods (Cook and Campbell, 1979; Cronbach et al., 1980;
Datta, 1997).
18
2.3 What is ‘Good’ Program Evaluation?
As described, evaluation is used in many program contexts, in many countries and across many
different disciplines. It is also used for many different purposes, and the range of things that are
included under the umbrella of program evaluation is considerable. Even within one project there
may be several evaluation initiatives under way. For this reason, in ‘good’ evaluation the choice of
evaluation approach needs to be context-specific and take into consideration the purposes for which
the study is being undertaken. This view is endorsed by many evaluators including Worthen et al.
(1997) and Owen (1993).
Notwithstanding such recognition, several authors have advanced normative models for what they
believe entails good evaluation. Predictably, there are considerable differences between these
models. Despite the differences, an examination of these models can offer many insights into what
entails good evaluation. Therefore, six different approaches to evaluation were examined in this
study and from these a series of premises for good evaluation were developed. I propose a
classification of evaluation consisting of the following six normative approaches, each of which will
be discussed in turn:
i. Experimental approaches
ii. Testing-objectives approaches
iii. Decision-management approaches
iv. Judgemental approaches
v. Pluralist-intuitionist approaches
vi. Theory-driven approaches
These six approaches are an amalgam of the meta-models of Smith (1994), Stake (1973), House,
(1978), Stufflebeam and Webster, (1981) and Worthen et al. (1997). In reality this classification,
and the ones it is based on, are over-simplistic because of the complexity of the models they attempt
to classify. Wadsworth (1991: 62) points out that ‘One would really need a three-dimensional map
on which to try to plot a full picture and even then some of the techniques would have to be moved
back and forth between one category and another’. However, the meta-model proposed does allow
an examination of the range of things that theorists contend are important for ‘good’ evaluation.
19
The theories behind both of the first two approaches in the meta-model (experimental approaches
and testing-objectives) have been partly superseded by more modern approaches. However, they are
included in the meta-model because they are still seen in practice and because critique of these
approaches paved the way for the more contemporary approaches; theory-driven, judgemental,
decision-management and pluralist-intuitionist.
2.4 Six Normative Approaches to Evaluation
i. Experimental approaches
Experimentalism, often referred to as the ‘classical paradigm’ in program evaluation, is
characterised by the work of Popper (1959), Campbell, (1991), and Cook (1966). Experimental
approaches construe evaluation as a knowledge generation research task. Therefore good evaluation
creates knowledge and theory about a program situation that can be ratified by empirical data.
Campbell focused much of this energy on the notion of minimising the ‘threats to internal validity’,
and developing the most plausible explanations for the results of experiments.
Experimental approaches are based on a Humean theory of causation. The basic task is to
hypothesise or demonstrate the constant conjunction whereby action ‘X’ produces outcome ‘Y’.
Advocates of such approaches promote the use of experimental and strong quasi-experimental
designs that provide the strongest causal inference. Experimental designs in evaluation in principle
follow the same rules as agronomic trials: the ‘treatments’ are randomly assigned (in this case to
program participants) to either a ‘treatment’ group or to a control group. Experimental approaches
may also have pre- and post-test designs, so that changes can be monitored before and after the
treatment period.
Experimental designs offer a logical approach for determining whether certain program variables
affect program outcomes. While experimental approaches can be criticised on several fronts, there
is little doubt that experimental evaluation paved the way for important insights into evaluation
theory, for example, the work of Campbell (1991, 1969) and Cook (Cook and Campbell, 1979) who
focused on the ‘internal threats to validity’ associated with experimental design.
However, for several reasons, experimental designs did not always prove their value, and many
20
expensive designs yielded inconclusive findings. Returning to the analogy of agronomic trials, the
secret to a good experiment is to ensure that the plots are randomly allocated and that all other
variables are controlled. Even with plants, this can produce results that are difficult to apply to the
reality of the farmers’ fields, as the controlled conditions of the research station do not correspond
to real life conditions. It can be even more difficult to control variables amongst human participants
and it is often unethical to subject one group of people to a treatment, and deny it to others. Even
where large trials have been conducted, the results have been disappointing. Variables between the
responses of different people within the same treatment group are seen as ‘noise’ in the experiment.
Therefore, the results of an experimental design only provide information as to whether the program
‘worked’ for the entire group of people or not. It provides no clues as to why the program may work
for some individuals and not for others.
Indeed, for practical purposes, experimental designs often exclude many of the contextual factors
that influence cause-and-effect relationships. It can be argued that these contextual factors are the
very thing in which evaluators should be most interested. Theory-driven approaches to evaluation
for example, reject a key notion inherent in the logic of experimental evaluation in that it cannot
fully take into account either the key mechanisms linking programs with outcomes or the richness
of heterogeneous contexts (Feinstein, 1998). Indeed, theory-driven evaluators such as Pawson and
Tilley (1997) attack experimental evaluation for yielding very little in terms of learning about
programs. Their reasoning is:
By its very logic, experimental evaluation either ignores underlying process, or treats them
incorrectly as inputs, outputs or confounding variables, or deals with them in a post hoc and thus
arbitrary fashion (Pawson and Tilley, 1997:54).
Despite their limitations, experimental designs are still used and valued in many circles – especially
in the United States.
ii. Testing-objectives approaches
Evaluation approaches in the testing-objectives category are focused on determining whether the
stated goals or objectives of a program have been achieved. Tyler (1967) was among the first to
develop this approach which he referred to as ‘educational evaluation’. Good evaluation under this
model depends on being able to accurately determine the extent to which stipulated objectives have
been reached. Tyler defined evaluation as the process of determining the extent to which the
21
objectives of a program are attained (Worthen 1997:82).
Guba and Lincoln (1989) relate, how in 1933, Tyler was engaged to carry out an eight-year study of
an alternative school curriculum. The idea was to collect information about the extent of pupil
achievement against defined objectives, to guide revisions to the curriculum. The results of each
trial were not available until after the trial was complete. This process was iterated over successive
course offerings until the curriculum was found to produce an appropriate level of achievement. In a
sense, this was formative evaluation and the evaluators’ role was that of describing the attainment
of objectives over time. A point to note about this approach, however, was that the results of the
evaluation were not made (publicly) available until after the program was complete. This implied
that the program could not be modified during implementation. This approach to evaluation is still
used in extension evaluation nowadays.
Objectives-testing approaches differ from the previously popular model of ‘comparative
experiment’ in that they do not involve the expensive and disruptive comparisons between
experimental treatments and the control. Since Tyler’s approach calls for the measurement of
behaviourally defined objectives, it concentrates on learning outcomes instead of organisational and
teaching inputs (Madaus et al., 1983). Educational evaluation that predated objectives-testing
approaches tended to focus on measuring the performance of pupils, rather than examining the
curriculum or program itself.
Careful articulation of the objectives of a program – an essential part of an objectives-testing
approach – can have positive ramifications for both program planning and delivery. Very few
evaluations are conducted without some consideration of the extent to which objectives have been
achieved.
However, critics of objectives-testing approaches found it unsatisfactory to be unable to make
corrections to the program until it was complete. In other cases, practitioners were unwilling to
stipulate pre-determined ‘outcomes’, as they were unsure at the onset of the programs about the
appropriate outcomes. Both of these criticisms can still be heard today of evaluations that are
strongly outcome (or objectives) focused. Another important criticism of this model was levelled by
Stake (1967), who pointed out the neglect of judgement in the objectives-testing model. He
suggested that evaluation requires standards against which judgements can be made and that the
inclusion of standards must be value-laden. Scriven (1967) also pointed out that the objectives
22
themselves should also be treated as problematic and subject to scrutiny. His argument was that
testing the extent to which the goals of a program have been achieved does not determine the worth
of that program for society in general; it does not assess the goals themselves.
iii. Decision-management approaches
Decision-management approaches aim to serve decision-makers’ needs in managing programs.
Evaluation models fitting into this category are Patton’s Utilization Focused Evaluation (1997), and
Stufflebeam’s CIPP model (context, input, process, and product) (Stufflebeam and Webster, 1981),
that relies on a modified systems analysis approach. Good evaluation, under this approach, would
produce findings that are used to bring about effective decision-making for the organisation. This
type of evaluation is largely judged by how well it has served those commissioning the evaluation.
For example, (Caron, 1993: 62) states that evaluation is ‘carried out for the organisation. It is a
function of management’. Therefore, in evaluation of this genre the aim is to serve the needs of
program managers rather than of the program’s clients or the wider public.
Patton, who is probably the most well cited advocate of decision-management approaches, defines
program evaluation as:
…the systematic collection of information about the activities, characteristics, and outcomes of the
programs to make judgments about the program, improve the program effectiveness, and/or inform
decisions about future programming. Utilization focused program evaluation (as opposed to program
evaluation in general) is evaluation done for and with specific, intended primary users for specific,
intended uses (Patton 1997: 23).
According to Patton (2000:1) Utilization Focused Evaluation is an ‘approach to making evaluations
useful, practical, accurate, systematic, and ethical’. This involves matching the evaluation approach
and the design to the information and decision needs of primary intended users, taking into account
‘other stakeholders, political factors, organisational constraints, project/program history, available
resources, and cultural factors of a specific evaluation context’ (ibid: 1). He suggests that this
allows for ‘situationally responsive’ evaluations.
While decision-management approaches offer sensible suggestions for increasing the likelihood of
the evaluation findings being used, they have been criticised for the apparent ‘cosy’ relationship
between the evaluator and the project management. For example Pawson and Tilley (1997) caution
that these pragmatic approaches ‘suggest a Rothschildian vision with research skills for hire’ – the
23
Rothschild principle being:
the customer says what he wants, the contractor does it if he can, and the customer pays (Department
of Health, 1993; cited by Pawson and Tilley 1997: 14).
Thus, decision-management models are challenged on their ability (or inability) to present
unpalatable information to the management. Some decision-management models can also be
questioned on their assumption that evaluation serves the needs of program managers and staff
rather than program clients or the greater public.
iv. Judgement approaches
Judgement approaches involve the professional judgement of experts. This approach includes
Scriven’s (1976) ‘Goal-free’ evaluation and Eisner’s (1985) Connoisseurial model of evaluation.
Here, evaluation is seen as a determination of the merit or worth of a program – and an evaluation
of this type would be judged on the basis of the accuracy and lack of bias in conducting a
judgement of worth or merit of the program. Definitions of evaluation for this are characterised by
the words of Scriven:
It’s extremely important not to over simplify the logic of evaluation by defining evaluation as, for
example, the provision of information to decision-makers. Evaluation is what it is, the determination
of merit or worth, and what it is used for is another matter (Scriven, 1980: 8).
Bad is bad, and good is good, and it is the job of evaluators to decide which is which. (Scriven, 1986:
19)
An extreme model from this genre is Scriven’s ‘Goal-free Evaluation’. ‘Goal-free evaluators begin
evaluations totally blind’ (Scriven, 1976: 137) to stated goals. Evaluators have to discover what
effect the program has and match their effects against the needs of those who they affect (Scriven,
1976: 137). The ‘Goal-free’ evaluator avoids contact with program staff, who may bias the
conceptualisation of the evaluation questions. The evaluator is required to judge the merit of the
program in terms of empirical evidence of the effects of the program intervention.
In Goal-free Evaluation, Scriven attempts to address the problem of over-emphasis on program
outcomes (or objectives), stating that they should be totally ignored. The evaluator’s job, according
to Scriven, is to locate any and all program effects, intended or not, that might help solve social
24
programs. His argument is that testing the extent to which the goals of a program have been
achieved does not determine the worth of that program for society in general; it does not assess the
goals themselves. Goals are a poor source of such effects, and are ‘often vaguely worded to muster
political support and rarely reflect side effects that are difficult to predict’ (Shadish et al., 1995:80).
Goal-free Evaluation does serve to remind evaluators of three very important points. Firstly, in
addition to examining stipulated outcomes, evaluation should look beyond the goals of the program
itself, ie., it should consider unexpected outcomes. Secondly, in some situations it may also be
necessary to question the value and logic of the outcomes themselves. A third contribution of Goal-
free Evaluation is the concept that good evaluation involves some judgement of merit or worth of
the program’s impact.
However, Shadish et al. (1995: 114) suggest that ‘some evaluators have difficulty accepting the
notion that they can, much less should, evaluate a program without knowing its goals’. While most
evaluators have heard of Goal-free Evaluation, they may not see it as central to their thinking about
evaluation, and they still use goals as the most common source of dependent variables (Shadish and
Epstein, 1987). Another criticism of the Goal-free Evaluation is that while it may be a very useful
theory, it is not necessarily a practical model. Indeed, few cases of Goal-free Evaluation have been
documented.
Critics of Goal-free Evaluation consider the term ‘goal-free’ evaluation to be a misnomer. The
evaluator does not get rid of all goals, but replaces the goals of the project staff with more global
goals based on societal needs and basic standards of morality (Alkin, 1972 cited by Patton
1997:183). Patton argues that Goal-free Evaluation:
…eliminates only one group from the game, local project staff. He (Scriven) directs data in only one
clear direction – away from the stated concerns of the people who run the program. He addresses an
external audience such as legislative funders. But, in as much as these audiences are ill defined and
lack organisation, I am unconvinced that the standards he applies are none other than his very own
preferences about what program effects are appropriate and morally defensible. Goal-free Evaluation
carries the danger of substituting the evaluator’s goals for those of the project (Patton 1997:182).
25
v. Pluralist-intuitionist approaches
Pluralist-intuitionist approaches share a common commitment to value pluralism; that is to identify
and preserve multiple value perspectives. This includes the models of Stake (1967), Guba and
Lincoln, (1989) and Wadsworth, (1991) and the participatory monitoring and evaluation approaches
to evaluation that are seen in development projects (Estrella and Gaventa, 1998). All the approaches
in this category are highly client centred and present a subjectivist ethic and epistemology within a
liberal ideology (Smith, 1994). Good evaluation under this normative model involves creating
increasingly sophisticated and shared constructions of reality. For example, evaluators employing
Guba and Lincoln’s ‘Fourth Generation Evaluation’ (1989) judge evaluation by the extent to which
it succeeds in involving a range of participants in sharing their views. Guba and Lincoln, (1989) for
example, advocate that evaluators ought to strive for, not validity, but increasingly sophisticated
constructions of reality. Some versions of pluralist-intuitionist approaches also focus explicitly on
social justice, and liberation of program beneficiaries from oppression.
Pluralist-intuitionist approaches evolved as a reaction to the limitations of the logical positivist
approach to knowledge generation such as that adopted in experimental approaches. To overcome
the limitations of these approaches, a number of overlapping frameworks emerged that were based
on the tradition of anthropology. Pluralist-intuitionist approaches include interpretivism, relativism,
naturalistic inquiry, constructivism, and feminist inquiry. What they hold in common is the rejection
of the existence of a singular knowable reality. This ontological position has significant
ramifications for the field of evaluation; evaluation is concerned with ‘apprehending reality’, thus
questioning the nature of ‘reality’ is clearly going to affect evaluation practice at several levels. In
Chapter 5, two models of this genre are examined in more detail.
One family of approaches that fits into this genre is the Participatory Monitoring and Evaluation
approaches (PM&E), also known as Participatory Learning and Action approaches. These
approaches are used increasingly in development projects. This type of evaluation is not generally
referred to in the main texts on program evaluation, but is gaining increasing attention in the
international literature of agricultural extension (Estrella and Gaventa, 1998; Guijt and Gaventa,
1998). Like Fourth Generation Evaluation, PM&E approaches share a common commitment to
participation of the beneficiaries of a program. However, PM&E approaches tend to emphasise
action a little more strongly, and to be influenced by models of action research. Good evaluation
26
under PM&E approaches would be based on the worldview of the people it is aiming to serve, and
would encourage participants to take self-directed action to improve their own or collective social
conditions. PM&E approaches were influenced by the Brazilian activist Friere, (1972) and the work
of action researchers such as Lewin (1948) and Whyte (1991).
PM&E approaches were developed in reaction to dissatisfaction with the rapid, yet participatory
approaches named ‘participatory rural appraisal’ (PRA), and the limitations and dangers of reliance
of non-participatory forms of monitoring. PM&E approaches are largely qualitative, participatory
approaches with a focus on organisational learning and empowering the beneficiaries of the project.
Pluralist-intuitionist models reinforce the notion that in good evaluation, a range of different
perspectives should be included. The evaluator is reminded that programs are conducted in a
political environment, and the values of the different stakeholders must be brought to the surface,
not excluded or ignored. Pluralist-intuitionist approaches offer a way of appreciating and
incorporating the different ‘theories of action’ or ‘constructions’ of the project held by different
stakeholders. These approaches can be especially valuable in the sort of evaluation that involves a
broad range of stakeholders involved with disparate views. They are often highly useful in
participatory programs, and particularly when the program intervention emphasises empowerment.
Like other approaches, pluralist-intuitionist models are not appropriate to every evaluation context.
In some cases, evaluations of this type may not meet the client’s information needs. Two examples
illustrate this point. Firstly, the client of the evaluation may not entertain the view that there are
many possible versions of ‘reality’ but instead may request a definitive account of the ‘facts’ about
program success. Secondly, clients who require firm recommendations to be a part of the evaluation
output may be dissatisfied with this approach. Evaluators who conduct pluralist-intuitionist
evaluation generally avoid making recommendations, and the evaluator may refrain from offering
concrete ‘recommendations’, and see their role as more of a ‘facilitator’ than a ‘judge’.
Many of the pluralist-intuitionist approaches, such as Fourth Generation Evaluation, strive to gain
consensus between all stakeholders. This can be questioned on two fronts. Firstly, the achievement
of consensus is not always a realistic goal, especially when different stakeholder groups may have
strong political differences. Secondly, the achievement of consensus is not always a helpful goal,
especially if it is won at the expense of ‘less powerful’ or minority opinions being silenced in
favour of the majority opinion.
27
vi. Theory-guided approaches
Recently there has been a growing interest in theory-guided approaches to evaluation (Pawson and
Tilley, 1997; Chen, 1990; and House, 1991). These approaches involve the construction of
‘program theory models’ – casual models that elaborate how a program is intended to achieve
intended outcomes. Bickman, (1987: 5) defines program theory as ‘the construction of a plausible
and sensible model of how a program is supposed to work’. Chen explains that program theory has
both descriptive and prescriptive concerns and defines program theory as:
A specification of what must be done to achieve the desired goals, what other important impacts may
also be anticipated, and how these goals and impacts would be generated (Chen 1990:43).
Good theory-driven evaluation would, therefore, develop highly plausible theory concerning how a
program works, in what situations and why. This theory would be the best explanation of empirical
data, and any alternative contending explanations would have been refuted by the empirical data. In
Chapter 5, one model of this genre is examined in more detail: Pawson and Tilley’s (1997) Realistic
Evaluation.
The advantages of theory-driven evaluation are that it goes beyond establishing whether a program
works, allowing an understanding of how a program works. Wortman (1983: 20) comments that
‘program evaluation is a multi-disciplinary and unfortunately largely atheoretical activity’. Chen
(1990) suggests that this atheoretical view tends to result in a simple input/output (or black box)
type of evaluation, characterised by adherence to a step-by-step cookbook method of doing
evaluation. He argues that such simple evaluations may provide evidence as to whether a program
works or not, but fail to identify the underlying mechanisms that generate the treatment effects, and
hence to pinpoint the deficiencies of the program for future program improvement. Therefore,
theory-driven approaches offer an important contribution to good evaluation: that evaluation should
move beyond merely asking why a program works, but look specifically at what aspects of a
program work in which situations and why.
Critics of this approach, such as Scriven (1994), disagree with the notion that understanding the
underlying theory of a project is critical and state that program theory is not needed for evaluation –
as it is not needed to determine whether something works or not:
28
In program evaluation, we are concerned to establish the merit, worth, quality or value of
programs…We do not need to know how the programs work or why they fail to work, or even what
their components are (Scriven, 1994:75).
Scriven (1994) contends that understanding the mechanisms behind an intervention can be
advantageous, but the main necessity is to get the evaluation right, and ‘jeopardising that by
diversion of effort into the direction of explanation diagnosis and remediation is all too common’
(ibid: 75). Indeed, it seems that in some cases, depending on the nature of the evaluation study, an
investigation into the underlying mechanisms might be something of an overkill; while unpacking
the black box is generally considered useful to any evaluation, it may not be necessary for all, and
its relevance depends on the nature of the evaluation inquiry (Scriven, 1994).
Depending on the nature of the evaluation questions, a theory-driven approach may or may not be
appropriate. Firstly, as argued by Scriven (1994), there are situations when the cost-benefit ratio of
conducting an extensive theory-driven evaluation is not justifiable. Secondly, in highly participative
programs there may be instances when the nature of the evaluation questions rightfully places more
emphasis on answering specific stakeholder concerns than on investigating carefully crafted
analytical suppositions.
Leviton (1994) states that theory-driven approaches are most seriously deficient in assignment of
value to community-based programs. She suggests that when ‘experts’ frame questions for the
community they encounter understandable fury. This is noteworthy as many extension programs are
highly participatory and community based in orientation. In many evaluation studies in extension,
the evaluators would endorse the position that the clients’ and key stakeholders’ values are the ones
that should be used to assign value. Thus, in these cases, the use of a deductive-normative theory
for developing the key evaluation questions would be rejected in favour of developing key
questions through the achievement of some sort of consensus from the project stakeholders with
regard to what they believe should be measured in the evaluation.
Theory-driven evaluation does not focus overtly on how unintended or unanticipated outcomes are
to be investigated. Because of this there is a tendency to get drawn into the micro mechanisms that
have been postulated, neglecting to look at the data afresh for new emerging patterns or
mechanisms. Hamilton-Smith and Hopkins (1998) suggest for example, that in Realistic Evaluation,
no overt demand is made to include unintended consequences once plausible and rival explanations
are developed. Theory-driven models have also been criticised as they do not focus on how the
29
evaluation findings are to be transferred into decisions that will lead to improved programs.
Commonalities and differences between evaluation approaches
The examination of different normative models illustrates that there are many different concepts
concerning ‘what evaluation is supposed to do’. However, it can be said that ultimately all of the
evaluation approaches have one thing in common; they attempt to bring about improved programs –
that is, programs that better meet the needs that they were designed to address. In turn, these
improved programs are all intended, in one way or another, to ameliorate social, economic or
environmental problems. Figure 2 shows a program theory model of the different approaches to
program evaluation. While each of the approaches strive to bring about improved programs, they do
this by different means, and therefore have different intermediate outcomes. Other authors have
presented program theory models for program evaluation including Patton (1997), Rogers (1996)
and Shadish et al. (1995).
Patton (1997) suggests that evaluation should be focused on using different organising concepts. He
suggests that in addition to program goals, evaluation can be focused upon: future decisions, critical
issues or concerns, key questions and the multiple perspectives of different stakeholder groups.
Figure 2 summarises the central organising concepts in each of the six normative approaches to
evaluation, as well as the outcomes and intermediate outcomes for each.
30
Figure 2
Figure 2 Outcomes and intermediate outcomes of six different normative approaches to
evaluation
The central differences between the six normative approaches reviewed, and summarised in Figure
2, give rise to the following questions:
• Which organising concepts should an evaluation be built around – goals, unexpected outcomes,
concerns and issues, or intended use?
• To what extent should the evaluation include judgement by the evaluator? What does this
judgement consist of? If not the evaluator, who should be making the judgements?
• What efforts should be made to ensure that the evaluation findings are used?
a) An improved project that can better meet needs or b) improvements in the next project
Needs are met and problems are ameliorated
Normative approach
Intermediate outcomes
Intermediateoutcomes
Outcome
Greater outcome
Management decisional approaches
Intended users involved in design and conduct of the evaluation – so accept findings
Information to make better management decisions
Key questions
Information about what works and does not work
Experimental approaches
Some of these cause-effect relationships are either refuted or supported
Cause-effect relationships
Judgemental approaches
Judgement of process and outcomes
Definition of standards of merit and worth against which to judge the approach
Intended and unintended outcomes and client needs
Organising concepts
Pluralist-intuitivist approaches
Intended users involved in design and conduct of the evaluation – so come to consensus
Mutual learning and self-directed action
Stakeholders concerns
Theory guided approaches
A greater understanding of the internal processes of the program
A refined theory of how the project does work in different contexts
Underlying program mechanisms occurring in contexts
Objectives testing approaches
Understanding of the extent to which defined objectives were met
Measurable objectives
Revised statements of objectives
31
• Who should the evaluation serve – society, the evaluation client, or the users of the service?
• To what extent should the key stakeholders be involved in the conduct and design of the
evaluation?
• Who should decide on the key evaluation questions or performance indicators, ie: who decides
what is measured?
• To what extent should evaluation be guided by the underpinning theory of the program
intervention?
These questions are not easily answered, yet the answers for each specific evaluation study may
guide the would-be evaluator toward a particular model of evaluation and away from others. I
suggest that within one large extension project it may be necessary to adopt more than one of these
normative approaches, entirely or partially, to meet the evaluation demands of stakeholders and to
lead to improved projects. Because of this, I suggest that a pragmatic approach be adopted, whereby
not only methods, but choice of normative approaches used to guide an evaluation be considered in
the light of the purpose of the evaluation study. In other words, the above questions could all be
answered with the statement ‘it depends on what you are trying to do’. This view corresponds with
that of Kaplan (1964) who suggests that the emphasis must be on making the methodology fit the
needs of the society, its institutions and its citizens, rather than the reverse.
Therefore, I advocate a framework for picking and choosing between different evaluation models.
‘Picking and choosing’ between different evaluation models can signify one of two things: choosing
one evaluation model in particular to guide the evaluation or choosing bits of different models, to
develop a ‘tailor-made’ evaluation. Either of the two approaches can be used.
2.5 Bricolage and Mixed-Approach Designs
Theorists argue about the extent to which it is acceptable to mix models and research paradigms
within an evaluation study. The contending points of view are that:
a) Research based on different discourse (or paradigms) is ‘incommensurable,’ i.e., cannot be
brought together in a coherent interpretation and cannot be reconciled (i.e., Guba and Lincoln,
1989).
b) Research is always potentially biased; hence we need to uncover through analysis and
argument, the biases inherent in different approaches and design approaches that systematically
32
address these biases, by putting together packages of imperfect theories and methods with
counterbalance the biases (i.e., Cronbach et al., 1980).
This thesis is premised on the second argument; that a combination of different models and tools
can better off-set bias, at the same time as meeting a range of evaluation needs. In the case of the
new genre of large extension projects, mixed methods designs occur in two distinct ways. Firstly, it
may be that within the evaluation bricolage several distinct evaluation approaches will be required
to serve different evaluation audiences. For example in a large hypothetical extension project, the
evaluation bricolage might consist of:
• internally conducted evaluation of the ‘logic’ of the core programs using Bennett’s Hierarchy
and subsequent refinement of the valued outcomes for each core program
• internally conducted evaluation of each of the core programs against valued outcomes
• externally conducted economic impact analysis of the whole project
• internally conducted reporting against predetermined milestones
• internally conducted evaluation of staff satisfaction, for project learning.
Secondly, mixed method studies may be used within one component of an evaluation bricolage.
Different methods may be used for different stages in the process of answering the same question,
or to validate the findings. For example, in the above scenario, the evaluation of achievement of
program outcomes could have been conducted by using a mix of focus group technique and
questionnaire.
In summary, the framework advocated in this thesis can be described as ‘bricolage’; a term that has
been used to describe qualitative research (Denzin and Lincoln, 1998) and cultural studies (Nelson
et al., 1992):
The bricoleur produces a bricolage, that is a pieced-together close-knit set of practices that provide
solutions to a problem in a concrete situation. The solution (bricolage) which is the results of the
bricoleur’s method is an emergent construction (Weinstein and Weinstein, 1991: 161; cited by
Denzin and Lincoln, 1998: 3).
In this case, the bricolage is the solution to a whole collection of evaluation needs that exist for one
large project. Figure 3 illustrates how a bricolage may be developed. The idea is that firstly the
existing evaluation and current practice should be audited. Following this, an investigation into the
33
needs of the stakeholders of the evaluation is conducted. This stage includes identifying and
negotiating with the evaluation audiences, determining resource availability, and if necessary,
drawing a program logic diagram to understand how the project is believed to meet needs. These
facets together allow an identification of evaluation needs and gaps. From this, the intention is to
select appropriate new approaches, and to trial and modify these to meet the evaluation needs of the
project. This process is cyclical; the whole cycle may have to be run several times in the course of a
project as evaluation needs may change.
Figure 3 The process of modifying an evaluation bricolage for a project
The idea of an evaluation bricolage is to build on existing evaluation practice and project
experience. Therefore, the aim of this thesis is to develop one new piece of the bricolage – that is,
an approach that can, in combination with other approaches, address the current challenges
associated with evaluating the new genre of agricultural extension projects in Australia. An
essential part of developing a new approach to evaluation is that it should be piloted and modified
to suit the particular program context, and to determine whether it does indeed meet needs, and
augment the existing evaluation bricolage.
Identify information and current evaluation practice (understand the existing bricolage)
Identify evaluation needs of the project
Choose or develop a new evaluation approach to meet needs/gaps
Pilot the new approach
Evaluate the new approach (meta-evaluation)
Review and revise the bricolage as evaluation needs are clarified and change
Determine the evaluation audience
Establish the resources available
Negotiate with evaluation audience
Clarify the program logic if necessary
34
2.6 A Normative Framework for Evaluation
Alkin et al. (1979: 25-26), in their review of evaluation literature, point out that different definitions
of evaluation lead to different notions of ‘good evaluation’. Clearly, depending on the normative
approach being adopted, different criteria can be applied in determining what constitutes good
evaluation. Still, it is possible to combine some of the features from all six approaches reviewed to
develop a useful guide for determining what constitutes good evaluation.
The following paragraphs outline a series of criteria that I suggest should be present in evaluation. I
describe eight premises, which I then re-phrase as meta-evaluation questions. In developing these
premises I have drawn strongly on the examination of the different normative approaches for
evaluation.
Premises for good evaluation
Firstly, good evaluation is relevant to the context and purpose in hand. Thus, evaluation models for
agricultural extension should consider the environment in which the program is being conducted
and the evaluation needs of the stakeholders. This view is strongly advocated by Patton in his book
Utilization Focused Evaluation (1997). For this reason, Chapter 3 presents an examination of the
context in which agricultural programs are planned, delivered and evaluated and Chapter 4
examines current practice in agricultural extension in Australia.
Secondly, as advocated by Rogers, (1996) good evaluation contributes to improved projects; that is,
projects which meet needs. Rogers credits this premise to Madaus, Stufflebeam and Scriven
(1983):
Ultimately the value of program evaluation must be judged in terms of its actual and potential
contributions to improving learning, teaching and administration, health care and health, and in general
the quality of life in our society (1983: 18).
Improved projects can come about in one of three ways; improving existing projects, developing
alternative projects, or developing new projects. The implication of this is that good evaluation is
able to show evidence that the evaluation did contribute to project improvements.
The third premise, borrowed from theory-driven evaluation, is that good evaluation considers the
theories underpinning the project (Chen, 1990). Considering the arguments of Pawson and Tilley
35
(1997) and Chen (1990), evaluation should be guided by program theory, and should not be simply
a black box venture. Even Scriven (1994), who levelled early criticisms of theory-driven evaluation
has recently conceded that while program theory may not always be essential, it is usually helpful.
The fourth premise is that good evaluation seeks to understand the differentiated nature of program
effects including any unanticipated outcomes. Pawson and Tilley (1997) stress that it is important to
look beyond whether a program works or not, and to consider the extent to which outcomes are
realised in different contexts. Scriven’s work on Goal-free Evaluation serves to remind us to look
beyond the program objectives and to also investigate unexpected outcomes. Whether an
appreciation of differentiated program effects is achieved through collecting the different
constructions of reality, (as in pluralist-intuitivist approaches) or by generating and testing theory
about how a program works in different contexts (as in a theory-driven approach), some notion of
the stratification of evaluation findings should be included. A purely positivist stance does not serve
evaluation well and may subsume genuine differences in findings in an attempt to present an answer
to the question, does the program work or not?
Fifthly, even if the focus on the program is not explicitly upon social justice, any process of
evaluation should be socially and politically just, and conducted in an ethical manner. The views of
all people affected by the project should be sought, rather than giving privilege to one potentially
dominant group. In good evaluation attention is paid to who is given voice in the determination of
the indicators for the evaluation. This is stressed strongly in some of the pluralist-intuitionist
approaches, particularly Guba and Lincoln’s (1998) Fourth Generation Evaluation.
Sixthly, the time and resources put into an evaluation should not outweigh the benefits. This is in
recognition of the need for what has been termed ‘real time’ evaluation. If evaluation findings are
intended to influence practice and policy, then they must produced in time to be useful to decision-
makers. Re-structuring and change seem to be endemic in organisations nowadays, so it is
imperative that evaluative information is timely, otherwise it will be dated by the time it is received.
Again the resources expended should be weighed against the benefits expected from an evaluation
study, for example for a quick snap shot evaluation, a detailed investigation of underlying causal
relationships (as in theory-guided approaches) might be uncalled for.
The seventh point is that, for any model of evaluation, the design, data collection and analysis of the
evaluation must be valid. This presents problems in defining criteria for validity that are equally
36
applicable across different paradigms. To some extent, each underpinning paradigm will come with
some notion of good design and data collection. For example, with the Fourth Generation
Evaluation model, good evaluation involves including as many relevant constructions of the
situation as possible. For the realists, good evaluation might concern developing theory that can be
most closely correlated with reality. For advocates of Utilization Focused Evaluation it might be
that program staff use the evaluation findings. So the premise here is that good evaluation employs
a design and data collection that is valid under the paradigm associated with the evaluation
approach.
The eighth point, strongly stressed by Scriven, is that evaluation should attempt to judge the merit
and worth of the project. Who it is that should be making this judgement will depend on the purpose
of the evaluation. It may not be the evaluator that makes this judgement, such as in the case of
participatory approaches to evaluation.
Meta-evaluation questions summarised
The eight premises for good evaluation can be reworded as meta-evaluation questions (see below)
These meta-evaluation questions will be used to guide the review of current evaluation practice
(Chapter 4), and for the meta-evaluation of the novel approach that is tested as part of the case study
(Chapter 8). The meta-evaluation questions are:
1. Is the evaluation relevant to the context and purpose in hand, and does it fulfil the evaluation
needs of the stakeholders?
2. To what extent can the evaluation contribute to the project being better able to meet needs?
3. Is the evaluation guided by program theory?
4. Does the evaluation make an attempt to account for different program outcomes?
5. Are the evaluation processes socially and politically just?
6. Is there an appropriate ratio of costs to benefits for the project from the process (and outcomes)
of the evaluation?
7. To what extent are the design, data collection and analysis of the evaluation valid?
8. Does the evaluation attempt to judge the merit and worth of the project?
37
Definition of evaluation
From the premises outlined above, and the discussion of normative models of evaluation, an
overarching definition of evaluation was developed adopted in this thesis. It is:
Program evaluation concerns the systematic collection of information, in order to improve
decision making and enhance organisational learning, with the ultimate aim of bringing
about programs that better meet needs and lead to amelioration of targeted social, economic
and environmental problems.
2.7 Conclusions
In this chapter an attempt was made to make sense out of the diverse and conflict-ridden field of
program evaluation. Over the last 30 years there has been an explosion of new models advancing
theory as to how best to conduct evaluation. For this thesis, a pragmatic stance is adopted, in which
I suggest that the choice of evaluation model should be grounded in knowledge of the program
context, and of the key evaluation task. In line with this, the following two chapters will examine
the needs within the context of agricultural extension in Victoria (Chapter 3) and the existing
evaluation practice in Australia (Chapter 4).
I suggest that new approaches should be selected (or developed) with an understanding of the
different approaches of program evaluation available, and with some notion as to what constitutes
good evaluation. New approaches should be developed through field-testing and meta-evaluation,
against the contextual needs of the organisation. There should be an understanding of how the new
approach attempts to contribute to bringing about improved programs that meet stakeholder needs,
and that ultimately contribute to ameliorating social, economic and environmental problems.
For this reason, a series of meta-evaluation questions were developed to guide the formation of a
new approach. Chapter 7 describes how a novel evaluation approach was modified, and
implemented across a Victorian extension project. In Chapter 9 this approach is evaluated using the
meta-evaluation questions developed. In this way the framework contributes to achieving the aim of
this thesis, which is to develop a model of evaluation for program evaluation of Australian
extension.
38
CHAPTER 3
EVALUATION NEEDS IN AGRICULTURAL EXTENSION
In this chapter I present an overview of evaluation needs in Australian agricultural extension
projects. Emerging demands for evaluation are linked to changes in both the philosophy of
extension and changes in the structure of extension institutions. In the last 10 years, Australia
has seen major changes in the way the Government manages public sector programs. There
have also been changes in the philosophy that underpins agricultural extension and
associated changes in extension practice. These changes have influenced the way publicly-
funded extension projects are planned, managed and valued, and present unresolved
dilemmas with regard to evaluating the new genre of projects. An analysis of the changing
nature of extension revealed the need for evaluation that can deal with participatory, group-
based extension, and that can identify and value diverse impacts. An analysis of the changing
organisational environment of extension revealed there is an increased demand for internal
evaluation that can deal with multiple audiences, and promote organisational learning and
communication, as well as meeting needs for accountability.
3.1 Introduction
Globalisation and new technology have led to changes that seem to affect every facet of life.
Agricultural extension is not immune to these changes and currently Australian extension
institutions are undergoing considerable structural and philosophical changes. Accordingly, changes
have occurred in the sorts of evaluation that are being demanded for these projects. To develop
effective approaches to project evaluation in these changing times, an appreciation must be gained
of the environment in which agricultural extension is delivered and evaluated. The aim of this
chapter is to describe the current context of agricultural extension in Australia and to identify the
implications for the practice of evaluation within this field. To achieve this, two main areas of
change are examined. The first addresses changes to the philosophy and theory underpinning
agricultural extension itself; the second incorporates changes affecting the organisations in which
39
extension is delivered.
I begin the chapter with a brief look at changes that are occurring at a global level and how these
changes are reflected in the practice of agricultural extension. I provide a description of the
development of extension models from the ‘science-push’ paradigm, to new approaches influenced
by the emergent ‘farmer-first’ paradigm. I emphasise the move towards more participatory, group-
based and process-oriented forms of extension. Following this, I examine the changes occurring in
the organisations that deliver extension. As each state of Australia has a different story to tell, I take
an in-depth look at the case of Victoria. In doing so, I focus on the impact of recent reforms in the
Agriculture Division in the Victorian Department of Natural Resources and Environment, and
specifically on the provision of agricultural extension. I make an attempt to unravel the various
reforms and trends occurring and describe the consequent challenges posed both for the practice of
agricultural extension and of project evaluation in agricultural extension.
3.2 A Changing World
Increasing globalisation, improved communications, and new technologies have resulted in growing
social, political and economic interdependence at a global level. Even small shifts within society are
felt throughout the world. Stock market jitters in Tokyo are reflected in London and New York in a
number of hours. Familiar patterns of family, community, business, politics and even nation are
breaking down. At the same time there is something else new in the air, a new emphasis on unity
and integration, talk of new politics, even a new society. The old ‘modernist’ or positivist vision of
one truth, one expression of reality, one best way of doing things seems problematic in this
globalised world. Instead, there is a need for a more pluralistic vision that can accommodate the
multiplicity and diversity of experience.
There are several features of this new global rhetoric that cross many disciplinary boundaries and
countries. From agricultural extension (Chambers et al., 1991), to quantum physics (Zohar and
Marshall, 1994), to the ‘third way politics’ (Giddens, 1998) to action research (Reason, 1988) and
to program evaluation (Guba and Lincoln, 1989). Four main elements recur in this globalised
discussion, i.e, things must be done in ways that are:
40
• Holistic – due to increasing globalisation and improved communications we must be aware of
the interconnectedness nature of interventions. There is a tendency to encourage collaboration,
partnerships, and dialogue between organisations that were previously perceived as separate.
• Pluralistic – due to increased globalisation and collaboration we must learn to live with many
points of view, many different ways of experiencing reality.
• Responsive – new systems must be flexible and less hierarchical. Ambiguity, rapid change and
greater complexity increasingly dominate events and patterns of relationship in both domestic
and public life. New family structures, new technological systems, new and shifting markets all
demand flexible responses.
• Bottom up or emergent – increasingly citizens, have grown impatient with political structures
that are implemented in a top-down manner. More citizens are being invited to join in the
decision-making processes.
This global rhetoric also appears in texts associated with the practice and philosophy of agricultural
extension, where the traditional top-down, ‘science-push’ approach to extension is largely being
replaced by extension that is described as holistic, responsive, pluralistic and bottom-up. At the
same time this global rhetoric also appears in literature associated with public sector management.
Indeed, the public sector organisations that have traditionally delivered extension have been subject
to many changes associated with a new type of global public sector management. For example, in
extension organisations in Australia there is a tendency to move toward organisations with fewer
hierarchical levels, emphasising new partnerships with the private sector, and with more farmer
representation at many levels of the organisation.
3.3 Changing Philosophy of Agricultural Extension
In this thesis, I refer largely to publicly-funded extension, while acknowledging that there is
increasing involvement of the private sector in the provision of advice and other information to
farmers. Ultimately, governments fund extension to enact policy. Because of this, extension
changes with the uses to which it is put and the historical context in which it is deployed (Röling,
1988). Changes associated with the paradigms that underpin extension are influencing how
extension is conceptualised, how it is conducted, and subsequently how it should be evaluated.
41
In the following sections, the concept of extension will be examined followed by a discussion of the
movement from a worldwide adoption of the ‘science-push’ paradigm in extension, to the
emergence the ‘farmer-first’ paradigm, and finally to the current situation in Australia where there
is no idealised model of extension in practice.
Conceptualising extension
Before exploring the changing nature of the paradigms that underpin extension, an overview of the
range of things that can be signified by the term ‘extension’ will be provided. At the heart of
extension is the contradictory notion of inducing behaviour change by voluntary means, rather than
by regulation. Behaviour change is a complex thing to enact on oneself, and even more so to enact
on others. Inducing voluntary change is generally only successful when the target person’s goals
are also satisfied in some manner. Differences in the way that extension is conceptualised tend to
reflect the tension between inducing behaviour change and satisfying client goals. As a result of
this tension, extension is a messy and ill-defined concept.
Röling (1998: 36-37) points out how in different countries extension has been conceptualised and
labelled quite distinctly. For example in French it is refereed to as vulgarisation: a simplification of
information so that ordinary people can understand; in Dutch it is referred to as voorlichting:
keeping a light in front of someone to allow him or her to find their way; in Spanish it is referred to
as capacitación: empowering someone to achieve things for themselves. Van Dissel (1986)
contends that as a result of the confusion, extension has never established an international
classification of extension publications. This helps to explain why there is no uniform definition of
extension. Zuurbier (1983; cited by Röling, 1988: 39) examined the various definitions of extension
across languages and literature and determined five common elements:
• extension is an intervention
• extension uses communication as its instrument to induce change
• extension can be effective only through voluntary change
• extension focuses on a number of different target processes and outcomes, which distinguish it
from other communication interventions
• extension is deployed by an institution.
42
These common elements provide an overarching description for extension. However, it is also
useful to understand the differences between alternative extension traditions. Röling (1988: 37)
describes four traditions, each of which are present to a lesser or greater extent in different cultures
of extension:
• Informative extension – where the emphasis is on providing information to help the individual
to make optimal decisions with respect to achieving her/his own goals.
• Emancipatory extension – where extension is seen as an instrument of emancipation of the
poor, as advocated by Friere (1972). This sort of extension is most usually carried out by
voluntary agencies who seek to assist the rural poor, often in low-income countries.
• Human resource extension – where the emphasis is on the development of human beings in
terms of enhancing her/his capability to make decisions, to learn, to manage, to communicate
with others.
• Extension as a persuasion – where the emphasis is on extension as a policy instrument for
achieving societal objectives or collective utilities.
In Australia, much of the publicly-funded extension appears to fit in with Röling’s tradition of
extension for ‘persuasion’, with export goals, national food security and environmental protection
being the driving issues. However, for farmers to be ‘persuaded’ into voluntary change, farmers’
goals must also be satisfied. As a result of this, many extension projects appear to come in various
combinations of Röling’s four traditions.
A frequently cited definition of extension is that of Van den Ban and Hawkins who define extension
as ‘the conscious use of communication of information to help people form sound opinions and
make good decisions’ (1988: 9). However, this definition provides no clues as to who decides what
constitutes ‘sound opinions and good decisions’. In publicly-funded extension, the assumption is
that ‘good decisions’ constitute those decisions that lead to behaviour change congruent with the
current government policy. The idea behind government funding for extension is to bring about
mainly public rather than private benefit. While public and private benefits often overlap, it is
possible that a publicly-funded extension activity may aim for voluntary change in the behaviour of
a farmer, which may not necessarily benefit that individual farmer at all. This illustrates the
ambiguous nature of the term ‘extension’.
Historically, the rhetoric of agricultural extension worldwide has shifted from an emphasis on
43
production, at the beginning of the century, to productivity (or efficiency) based agriculture, to the
more recent philosophy of sustainability (Guerin and Guerin, 1994). Pretty (1995) suggests that we
are currently entering an era of social capital, where farmers are now considered to be the potential
solution rather than the problem, i.e., the role of individual capacity is paramount. Through each of
these eras, adoption of relevant technologies and innovations by the farming community has been
critical to improve and maintain agricultural production and to meet changing demands (Campbell,
1980). However, the way in which adoption is brought about, and the way in which innovations are
developed has been widely contested. Two major contending paradigms are:
a) the ‘science-push’ paradigm that relates to Röling’s description of extension as persuasion
b) the ‘farmer-first’ paradigm that relates to either Röling’s description of extension as human
resource extension or informative extension.
The ‘science-push’ paradigm of transfer of technology
Under the reductionist paradigm of ‘transfer of technology’ (TOT), agricultural extension is seen as
a vehicle to transmit research findings to farmers, who then adopt the superior technology which in
theory leads to more efficient and effective production. This view of extension can be described as
‘science-push’, as research results are extended from scientists to the farmer in a linear manner.
Under this model, highly skilled and trained scientists set the research agenda and extension is
usually subservient to research. Engel and Van den Bor (1995: 2) state that this paradigm ‘led us to
believe that extension is an institution which merely channels knowledge without adding value to
it’.
A central concept behind the TOT model was Roger’s (1962) ‘diffusion of innovations’ theory,
whereby extension officers would transfer the latest research findings and technologies to
innovative farmers. The theory was that this would trickle down (diffuse) across the whole farming
community. An emphasis was placed on developing agricultural technologies on research stations,
and bringing about mass adoption of these technologies as quickly as possible to maximise
productivity gains (Coutts, 1997). Although social aspects are considered under the TOT model,
extension was mainly conceived in the context of increasing the rates of adoption. Thus, projects
governed by the TOT paradigm are traditionally evaluated against the extent to which the new
technology is adopted and sometimes the consequent change in production. Despite its prevalence,
its success is said to have been limited to homogenous groups of farmers who already have a high
44
level of access to information and resources or to homogenous groups that have been specifically
targeted (Röling and Jiggins, 1987).
The ‘science-push’ paradigm remains the basis of much research and advisory structure in Australia
(Guerin and Guerin, 1994). On a global level, the 1970s witnessed the beginning of disillusionment
with the TOT model. Much has been written on the shortfalls of the TOT and the ‘science-push’
approach (Chambers, 1983; Dillon, 1965; Vanclay and Lawrence, 1995). The central arguments are
that:
• technologies developed on controlled research stations did not diffuse across the farming
population as expected
• the benefits of the research tended to be distributed inequitably across the farming community
• the products of agricultural science were uncritically accepted by researchers as improvements
• the focus on disciplinary research was unable to provide solutions to problems at the whole
farm level, or for complex farming systems
• the emphasis on production overlooked issues of environmental and social sustainability
• the emphasis on the supremacy of scientific knowledge overlooked local or indigenous
knowledge.
Despite the fact that ‘science-push’ model is still alive in Australia (Guerin and Guerin, 1994), in
the last 30 years the farmer-first paradigm has emerged, and has led to a new family of approaches.
These new approaches include: Farming Systems Research (FSR), Farmer Participatory Research
(FPR), Participatory Technology Development (PTD), farmer-first research, Participatory Action
Research (PAR), co-learning and experiential learning. All of these approaches emphasise farmer
involvement in the identification and testing of suitable innovations or ideas to improve their
situations.
The emergence of the farmer-first paradigm
The 1970s saw the emergence of a new interpretation of the concept of non-adoption. Instead of
non-adoption being considered an issue of ‘farmer failure’, it became re-conceptualised as an issue
of the inappropriateness of recommended technologies or the priorities and processes which
generated them (Chambers et al., 1991). There is now considerable agricultural research and
development activity that was instigated on the basis of claims that farmer participation is critical to
45
the generation of technologies that are relevant to farmers (Bebbington et al., 1994; Foale, 1997).
Chambers contends that:
With farmer-first, the main objective is not to transfer known technology, but to empower farmers to
learn, adapt and do better; analysis is not by outsiders – scientist, extensionists, or NGO workers – on
their own but by farmers and by farmers assisted by outsiders; the primary location for R&D is not the
experiment station, laboratory or greenhouse…but farmers’ fields and conditions; what is transferred
by outsiders to farmers is not precepts but principles, not messages but methods, not a package of
practice to be adopted but a basket of choices from which to select (1999: 182).
In Australia similar concepts emerged during the 1960s and 70s (Williams, 1968; Bardsley, 1981).
Researchers were seen to have more of a support role, and extension had more of a facilitation role
than in the past. The shift towards the farmer-first paradigm is associated with several related
approaches to extension, three of which are presented in the following sections.
Systems based approaches to extension
In the 1970s Farming Systems Research (FSR) was developed to address some of the constraints of
the TOT model. This systems approach combines elements of research with extension. FSR
considers the whole farm rather than studying individual components, is conducted on the farm as
opposed to the research station, and ideally involves farmers in defining the research agenda. The
adoption of farming systems research at most international agricultural research institutes has had a
large impact on national projects in many countries (Anderson and Dillon, 1985). In Australia,
systems based models of research and development have been recommended or applied in some
regions (see McCown, 1991; Clark et al., 1996; Foale, 1997) although the approaches vary widely.
It has been suggested that farming systems research has made a huge contribution to the ‘farmer-
first’ paradigm by revealing the complexity of farming systems and of the decisions which face
farmers, the limitations of disciplinary research, and the recognition of farmers as innovators and
experimenters themselves (Chambers et al., 1991; Rhoades and Boothe, 1982; Petheram and Clark,
1998).
Experiential learning, co-learning approaches and action research
The Hawkesbury model of agricultural systems development (Bawden, 1991; Macadam, 1996) is
based on experiential learning (Kolb, 1984), which encapsulates the concept of learning through
experience. One way it differs from FSR is that it emphasises the human activity system, rather
than the physical/biological system (Petheram, R.J., pers. comm., 2000). Kolb (1984) defines
46
experiential learning as the process whereby knowledge is created through the transformation of
experience. The cycle of experiential learning is based around a cycle of concrete experience,
reflective observation, abstract conceptualisation and active experimentation.
Participatory Action Research (PAR) or Action Research is similar in many ways to experiential
learning models. PAR, as its name implies, pursues both action and research outcomes. It is most
commonly done with high levels of participation of farmers and agency. Reason (1988) suggests
that PAR is probably the most common collaborative research approach. Its purpose is to bring
about planned change in such a way that there is also a contribution to understanding of the system
which is being changed, and the methods which are used. Hart and Bond (1995) selected seven
criteria to distinguish action research from other research methodologies. The criteria are that action
research:
• is educative
• deals with individuals as members of social groups
• is problem focused, context-specific and future-oriented
• involves a change intervention
• aims at improvement and involvement
• involves a cyclical process in which research, action and evaluation are interlinked
• is founded on a research relationship in which those involved are participants in the change
process.
These criteria could equally be applied to experiential learning, another co-learning model that has
been applied in Australian contexts. Indeed Roberts (1997) suggests that action research may be
viewed as connected cycles of experiential learning and that action research and experiential
learning may be interpreted as a single phenomenon described from different perspectives. All these
experiential approaches have reflection and evaluation phases built into their cycles. Experiential
approaches produce context-specific outcomes, which will differ greatly from one project to the
next.
47
Group-based approaches to extension
In Australia, particularly with the advent of Landcare, there has been an increasing use of group-
based approaches to extension. Before the 1980s most extension was either done on a one-to-one
basis or at field days, or through mass media. Coutts states that:
From my perspective the appointment of Landcare and Catchment facilitators was the greatest single
challenge to extension officers operating in the traditional mode of one-on-one advice. These
facilitators came in without the strong technical background of extension officers – and without the
baggage of associated with the extension culture. They demonstrated a new paradigm in working with
groups of producers (Coutts 1997: 25).
Community-based groups could well form the basis for a trend towards research, extension, and
advisory systems that are bottom-up or (farmer-first), rather than linear (Campbell and Junor, 1992).
The idea of giving legitimacy to local knowledge and of working with groups, demands new skills
in those involved in extension, and in those evaluating the effort.
Multiple approaches to extension
In the year 2000, there is no one clearly idealised model of extension being practised in Australia.
Coutts (1997) describes extension as slipping into a range of purposeful and complementary
models, shaped by political and funding realities, rural and societal needs, and a greater
understanding of communication, facilitation and adult learning. Vanclay (1994) suggests that
Australian public extension is facing a crisis of effectiveness and theory, with extension practices
not working, farmers failing to adopt many recommended practices, and agencies rejecting the
traditional extension model without a cohesive, coherent or widely accepted alternative. Whether
viewed in a positive or negative light, the above literature indicates that there is no single theory
currently underpinning current practice.
While acknowledging many exceptions, the late 1980s to 1990s saw the emergence of two different
approaches with regards to planning and managing agricultural extension projects in Australia. On
the one hand, some projects became increasingly targeted towards productivity, aiming to achieve
pre-stipulated export targets. Projects were given names reflecting the export targets that had been
stipulated at the state level, i.e., ‘Target 10’, ‘Farm Management 500’, and ‘Target 250’. On the
other hand, a movement emerged calling for the need to embrace the concept of ‘learner focused’
48
extension. This movement embraces many of the concepts that underpin the farmer-first paradigm
and the co-learning and experiential approaches. These projects such as ‘Farm$mart’, ‘Topcrop’
and ‘Bestwool’, aim to help the farmers to help themselves. Projects are often organised around
farmer groups that meet regularly, where participants identify critical issues for learning and action.
While these approaches to planning sound contradictory, they often sit side by side within the same
project. For example, the Victorian Target 10 Dairy Extension Project is strongly outcome-focused
and has an explicit target to increase the production of milk. However this project is currently
attempting to move towards a more learner-focused approach, through discussion groups and wide
representation of farmers of steering committees and in project redevelopment (Boomsma, J. pers.
comm., 2000).
The implications for evaluating new approaches to extension
Changes in the philosophy of extension have seen the emergence of new approaches to extension.
The following characteristics of these new projects have practical implications for evaluation, and
will be discussed in the next section. These are:
• increasing participation of farmers in the evaluation of extension activities
• increasing process-orientation
• increasing diversity in outcomes
• increasing complexity in the scope of projects
• increasing number of organisations working in extension.
Increasing participation of farmers in the evaluation of extension activities
Worldwide, development and extension projects are becoming more participatory in focus. The
past two decades have seen an increased recognition of the importance of participation by
beneficiaries (and a wide range of other stakeholders) in decision making. There has also been
increasingly wide and deep discussion about the theory of participation appropriate levels of
participation (Pretty et al., 1995). For extension projects operating at either end of the participation
continuum, increasing calls for accountability, efficiency and improved management performance
have led to a more intense focus on monitoring and evaluation. But in participatory projects three
additional questions emerge:
49
• whose reality counts?
• who decides what is measured in the evaluation?
• who should be making the evaluative judgements?
One of the major challenges facing the field of international development in the 1990s has been
how to measure the impact of participatory projects in a manner that is congruent with the
philosophy of these projects (Oakley et al., 1998). The over-riding concern is for the process of
evaluation to reinforce, rather than inhibit, participation and empowerment of beneficiaries.
Clearly, external evaluation, based on outsider values as to what constitutes success, would not be
appropriate. In short, participatory projects require participatory evaluation; approaches that allow
the stakeholders and beneficiaries to have a say in which changes are important, and which changes
should be measured. Patton, writes that participatory evaluation:
…is most appropriate where the goals of the project include helping participants become more self-
sufficient and personally effective. In such instances...evaluation is also intervention orientated in that
the evaluation is designed and implemented to support and enhance the program’s desired outcomes
(1997: 101).
Evaluation that can meet these requirements is referred to as ‘participative’ or ‘participatory
evaluation’, or ‘participatory monitoring and evaluation’ (PM&E), or in the US, ‘empowerment
evaluation’. In this thesis this genre of evaluation is referred to as participatory evaluation.
Participatory evaluation promotes the involvement of a wide range of stakeholders, employing
methods that encourage an equal expression of views and sharing of lessons.
Increasing process-orientation
The idea of process-orientation in extension projects was first seen 20 years ago in the field of
overseas development projects. Farrington and Nelson (1997) suggest that in agricultural
development projects in lower income countries, there is an increasing tendency towards
participatory projects with a ‘process-orientation’. The term ‘process-orientation’ in this context is
used to describe projects that do not have rigidly defined goals at the onset but have a defined
purpose and a set of potential goals. The concept of process-orientated projects emerged while
attempting to encourage ‘bottom-up’ projects. This was problematic when predicted outcomes and
activities were pre-stipulated in a ‘top-down’ manner. Process-projects are a natural evolution of
the concept of beneficiary participation in project planning and decision-making and defining
intended outcomes. If beneficiaries are to identify their development needs as part of the project
50
process, then it makes sense to have a project design that does not have predetermined outcomes.
The Landcare movement in Australia, (although it developed independently from development
studies literature) exhibits properties of process-oriented projects.
Farrington and Nelson (1997) suggest that projects with a process-orientation ‘are difficult to
evaluate by traditional means as they are carried out in variable, unpredictable situations; they
produce outputs that are hard to measure objectively and often have permeable boundaries and less-
than-direct relationships between inputs and outputs’. Evaluation of this type of project is best
achieved though inductive, participatory methods, where dialogue and exploration are encouraged.
Objectives-based evaluation is unlikely to capture the variable impacts of projects; instead
exploratory, participatory, needs-based evaluation is more appropriate.
Increasing diversity in outcomes
Another consequence of the new paradigm approaches to extension is recongition of diversity.
Farmer-first approaches tend to be contextual in focus and are not designed to produce generalisable
results. Farmers are encouraged to seek solutions to their location-specific problems. As a result
there is often a great variety in terms of the experiments conducted by the groups. These projects
tend to produce diverse, context-specific outcomes, which are difficult to aggregate in order to
provide information to evaluate the project as a whole.
Conventionally, planning and evaluation of simple extension projects have been managed through
the use of simple project models in which planned inputs lead logically to predictable outputs or
outcomes. When project models become more complex and flexible it becomes imperative to
expend effort in investigating the unintended or unanticipated outcomes. When evaluating complex,
diverse projects, it becomes critical to positively engage with unpredicted or unanticipated impacts
and diversity, rather than just an extra task to tack to the end of an evaluation study.
Increasing complexity in the scope of projects
The information/knowledge system (IKS) of agricultural extension in Australia is becoming
increasingly complex. The shift in focus of extension from production orientation to a holistic
view, including: sustainability, improved marketing, increasing attention to the management of the
farm business and maintaining profitability under declining terms of trade, adds to this complexity.
51
Increasing number of organisations working in extension
There are more actors operating in the capacity of knowledge bearers to farmers than in the past.
The implication of this increasing complexity of the IKS on evaluation is that it is becoming more
difficult to distinguish the impact of one project or actor from that of another (Coutts, 1997).
Evaluation studies need to take into consideration the implication of the integrated impact of the
workings of the IKS on the rural sector.
Summary of implications of evaluating new paradigm approaches to extension
The evaluation of participatory and process-orientated projects must take into account that the
outputs of such projects tend to be diverse, unpredicted and context-specific. Conventional,
objectives-based approaches are inadequate to capture the range of unpredicted and unanticipated
outcomes that these projects produce. In addition, external evaluation against pre-defined indicators
of success would be incongruent with the philosophy of these new paradigm approaches.
Nonetheless, these projects experience the same demands for accountability and demonstration of
impact as all publicly-funded extension projects.
Allen (1998: 629) contends that because participatory projects are designed to be responsive to
changing community needs, one of the most pressing challenges is to develop ‘participatory and
systems-based evaluative processes to allow for ongoing learning, correction and adjustment by all
parties concerned’. Alternative methods are needed for evaluating participatory, group-oriented
approaches to extension; as participatory projects require participatory evaluation. If evaluation
approaches do not exist within the current repertoire of extension evaluation tools, they need to be
developed or adapted from other disciplines. These methods need to be participatory, creative,
formative and empowering. Chapter 4 reviews the current practice of evaluation in extension to
determine whether any of the approaches already being practised in this field, meet such criteria.
52
3.4 Organisational Change and Extension Evaluation
Historical changes in the practice of evaluating extension in Australia
As discussed in Section 2.2, agricultural extension was one of the first disciplines to conduct
program evaluation. As early as 1947, some major projects in Australian agricultural extension
were evaluated as part of their planning and funding process (Sharp and Lindsay, 1992). Over time,
the purpose and methods of extension evaluation have been strongly influenced by the
organisational environment in which farmers and extension agencies have had to operate. A key
element in this environment has been government policy. Changes in federal Government policy
and corresponding changes in evaluation practice can be loosely characterised by decades, although
it must be acknowledged that these changes occurred somewhat differently in different states of
Australia, each with unique state elected governments.
In the 1960s and early 1970s agricultural extension was seen as a mechanism to solve farmers’
problems, and with this view evaluation was used as a tool to provide feedback, to improve the
services to farmers. The approach characterised the optimistic nature of this period. Evaluation was
much discussed at extension conferences, and national workshops were conducted to focus attention
on taking a more professional approach to evaluating the efficiency and effectiveness of extension
(Hynes, 1963). Issues such as how to measure behavioural change and the need for open-ended
questions were addressed (Riches, 1973). Bardsley (pers. comm., 1997) recalls that in the optimism
of the period, extension workers felt valued by their farmer clients.
Bardsley (pers. comm., 1997) suggested that during the 1970s under the Whitlam Government,
extension became progressively more of a political tool used to bring about policy changes desired
by government, than previously. There was also a growing need to justify the spending of public
funds. With this change in perspective towards agricultural extension came a need for evaluation
methods to be more centred around accountability. Much of the evaluation carried out at this stage
was based upon assessing the achievement of institutional goals or the achievement of objectives of
individual extension officers (Sri Pathmanathan, 1978). Little attention was paid to whether the
clients’ needs were being met by the project (Bardsley, 1981). To an extent, agricultural extension
became more of an arm of government policy and the resulting projects were characterised by a
53
focus towards national interest, which was sometimes at odds with individual farmer interest.
In the 1980s governments began to question whether agricultural extension should be publicly-
funded and thus emerged the principle of user pays. At the Australian Agricultural Extension
Conference in Brisbane in 1987, the question of funding government extension services received
considerable attention. The majority of the discussion concerned the application of ‘user pays’ as
this was seen as the most effective way to overcome the decline in extension funding. With this
shift towards a decrease in public spending in agricultural extension, came a greater demand for
evaluation of both publicly and privately-funded projects. Private enterprises wanted to make sure
that projects, which were funded with private money, were effective and accountable. Thus began
the trend for evaluation to be used as a tool for justification and accountability of spending.
In the 1990s, for some states of Australia, the funding for extension appeared to stabilise, and the
prospect of a totally user-pays system seemed to fade (Coutts, 1997). However, a user-pays system
was actually implemented for the provision of agricultural extension in Tasmania for a period,
although it is now reported that some aspects of extension are being recognised as public benefit
and are attracting public funding (Coutts, 1997).
The 1990s was increasingly influenced by changes that were occurring at a global level and referred
to as ‘New Public Management’ (Hood, 1991). These reforms included the introduction of the
purchaser-provider model for publicly-funded agricultural extension in many states of Australia
(Marsh and Pannell, 1997). This is one of a plethora of recent political reforms that have been
introduced in a more general attempt to mimic free market conditions. These reforms, which are a
worldwide phenomenon, pose considerable challenges for the evaluation of extension projects and
projects. The reforms affect both the product of extension and the expectations of how this product
should be evaluated and these are discussed in more detail in the next section.
Reforms and new public management
Much of the literature concerning New Public Management (NPM) is from other sectors and other
countries, and does not specifically refer to agricultural extension. However, as I will describe, the
literature of NPM adequately addresses the changes occurring for some of organisations that
provide extension in Australia – changes that are part of a global phenomenon. The last 10 years
have seen major changes to the way public sectors are managed in much of the Western world.
54
These changes have variously been referred to as ‘New Public Management’ (Hood 1991) ‘New
Managerialism’ (Pollitt, 1997) and ‘Contractualism’ (Alford and O’Neill, 1994). These reforms
have involved creating a more ‘business-like management’ of the public sector (Lane, 1997).
Martin and Sanderson (1999) suggest that NPM has typically consisted of:
…a cocktail of cost reductions, devolution of management responsibilities, ‘neo-taylorist’ performance
management system, new market and quasi-market mechanisms, greater ‘customer focus’ and new
service standards. Taken together these reforms represent a decisive shift away from management
based on tight ex-ante control of inputs, towards a framework which devolves responsibility for service
delivery to front-line staff within a system of ‘continuous monitored management by objectives with
accountability for results (ibid: 246).
O’Faircheallaigh et al. (1999: 4) suggest that under this New Public Management, governments of
OECD countries are calling for less ‘red tape’, more emphasis on results, greater efficiency,
reduced staff and user-charging for services in public agencies. Governments are increasingly
asking whether they should remain involved in or withdraw from the activities that they presently
undertake. Rivlin (1996, cited by O’Faircheallaigh et al., 1999: 1) suggest that these new public
management models attempt to deal with problems arising from:
• globalisation – global pressures to cooperate and compete in new ways
• dissatisfaction – ever rising expectations of citizens
• budget stringency – the need to curtail expenditure and reduce deficits.
O’Faircheallaigh et al. (1999) provide a useful framework for unravelling and describing the
plethora of changes that are occurring under New Public Management worldwide. They consider
the reforms in terms of the disciplines that influenced them, namely: economics, organisational
theory and management, political science, and marketing. In the following sections
O’Faircheallaigh et al.’s framework is summarised, then applied to the particular case of
agricultural extension in Victoria, in attempt to understand the emerging implications for program
evaluation in this sector.
Influence of economics on public sector management
O’Faircheallaigh et. al. (1999: 36) explain how economics has helped shape many of the tenets of
NPM. Economics, as practised in many western countries, has long been dominated by the
assumption that the market represents the most efficient mechanism for allocating economic
55
resources in society. Economics does not assume a role for government as a given. Economists see
the fundamental roles of government in as allowing market mechanisms to operate. Thus,
economics explains or justifies the role of government in terms of ‘market failure’(Freebairn, 1989).
O’Faircheallaigh et al. (1999), state that economic rationalists are particularly concerned with:
• technical efficiency – the relationship between the value of resources consumed in a specific
economic activity (inputs) and the value of resources that are produced as a result of that
activity (outputs) [Benefit-cost analysis]
• allocative efficiency – involves the degree to which the overall allocation of resources between
alternative, competing uses maximises total economic output.
Pusey (1996) points out that the theory of economic rationalism has had growing influence on
policy decisions in many liberal democracies, and a substantial impact on the policies of
governments towards their public sectors.
Influence of organisational theory and management on public sector
management
O’ Faircheallaigh et al. (1999: 43) point out that ‘organisational theory and management’ represents
not so much an established discipline but rather an approach that suggests that the design and
management of organisations is a subject that deserves separate and sustained study in its own
right.’ They go on to explain that its basic assumption is that that all organisations have features in
common, and that a systematic study of these features and issues can allow general conclusions
about the utility of particular organisational forms and processes in certain contexts. In this area of
study O’Faircheallaigh et al. (1999: 44) point out three major persistent themes:
• The need to re-structure organisations. Key issues include: the numbers of layers in an
organisational hierarchy, and where to locate and concentrate authority for decision-making and
how to allocate tasks in relation to organisational structure.
• The question of how large, complex organisations can be directed to achieve goals set by
relevant decision-makers.
• The promotion of strategic alliances and other collaborative mechanisms designed to exploit
complementarities and to achieve efficiencies through agreed divisions of labour (ibid).
56
Influence of political science on public sector management
O’Faircheallaigh et al. (1999: 39) state that political science (or politics) is concerned with the
‘good society’ and the allocation of resources. Public policy is concerned with ‘who gets what,
when and how’ (Lasswell, 1936). O’Faircheallaigh et al. explain how any change to the way in
which the public sector functions will be assessed, in part, on the basis of its impact on the capacity
of citizens to enjoy and exercise rights. Thus the relationship between inputs and outputs (technical
efficiency) is not the only criterion for gauging the desirability of public sector change. In modern
society it is impossible for the vast majority of citizens to participate directly in collective decision-
making. Thus, a critical issue becomes the mechanisms through which those who do participate
directly (i.e, elected politicians) are held accountable for their actions (O’Faircheallaigh et al.,
1999).
O’Faircheallaigh et al. (1999: 43) state that political scientists are likely to view the ‘performance’
of the public sector in ways that are both complex and not easily amenable to quantification.
Assessments of performance are likely to involve trade-offs. Efficiency is important, but so is
respect for citizens’ rights and the promotion of an effective accountability mechanism
(O’Faircheallaigh et al., 1999: 43).
Influences of marketing on public sector management
O’Faircheallaigh et al. (1999: 46) describes marketing as an activity that ‘has traditionally been
associated with the distribution and promotion of commodities, and more recently of services, in the
private sector’. They suggest that the application of a marketing approach to the public sector
involves significant ambiguity. In the private sector consumers exercise their power by their choice
to purchase or not, and thus their power is contingent on consumer capacity to pay rather than on
the intrinsic needs or rights of citizens to basic services. Thus arises the danger of allocation of
resources based on capacity to pay. On the other hand, application of a marketing perspective could
be seen as a way of reasserting the primacy of client interests as opposed to bureaucratic or
professional interests (O’Faircheallaigh et al., 1999).
57
3.5 A Case Study of the Agriculture Division in the
Victorian Department of Natural Resources and
Environment
As each state of Australia has a unique history of public sector management and provision of
extension, a case study of the reforms in Victoria is provided, rather than attempting to generalise.
This includes a brief examination of the State government policy, followed by a more in-depth
analysis of the new public management (NPM) reforms on the Agriculture Division of the Victorian
Department of Natural Resources and Environment (NRE). The implications these reforms pose to
the practice of program evaluation are also considered.
The field work for this thesis was conducted from 1997-1999, under a Liberal state government.
Therefore the emphasis will be on reforms in NPM that were largely brought about between 1992
and 1998 when Victoria was under this Liberal state Government, with Jeff Kennett being the state
Premier. The most comprehensive statement of the Kennett model was spelt out in the 1993
Management Improvement Initiative 1993 (cited by Alford and O’Neill, 1994: 4):
• Focus on clear responsibility and accountability for results – ie, emphasis on outputs and
outcomes
• Empowering consumers – fostering consumer choice
• Minimising government bureaucracy – focusing departments of core functions
• Preference for market mechanisms – privatisation, contracting out, and competition policy
• Professional and business-like management of public agencies.
This statement is very much in line with the rhetoric of New Public Management that appears to be
influencing public sector management in most OECD countries.
The following section focuses on the impact of new public management reforms on one government
department: the Agriculture Division, the Victorian Department of Natural Resources and
Environment (NRE), and specifically on the provision of agricultural extension. The Agriculture
Division is the part of the Department of NRE that is responsible for service delivery to agricultural
industries and includes the state’s purchasers and providers. The range of services includes
extension, research and development, policy, regulation and trade development.
58
Influences of economics on the Agriculture Division
In July 1996 the purchaser-provider model was introduced for publicly-funded agricultural
extension in Victoria. Consequently, for the latter part of 1990s, the language used became that of
purchasers and providers, contracts and tenders. The major issues that related to the move towards
the semi-privatisation of the provision of public-sector services included the increasing pressure on
funding and the purchaser-provider split.
Increasing pressure to justify funding
Watson et al. (1992), in a Review of Field Based Service, recommended that extension in the
Victorian Department of Natural Resources and Environment should concentrate on areas of market
failure where the private sector was unwilling or unable to provide services to farmers. This
represents a departure from the worldwide tradition of governments taking the financial
responsibility for providing agricultural extension services. The prime reasons were to increase
agricultural production or to improve the efficiency of agricultural production (Lees, 1990). Both
reasons were concerned with increasing output to provide more food, fuel, and fibre to meet
growing domestic or export demand. A Victorian state departmental corporate plan gave the
overall rationale for the continuation of government extension services as to support increased
income from primary production within sustainable agricultural systems (Watson et al., 1992).
However, the relative importance of agriculture in the economic growth of industrialised countries
has markedly declined (Blank, 1999). A system of user pays is frequently held up as the most
effective way to overcome the decline in extension budgets. This change to a user-pays system has
already occurred for the provision of agricultural extension in Tasmania and New Zealand (Marsh
and Pannell, 1997). However, in Victoria the prospect of moving to a user-pays system has
somewhat diminished (McDonald, B. pers. comm., 2000). Nevertheless, public funding for
agricultural extension is frequently questioned and projects must meet increasingly high standards
of accountability and quality to prove their worth. Well-designed evaluation strategies that include
some formative component of evaluation are seen as an essential basis for planning, implementing
and improving high quality agricultural extension projects, and hence assuring public benefits from
research and development expenditure. The pressure is also increasing to prove that they have
produced outcomes that deliver public benefits. Thus, both formative and summative evaluation of
extension projects is critical to justify continued public funding.
59
The purchaser-provider split
In Victoria a version of the purchaser-provider model was introduced in July 1997 for government-
provided agricultural extension. With the separated roles of purchaser and provider, the purchasers
specify the services required, and assess their provision by the providers. In simplistic terms, the
purchaser role is responsible for investing resources wisely, and demonstrating the value of the
investment. The provider role is to provide the service contracted and to assist with demonstrating
the value of the service. In Victoria, the purchaser-provider model has not been implemented as
‘purely’ as in other states such as Western Australia. In Victoria, certain providers are nominated as
‘preferred providers’ and this includes the providers internal to NRE (Shaw, H. pers. comm., 2000).
Elsewhere, purchasers have more freedom to purchase across a wider variety of providers.
According to internal documents (NRE, 1997), the purchaser group in NRE aims to increase
impact, decrease overhead costs and to improve evaluation, among other things. The purchaser
group also has a strong outcomes focus. These outcomes are broadly defined at the outset of a
project, then later in the project development cycle they are defined more specifically, i.e.; to be
specific, measurable, accountable, realistic and timebound (SMART). The projects encouraged by
this group are to be focused on the achievement of outcomes and should have a built-in evaluation
strategy. To obtain tenders, providers will have to move towards projects with a built-in evaluation
component (NRE, 1997). It seems the development of evaluation plans conducted largely by
internal evaluators is being favoured.
The purchasers are expected to define outcomes for industry. From 1997 onwards project proposals
without clearly defined outcomes and built-in evaluation strategies are less likely to obtain funding
(McDonald, B. pers, comm., 1999). In addition, evaluation information is increasingly being seen as
important in terms of communication between purchasers and providers (McDonald, B. pers.
comm., 1999). Under the new model, purchasers and providers no longer work alongside each
other in a manager-employee relationship. Added to this is an increased number of investors and
co-deliverers, which means that projects have become more complex to manage.
Another interesting dilemma arises out of this new relationship. While the purchasers may not be
actively involved in delivering services, the farmer-clients will blame the government (and it is the
purchasers who receive this feedback) if services are not considered up to scratch. Thus evaluation
and accountability become important in the relationship between the purchasers and the providers.
60
Influence of organisational theory and management on the Agriculture Division
The Agriculture Division in Victoria has had up to 2,500 projects operating at one time. In an
internal report in 1997, a strategic objective was to reduce this to a portfolio of 50 projects by 2001.
Since then, many of the smaller projects have been amalgamated and internal documents state that
funding will only being given on the proviso that the new projects feature an investment of greater
than $1m pa and have quantified, timebound outcomes upon which all the project work is focussed
(NRE, 1997). This move to large projects is intended to increase administrative efficiency and to
focus more effort on achieving fewer, but more substantial outcomes. This is accompanied by a
change in the organisational level at which evaluation is being conducted. Previously, much
evaluation was done at the portfolio level and economic evaluation and performance indicators
were applied across an entire industry. Lately, with the move to smaller government, evaluation
responsibilities have been partially devolved to those who manage these mega-projects.
Promotion of strategic alliances
One of the Agriculture Division’s strategic objectives is to double the level of non-Victorian
Government investment in Primary Industries brokered projects (NRE, 1997). The idea of this is to
promote strategic alliances with other investors, such as the Rural Industry Research Corporations
(RIRCs) and other purchasers of NRE services, interstate agricultural departments, universities,
CSIRO, and other Commonwealth agencies, and private sector investors. To improve their ability
to create strategic alliances, the purchasers involve all the co-investors in all stages of the project
management (NRE, 1997).
Promotion of evaluation as organisational learning
In addition to its accountability purposes, evaluation under the new reforms, is also expected to
contribute to continuous improvement of extension activities (NRE, 1997). Rist (1997) suggests that
with the move to smaller government, public sector managers are expected to learn from their
mistakes and design ways of constantly improving their performance. Public sector managers are
now expected to be flexible, responsive, cost effective and innovative. It seems that the devolution
of evaluation to the project level is expected to result in increased utilisation of evaluation findings,
i.e., that projects will use the findings to improve their practice.
61
Implications of restructure for extension evaluation
As a result of the restructuring and increased collaboration, the new organisational structure under
which the Agriculture Division’s extension projects operate is complex. These mega-projects may
have several purchasers, influencers, and co-providers. For example, the Target 10 Dairy Extension
Project has four funding organisations, five investment managers and four providing organisations.
Added to this, other key stakeholders (farmer groups, environmental groups and industry bodies)
have voice, or representation in all the categories (McDonald, B. pers. comm., 1999). The various
organisations involved have distinct organisational values and cultures; as exemplified by the
disparate cultures of private agribusiness, university, and the public sector. The diversity of the
organisations the investors represent (not to mention the personal values any additional investors
bring), means that there is also a great range of values to manage while evaluating these mega-
projects. For example, the views of a farmers’ union representative on a good project outcome may
be quite different from those of a public sector investor.
In addition to diversity in terms of investor expectations of project outcomes and impacts, the
various purchasers may also have different evaluation information requirements. There are also a
whole range of other stakeholders, such as the farmers themselves, whose needs may need to be
addressed in an evaluation. Toulemonde et al. (1998) suggest that managing diverse values of
stakeholders in complex projects can be problematic. They point out that ‘partners that are
politically independent, have legitimately differing objectives, and are not accountable in the same
way to citizen partnerships, tend to complicate evaluation work at each stage’ (ibid: 171).
Taking this organisational complexity into account, it is vitally important that time is allocated for
the various stakeholders to enter into a meaningful dialogue about what is happening in the field,
and whether these experiences represent the sort of outcomes that are desirable. Project managers
are nowadays expected to provide evidence that they are capable of reflective practice and
managing organisations in a context of rapid change.
Influence of political science on the Agriculture Division
Internal documents of the Agriculture Division in Victoria bear witness to that fact that it is not
considered sufficient to produce only a cost-benefit analysis in evaluating projects and there are
frequent calls for identifying ‘who gets what when and how’ (NRE, 1997). There is also an
emerging disquiet about the limitations with the current performance measurement systems and
62
their capacity to deliver useful information (Winston, 1997; McDonald and Kefford, 1998). The
result of this is that purchasers are demanding that the new project proposals include evaluation
strategies that accurately ascertain:
• The return on investment in programs and projects (economic, tangible, and intangible)
• the requirements of customers and how well these have been delivered
• the practices and procedures for continuous improvement in project delivery.
Implications of politics for extension evaluation
Evaluation strategies will need to incorporate ways of determining whether both tangible and
intangible outcomes have been achieved, as well as describing the array of unexpected impacts
caused by the project interventions. These requirements have led to a move away from reliance on
economic analysis and simplistic performance indicators, towards the use of mixed methods
(including qualitative approaches) to capture the array of impacts, some of which may not be
quantifiable.
Marketing influences on extension
The Agriculture Division is moving towards a philosophy of market-driven service provision. They
aim to respond to the expressed needs of their client market and have set up formal links with
industry in an attempt to ensure that industry needs are met. These partnerships have a direct input
onto the strategic planning process used to assess industry priorities. As mentioned earlier, state
funding of extension projects is contingent on co-funding from the respective industry body. For
example, the Target 10 Dairy Extension Project receives 60% of funding from NRE and 40% from
the Dairy Research and Development Co-operation, which is an industry body (funded by levies on
milk).
Implications of marketing for extension evaluation
Under the new environment of extension, evaluation needs to provide sound information about the
customers – both co-investors and service users, their respective requirements and how well these
are met. This means there is a requirement for needs analysis, market segmentation and customer
satisfaction assessment (McDonald and Kefford, 1998).
63
Summary of changes in the Agriculture Division
The new public management reforms in the Agriculture Division, NRE, Vicotira, are summarised as
follows.
a) The appropriateness of continued public funding for extension was questioned extensively in the
late 1980s and early 1990s. Despite the fact that funding has stablilised recently, there is constant
pressure to justify public funds for extension. Evaluation is being seen as a way to improve the
efficiency of extension and to verify that it is returning benefits from government expenditure on
research and development.
b) With the purchaser-provider split, accountability and communication of evaluation findings
between the purchaser and providers has become more important. As a result of restructuring and
increased collaboration, extension projects have become bigger and more complex in structure. The
various organisations involved have distinct organisational values and cultures. There is diversity
in terms of investor expectations of project outcomes and impacts and the various stakeholders have
different evaluation information requirements. Evaluation approaches are needed which can deal
with complexity and value pluralism.
c) The organisational level at which evaluation is being conducted and directed has changed.
Previously evaluation was mainly economic and performance indicator-based, and was done at the
portfolio level. Other sporadic non-economic evaluation was directed from the central portfolio
level, which consisted largely of ‘expert review’ (where the views of key stakeholders or people of
influence were sought) (McDonald, B. pers. comm., 1999). In the late 1990s a trend began for non-
economic evaluation to be planned and directed by those who manage the mega-projects. The trend
is towards internal evaluation, with facilitation by evaluation experts, and some contracting out.
d) At the project level, there is demand for formative evaluation to enable organisational learning.
The intention is to enable projects to be flexible, responsive, and cost effective. At the project level,
summative evaluation is being demanded, to provide evidence of project outcomes and impact.
Evaluators are being asked to determine whether both tangible and intangible outcomes have been
achieved, describe the array of unexpected impacts caused by interventions and determine the
extent to which the farmer-clients perceive that needs have been addressed.
64
Contradictions in policies
New Public Management reforms have been influenced by economics, political science,
organisational theory and management and marketing. These reforms have occurred widely in The
Agriculture Division. Considering these recent changes together there are a number of striking
contradictions in policy, including:
• The Competition Policy (to increase public sector efficiency along private sector lines), versus
the policy of encouraging collaboration (with the move to smaller government, the public sector
providers are being encouraged to collaborate with industry to achieve outcomes).
• The policy of emphasising specific, measurable, timebound outcomes versus the call for
evaluation that captures intangible outcomes, and to capture change that cannot be quantified.
• Policies that aim to benefit the economy, rather than the individual farmer, yet a heavy focus on
(individual) farmer satisfaction.
• Delegation of delivery to providers, yet public accountability for impact resides with the
purchasers.
Given these contradictions, I suggest that the new model of public-sector management, as applied in
the Department of NRE in Victoria, has not yet stabilised. There appear to be serious limitations in
the application of the private sector management approaches, that are strongly reliant on the market
mechanisms, for public sector management. Pollitt (1997) suggests that most of the reforms are
more a matter of faith than of carefully evaluated policy and proven achievement. Project
evaluation seems to be wedged in the midst of all this chaos, and is increasingly being looked to as
the vehicle to solve problems created through changes in policy; especially concerning the problems
that complex project structures and new organisational relationships pose. Rist (1997) suggests that
the expectations for what evaluation can accomplish are growing, almost in direct proportion to the
pressures being felt by those in the public sector.
65
3.6 Conclusion
In Australia, since the introduction of the NPM reforms, the demand for evaluation appears to have
increased both in the extent of evaluation activity and in the diversity of the forms of evaluation.
Overall, evaluators are being asked to capture more information, for a larger audience. Since the
purchaser-provider split, purchasers are demanding more information to ensure that they are
investing wisely, as they are accountable both to the state and to the farmer-clients for achieving
project outcomes. With the move to fewer, larger projects with high levels of collaboration there are
more audiences for the evaluations of these projects than in the past. These audiences have an
increasingly diverse range of expectations in terms of evaluation findings. This means that
evaluation strategies will have to be negotiated carefully with the relevant stakeholders.
Despite the fact that the task of evaluation is becoming increasingly complex, in practice it seems
that a greater emphasis is being placed on developing an internal evaluation capacity amongst staff
at the project level. This internally driven evaluation is intended to satisfy accountability
requirements, to promote organisational learning and to provide credible information for
communication between the now separated purchasers and providers.
There are significant changes in the philosophy that underpins the conduct of extension. Evaluating
participatory and process-orientated projects is complex as the outputs of such projects tend to be
diverse, unpredictable and context-specific. Full participation in extension projects implies that
participants have control over intended outcomes, not just the process of achieving these outcomes.
Conventional, objectives-based approaches are inadequate to capture the range of unpredicted and
unanticipated outcomes that these projects produce. External evaluation against pre-defined
indicators of success would be incongruent with the philosophy of these new paradigm approaches.
To evaluate new, participatory, group-oriented approaches to extension, evaluation is required that
is participatory, process focused, inductive and largely qualitative.
I suggest that evaluation can be tailored to meet some of the challenges posed by new public
management reforms and by the changes in the culture of agricultural extension itself. A way
forward could involve a combination of formative evaluation, and summative evaluation. The
formative evaluation should be participatory yet facilitated by skilled evaluators and aimed at
organisational learning and helping communication between the array of stakeholders, including the
66
purchasers. It could be focused more on process than outcomes. One way of doing this would be
through a participatory monitoring system (such as is practised in the development projects in lower
income countries), in order to allow continuous learning, and involvement of various stakeholders.
I suggest that the summative evaluation component be designed to meet the need for credible
evaluation with high face validity to satisfy accountability purposes. This summative component
should go beyond simply asking whether a project works or not, and also determine for whom it
works in what situation and why (some of the theory-driven approaches as described in Chapter 2
could be appropriate). In the following chapter an examination of current evaluation practice in
extension will be presented in order to determine whether approaches to extension evaluation exist
that can meet these needs.
67
CHAPTER 4
REVIEW AND DESCRIPTION OF CURRENT PRACTICE OF
EVALUATION IN AGRICULTURAL EXTENSION
In this chapter the findings of a review of evaluation in agricultural extension in Australia are
presented. Fifty evaluation documents were categorised and analysed and using Owen’s
(1993) meta-model that is based on distinct purposes of evaluation. Nine case studies are
presented to illustrate the range of forms and approaches in practice. Where possible, the
evaluation studies were examined against the meta-evaluation criteria layed out in Chapter 2.
I found that there is very little monitoring of projects, and a tendency to use a handful of
familiar methods and approaches. Most of the studies were limited to a summative
assessment of project impact, and there was a pre-dominance of objectives-testing
approaches. I argue that while there is a range of forms and approaches to evaluation in
practice, the current suite of approaches is inadequate to meet all demands for the evaluation
of extension in Australia. In particular, no evaluation study was found that facilitated on-
going communication between multiple project stakeholders.
4.1 Introduction
I have argued that approaches to evaluation should meet certain premises for good evaluation and
be grounded in knowledge of context-specific evaluation needs (Chapter 2). An analysis of the
changing nature of extension organisations revealed an increasing demand for evaluation that can
deal with participatory, group-based extension models, provide accountability and promote
communication and learning between multiple stakeholders (Chapter 3). I suggested the evaluation
needs of large extension projects are varied and can only be met through a ‘basket of choices’ of
approaches to evaluation. Even for one project, an evaluation bricolage consisting of a number of
approaches and methods may be required to meet all the evaluation demands. In order to address
some of the challenges facing extension evaluation, this thesis aims to develop a model of
68
evaluation that can complement the current range of evaluation praxis. Therefore, before selecting
an evaluation model to adapt and pilot, an understanding of the current evaluation practice must be
gained.
This chapter addresses the question: What evaluation is already occurring in agricultural extension
in Australia? As a result of the scarcity of published literature, I conducted a review of mainly
unpublished reports of evaluation that were conducted on agricultural extension projects in
Australia over the last 10 years. This review was previously published in conjunction with several
other authors, (see Preface; Dart et al., 1998). The other authors helped collect the review
documents and write the final report (110 pages). I was solely responsible for reviewing and
analysing all the documents collected. I authored all the excepts that are presented in this chapter,
which I substantially revised since the Review of Evaluation in Agricultural Extension was
published.
The chapter begins with the background to, and justification for, the approach taken in reviewing
evaluation practice. I then present the findings of the review. The main method of presenting the
results of the analysis was to sort the sample of evaluation studies according to Owen’s (1993)
classification of evaluation forms. I then present some case studies to illustrate the different uses of
evaluation in extension. An attempt is also made to analyse the information in ways that may
provide insights into the range and value of evaluation studies being conducted within the field of
agricultural extension. After presenting the case studies I attempt to use the meta-evaluation
questions developed in Chapter 2 to review the 50 studies en masse.
4.2 Background to the Review
In 1997, at the start of this study, I conducted a search for published literature concerning evaluation
in the field of agricultural extension in Australia. I found very little published material. However,
published literature covers just one part of the ‘lived experience’ of applied evaluation: only the
information that authors choose to write about comes into the public domain; other potentially
valuable information often remains unrecorded and hidden. It could well be that practitioners do not
fully realise the value of their experiences for others in similar situations. The lack of information in
the public domain is also no doubt influenced by the confidential nature of many evaluation
findings. Another limiting factor may be the lack of fora for dissemination of this literature. There is
currently no Australasian journal of extension in which these findings and methodology might be
69
published.
The published literature existing tends to recommend the need for more evaluation and monitoring:
Lees (1990); McDonald and Kefford (1998); Dunn (1996); Woods et al. (1993); and Russell et al.
(1989). This finding was substantiated in the early stages of collecting material for this review. A
very common response to our request for information was that practitioners were just beginning to
learn about evaluation and had no written reports that they felt were good examples of evaluation to
include as case studies. This is reflected in the fact that of over 660 members of the Australasian
Evaluation Society, in 1997 only two listed agriculture as an interest (Australasian Evaluation
Society, 1997).
Indeed, there appears to be substantially less information available on extension evaluation
nowadays, than on evaluation of health or education programs where the field of evaluation is
constantly evolving to meet the needs of new programs and new ways of thinking. In 1997, I found
the existing reference information on extension evaluation to be largely out of date and inaccessible
to many extension agents. The current lack of literature concerning program evaluation in the field
of agricultural extension is perplexing to those who are aware that historically many important
contributions to program evaluation have been made from the field of agricultural extension (see
Section 2.2). Perhaps this can be accounted for by the character of agricultural extension itself.
Agricultural extension aims to change behaviour through the use of communication. Behaviour
change is commonly perceived to be difficult and complex to measure, especially in terms of
quantitative data. It is worth recalling that few agricultural extension workers have traditionally had
training in social science disciplines and they are likely to be unfamiliar with the range of
qualitative data collection methods.
However, since the completion of the review in 1998, the picture has been changing. Throughout
Australia training courses in program evaluation have multiplied. In Victoria a special project
named the ‘Evaluation Pilot’ was initiated in 1997 to facilitate over 50 major projects in the
Agriculture Division of NRE, to establish evaluation plans. This initiative involved intensive
training for staff in a range of evaluation issues, and individual support to projects. By May 2000,
over 50 staff from the Agriculture Division had been through this training, with another 50
participants expected to complete the training by 2001. At a national level the Rural Extension
Centre in Queensland and the University of Melbourne also offered increased provision of training
in evaluation specifically for extensionists. However, in 1998 when I completed the research for the
70
review of current practice, many of new and innovative approaches to evaluation were still in early
stages of implementation, and therefore were not included in this review.
4.3 Approach Taken
Documents were collected from over 80 evaluation studies from a wide range of sources, ranging
from rural extension centres, to universities and agriculture departments in various states, to private
consultants and individuals in the field of agricultural extension. These documents came from all
six states of Australia and were conducted between 1990 and 1997. This list grew to over 100
respondents from all states and some overseas specialists, through advice and further contacts
received from individuals and members of the Australasian Pacific Extension Network. Of these
documents, 50 were selected to provide the data for an analysis of evaluation in extension. The
remaining reports were excluded as they contained insufficient information, were primarily
evaluations of ‘one-off’ workshops, or were guides as to how to conduct a hypothetical evaluation.
This sample contained some inherent bias in that it comprised mainly documented evaluations and
did not include internal departmental monitoring and evaluation systems (although anecdotal
evidence suggested that little of this is occurring), or informal evaluations. Macro-level evaluations
were also excluded, and some innovative forms of evaluation research may have slipped through
the net because they are not always labeled as evaluation. It does however, provide a first glimpse
of the overall picture of approaches to evaluation in agricultural extension nationally.
Process used to review the evaluation studies
The 50 evaluation studies in the sample were characterised according to variables relating to (a) the
program, (b) the evaluation, and (c) the type of methods used. The evaluations were then classified
according to Owen’s meta-model (see below). Where possible, the meta-evaluation questions,
established in Chapter 2 were then used to critically examine the evaluation studies.
Initially, I intended to classify the case studies in terms of the normative models to which the
approaches related – such as the framework presented in Chapter 2 (experimental, objectives-
testing, judgemental, decision-management, intuitionist-pluralistic and theory-driven approaches).
However, on reading the evaluation documents, it became clear that almost none of the documents
explicitly referred to the underlying approach or model of evaluation. For example, only two
71
theory-guided approaches to evaluation were found, no overtly pluralistic-intuitionist approaches
were referred to, and only one document referred to the use of a decision-management approach
(Utilization Focused Evaluation). Most of the evaluation studies did not refer to any evaluation
literature, or popular models of program evaluation. Indeed, anecdotal evidence suggests that at the
time of the review, few people practising extension evaluation had any formal training in evaluation
and therefore they used the skills they have from other disciplines such as extension, or rural
sociology, to structure the evaluation. Some studies were based on social-science premises, others
on participatory rural appraisal approaches used in the practice of extension. For this reason, the
evaluation studies were classified and examined against a framework that differentiates between the
purposes of evaluation studies.
Owen’s meta-model
The conceptual meta-model of Owen (1993) was adopted to classify examples of evaluation studies
obtained from the field. This meta-model uses a relatively straightforward framework based on
purpose, and results in five forms of evaluation:
• Form 1 Impact evaluation
• Form 2 Evaluation in program management (monitoring)
• Form 3 Process evaluation
• Form 4 Design Evaluation (to clarify the program design)
• Form 5 Development evaluation (to develop new programs).
In essence, these forms relate to the different types of information that program staff might wish to
gain. Table 1 summarises the meta-model, and shows how each form is related to a specific
purpose. The forms of evaluation are also related to a timeframe within the program cycle.
72
Table 1 Owen’s five forms of program evaluation
Form of evaluation
Form 1
Impact evaluation
Form 2
Evaluation in program
management
Form 3
Process evaluation
Form 4
Design evaluation
Form 5
Evaluation for development
Ultimate purpose
Justification Accountability Improvement Clarification Synthesis
State of program
The program is settled
The program is settled
The program is at developmental stage
The program is at developmental stage
The program has not yet been implemented
Timing of evaluation
After program Runs throughout the program
During program implementation
During program implementation
Before program is implemented
Focus of evaluation
On outcomes/ delivery of the program
On delivery/ outcomes of the program
On delivery of the program
On the design of the program
On context of the program environment
Typical approach/ Methods used
Objectives-testing evaluation, needs-based evaluations, economic evaluations
Program monitoring, system evaluation, use of performance indicators over time
Implementation studies, action research, responsive evaluation, topical RRAs or PRAs, process evaluation
Evaluability assessment, program logic, accreditation
Needs assessment, review of best practice, exploratory RRA (rapid rural appraisal) ex-ante evaluation
Time span Program finish Throughout program
Program is settled
Program is developing
Program synthesis
Form 1 Form 2 Form 3 Form 4 Form 5
Source: adapted from Owen (1993: 22)
The domains of evaluation
Owen’s framework refers to five ‘forms’ of evaluation distinguished by the main purpose of the
evaluation. The extensive literature in program evaluation discusses various ‘models’ and ‘methods’
of evaluation. In Chapter 2, I presented a meta-model that categorised evaluation into six broad
normative approaches to evaluation. These various ‘domains’ in evaluation (form, framework,
model, and method) can be confusing, so an attempt is made here to explain the relationships
between various terms in common use in evaluation literature. Figure 4 illustrates the conceptual
relationship between the different domains of evaluation for the evaluation bricolage for a
hypothetical program. These terms were defined in Chapter 2.2.
73
Evaluation bricolage for one project Form 4 Normative Form 1
Evaluate program models Impact evaluation logic
m6 Form 2 m5
m1 m2 Monitor program
m4
m 3
m represents a method of collecting and analysing data
Figure 4 Example of an evaluation bricolage for an hypothetical program, showing three
domains
In Figure 4, the large ellipse represents the evaluation bricolage for an individual project. Within
this ellipse are smaller ellipses representing different ‘forms’ of evaluation. For this hypothetical
program, three forms of evaluation are required to meet three different evaluation purposes.
Conceptual models influence the choice of methods of data collection or creation and the forms of
evaluation adopted. It should be noted that one method of data collection or creation could be used
for more than one form of evaluation and that existing data may also be used. In this review the
evaluation studies were categorised according to form, rather than method, or model.
4.4 Description of the Evaluation Studies
Figure 5 shows the distribution of studies according to Owen’s (1993) forms. A difficulty
experienced in using this classification was that studies could often be placed in more than one
category, especially with regard to the time frame that Owen attached to the meta-model (see Table
1). The distribution of the studies according to a slightly modified Owen categorisation is shown in
in Figure 5.
Existing data
Existing data
74
Figure 5 Distribution of the evaluation studies according to Owen’s five forms
The distribution in Figure 5 shows that impact studies were the most common, but that process
evaluation (Form 3) was also frequently used. From information on the date of the studies there
appears to be a recent trend towards process evaluation, and also towards the use of evaluation for
formative rather than summative purposes in agricultural extension. This trend is congruent with the
rhetoric of many current extension funders who promote evaluation for learning, in addition to
evaluation for accountability purposes (for example the Agriculture Division in Victoria, as
discussed Section 3.5).
Process evaluation
37%
Evaluation for development
4%Evaluation for
design clarification
9%
Monitoring7%
Impact evaluation43%
75
4.5 Case Studies of Different Forms of Evaluation in
Agricultural Extension
In the following sections the five forms of evaluation from Owen’s meta-model (Owen 1993) are
outlined and illustrated with case studies from agricultural extension. Each of the case studies is
examined in terms of what it can offer the ‘basket of choices’ for extension evaluation, and
particularly what the approaches can contribute to the meeting demands for evaluation for large
extension projects (Chapter 3). The discussion in Chapter 3 revealed that evaluation is needed
which can:
• show the extent to which predetermined objectives have been achieved and satisfy
accountability requirements
• provide accountability for group-based models of extension that produce diverse, context-
specific outcomes
• include participatory process that foster learning of the participants and augment the
intervention effort
• be conducted as part of an internal evaluation capacity and hence foster organisational learning
• provide performance information for communication and between multiple stakeholders
• involve evaluation processes that foster dialogue between multiple stakeholders.
Form 1: Evaluation for impact assessment
Owen (1993: 23-24) describes ‘impact evaluation’ as evaluation that leads to a decision about the
worth of a program and which often has a strong summative emphasis. This type of evaluation is
generally carried out at the end of the program, or when a program is at a settled phase. The
principal foci of impact evaluations are to:
• understand the outcomes of the program
• justify program spending
• gain guidance about what to do next.
This was the most commonly found form of evaluation. Within this category of impact evaluation
in agricultural extension, several approaches were found. Each of these approaches is illustrated
76
with a case study under the following headings:
• cost benefit analysis for impact evaluation
• objectives-testing impact evaluation
• needs-based, participatory impact evaluation.
Cost benefit analysis for impact evaluation
Cost-benefit analysis (CBA) is a procedure for comparing alternative courses of action (or no
action) by reference to the net social benefits that they produce. A net social benefit refers to the
difference between social benefits and social costs. CBA is a method for organising information to
aid decisions about the allocation of financial resources (Department of Finance, 1991). The method
can be applied to specific projects or to larger programs as a whole. Its power as an analytical tool
rests in two main features:
• costs and benefits are each expressed (as far as possible) in money terms and hence are directly
comparable
• costs and benefits are valued in terms of the claims they make and the gains they provide to the
economy as a whole, so the perspective is a global one rather than one of any particular
individual, organisation or group.
CBA is widely used as a generic tool in evaluating the financial impact of agricultural programs. Its
application may also be ex-ante (ie. before implementation). However, cost-benefit analysis is often
used when a program has matured or ended, as part of the evaluation of impact or outcomes.
Practical constraints in using CBA concern the difficulties in assigning money values to the costs
and benefits of a project or program (Department of Finance, 1991). Ultimately CBA is a tool for
summative evaluation: to provide information for those who are to make judgements about whether
the program outcomes justify the expenditure.
The use of CBA is commonly criticised by agriculturalists on the grounds that unrealistic
assumptions are often used in estimating costs and returns, and because of the failure of the analyst
to take account of important issues. The findings of CBA often carry considerable weight in policy
decisions regarding the allocation of future funding. The fact that CBA reduces impact to dollar
values enables comparisons to be made across programs with very disparate outcomes. It has been
argued of late in the Agricultural Division, that agricultural extension professionals can often
77
provide quantitative information that can greatly improve the estimates and assumptions used by
economists in CBA (McDonald, B. pers. comm., 1999).
One of the main advantages of CBA is that it has high face validity with the Treasury and with the
people who make decisions about future funding. It offers a way of providing accountability for the
balance of benefits to costs. However, CBA has inherent biases and limitations, and should be seen
as one part of a comprehensive evaluation strategy rather than a stand-alone approach. For
example, it offers little to extension officers in terms of learning. It is an example of ‘black box’
evaluation, providing no information about why certain impacts have or have not occurred. It is
generally carried out by an external evaluator; this could be deemed as incommensurable with the
ethos of participatory projects, especially if it were to be the only evaluation tool used. It also
provides little in terms of facilitating communication and dialogue between project stakeholders. It
does not explore unanticipated outcomes, and is limited to considering impacts that can be
converted into dollar values.
Case Study 1 provides an example of CBA conducted on the Target 10 Dairy Extension Project in
Victoria.
Case Study 1 Cost–benefit analysis of the Target 10 dairy extension program
The program
Target 10 has the aim of enhancing the viability of the dairy industry through programs that profitably increase the consumption of pasture per hectare. Information from research and development on pasture utilisation is extended to farmers through courses, discussion groups, newsletters, target graphing, re-designed feed planning, comparative analysis, field days, focus farms and demonstrations and other media.
Goal of the evaluation
To estimate the economic benefits and to conduct a cost–benefit analysis of the expected results.
The evaluation and assumptions
This was a preliminary evaluation based on benchmark studies prior to the project launch in 1993. It does not consider all benefits from project activities, i.e., those not directly associated with the project aim. The first step in the cost–benefit analysis was to estimate the average amount of pasture consumed/ha and the amount of milk produced had this project not been implemented in 1991. These and other assumptions made are listed below:
level of pasture utilisation: 6.5 tonnes/ha (range 4-9 tonnes in Victoria) production from increased use: 7% increase in milk results from 10% increase in consumption milk production in Victoria: 511 514 litres/farm on 8469 farms (Q0)
78
levy and transport charges: 2.3 c/l (levy) and 2.7 c/l (transport) milk price received (P): 24.5 c/l – levies & transport charges = 19.56 c/l (P) average cost of production: 14 c/l before implementing Target 10 (AC0)increase in producer costs $2000/farm/yr required for implementation of Target 10
The maximum potential benefits resulting from Target 10 were calculated using the incremental profit method (Morrison 1993), i.e., from P = (P*Q1 – AC1*Q1) – ( P*Q0 – AC0*Q0) where Q1 (Q0 plus 7% increase in production) and AC1 are the Quantity of milk and Average cost with Target 10, respectively. The estimate of maximum potential per farm was $4728 and for Victoria was $40 038 500.
Method (the cost–benefit analysis)
The total nominal cost of the Target 10 Project was $9 198 000 from various sources. The spreadsheet model Appraisal (Appleyard 1996b) was used for the cost–benefit analysis. The benefit of the project is ‘accelerated adoption’. Appraisal was run twice: first using the adoption profile had Target 10 not occurred, and second using the adoption profile with Target 10. The Net Present Value (NPV) was calculated as the difference between the NPVs of those two scenarios. To calculate the BCR (benefit cost ratio), the difference between the two present values of the benefits was divided by the present value of the project’s costs.
Target 10 was designed to accelerate the rate of adoption of favourable innovations related to pasture utilisation (years from information being available to uptake by farmers). In addition it was expected to increase the level of adoption (percent of farmers). It was assumed that without the project, maximum adoption of the relevant practices would be 32%. In 1997 40% of Victorian farmers attended Target 10 training courses across Victoria. Some further assumptions and the results of applying the Appraisal model are summarised in the table below: Assumptions Without Target 10 With Target 10 Year adoption begins 8 2 Maximum adoption 32 40 Years to obtain maximum adoption 15 7 Maximum potential benefit 40 038 519 40 038 519 Cost benefit analysis Present value benefits ($) 57 404 678a 120 084 711b
Present value of costs ($) 6 763 653c
Net present value ($) (b-c)-a 62 680 033 Benefit-cost ratio 10.27 IRR (%) 140.15
A sensitivity analysis was conducted for a number of possible adoption profiles. The year adoption begins and the maximum level of adoption without Target 10 are of particular interest because these contain the greatest amount of uncertainty. The year to reach maximum adoption was held constant and the results are shown in Figure 6:
79
-100
1020304050607080
2 3 4 5 6 7 8 9 10 11 12
NPV
$m
Year adoption begins
Max adoption 32% Max adoption 36% Max adoption 40%
Figure 6 Graph of net present value at different adoption rates
The graph shows that the year in which adoption would have begun without Target 10 has a large effect on the project’s outcome. The longer it would have taken for adoption to occur, the greater the benefit from extension work. The greater the acceleration due to Target 10, the greater the benefits. The graph also shows that the increase in maximum adoption resulting from Target 10 also affects the size of the benefits. A break-even analysis was conducted to work out the maximum costs that the producer could incur from implementing Target 10, before incurring a loss. Assuming a 7% increase in milk production from a 10% increase in pasture utilisation, this was found to be $5000 per farm.
Use of findings
Using the assumptions stated, the analysis indicates that extension has sped up adoption by six years, resulting in a net benefit of $62 million to the Victorian dairy industry. This preliminary analysis helped to validate project plans, and will be followed up by other evaluations by means of surveys of change in pasture use and farm profitability (Boomsma et al., 1996).
Sources: Boomsma et al. (1996); Appleyard (1996a); Appleyard (1996b) and Morrison, (1993).
Objectives-testing impact evaluation
Objectives-based evaluations judge whether the stated goals or objectives of a program have been
achieved. Tyler (1967) was amongst the first to develop and use goals-based evaluation. In this
approach goals are taken as given, and decisions about the success of a program are deduced from
whether the program has met its goals. According to Owen (1993) the main tasks in planning a
goal-based evaluation are to:
• determine the ‘real’ goals of the program
• decide how to determine whether the program has led to the attainment its of goals.
80
A standard procedure in objectives-based evaluation is to develop measures of goals that have
strong face validity. However, sometimes it is necessary to use substitute measures or indexes that
stand in the place of preferred or ideal measures. An example of a goal-based evaluation is given in
Case Study 2. In this case study, the goals of the Landcare program was taken as ‘given’, and the
evaluation essentially tested whether farmers in Landcare groups had greater awareness of
environmental issues than farmers who were not members of Landcare groups. This type of
evaluation has strong credibility for evaluation audiences in terms of proving that a program has
met some of its defined objectives and may be an important part of an evaluation bricolage of a
program.
However, it does not meet all evaluation needs and it has inherent bias and limitations. Firstly, it
does not investigate the validity of the objectives of the Landcare program. Landcare is a group-
based project, and each group may have individual objectives. This evaluation does not attempt to
explore these context-specific outcomes. The evaluation was externally conducted, and therefore,
would not foster organisational learning that a more internally derived process may have
precipitated. This evaluation is not participatory, and could even have encountered some hostility
by the Landcare group members, who could feel that the evaluation is being done to them. It does
not utilise evaluation processes that would foster dialogue between multiple stakeholders. The
evaluation design, being objectives-focused, does not investigate unanticipated positive or negative
outcomes of the program. It focuses on changes in attitude and knowledge, but does not investigate
changes in behaviour – which are after all the main purpose of extension. As in all quasi-
experimental designs, there is no foolproof way of proving that the two groups (those already in
Landcare groups and those not in Landcare groups) are in fact really representative of the same
population. Despite all these limitations the approach described in this case study was rigourous in
attempting to determine the extent to which valued objectives have been achieved, and may well
have served its purpose well. Evaluation of this type is a valuable contribution to the ‘basket of
choices’ for extension evaluation, but is not advocated as a stand-alone approach.
81
Case Study 2 Landcare: does it make a difference?
The program The Landcare program emerged in the state of Victoria during 1986 and has been spread throughout Australia as a model for effective community action to manage land degradation and assist the move to more sustainable land use. The evaluation described here was carried out in 1993, at a time when the Landcare program in Victoria could be described as in a settled state, with many groups having been operating for six years. This evaluation forms part of a wider Landcare evaluation undertaken within the Johnstone Centre in Wagga Wagga. This program is based on the assumption that Landcare group actions will facilitate the process of community development and thereby produce more aware, informed, skilled and adaptive resource managers with a stronger stewardship/land ethic, and result in the adoption of more sustainable natural resource management practices.
Goal of the evaluation To assess the effectiveness of Landcare groups by investigating the work of groups and their impact upon participants’ behaviour and on other rural landholders. Specifically, the study aimed to determine whether Landcare participants and non-participants were significantly different in terms of key social and farming variables and whether Landcare participation made a significant difference to landholders’ awareness of issues, level of knowledge of key resource management, and their stewardship ethic.
Form of evaluation The overall purpose was to investigate some aspects of the overall program aims. Owen’s Form 1, ‘Impact Evaluation’, best covers this example. This evaluation made summative judgements as to whether the program had met intermediate goals of the Landcare program, so took an objectives-testing approach. This evaluation attempted to measure the degree to which the goals (or substitute goals) have been met by program activities.
Method of evaluation A regional case study was carried out. The study comprised a comparison between Landcare respondents with non-Landcare respondents, and between respondents in Landcare areas with respondents in areas where Landcare groups did not exist. In April 1993 a 16 page survey was mailed to all land managers of rural properties in 12 catchment areas (nine Landcare areas and three non-Landcare areas) of Northern Victoria. Surveys were addressed to one listed owner per rural property, 30% of who were women. The researcher’s knowledge of the area and links with Landcare group there played a critical factor in the selection of the region. All groups surveyed had been operating for more than two years. The surveys were pre-tested using two focus group of Landcare participants and non-participants at separate locations. The individuals were contacted by telephone and given a brief overview of the project and invited to the pre-test focus group. The pre-test participants were mailed a survey before the focus group took place. Important revisions of the survey were made as a result of recommendations following the discussions at both meetings.
The questionnaire was a small (B5) booklet with a distinctive, authoritative cover and was posted with a covering letter and stamped return envelope. A reminder / thank you letter was posted after 8-10 days and a second mailout occurred after six weeks. Considerable effort appears to have been put into selecting appropriate topics to include in the survey. Factors such as property size, hours worked off-farm, length of experience as a farmer, membership of other groups, age of respondent and school education were correlated against participation in Landcare.
Use of findings The results were quantitatively analysed using chi-square and multivariate analysis. The statistical findings were then represented in tables with accompanying text to explain the findings. These findings were presented and published in a booklet.
Source: Summarised from a report by Allan Curtis and Terry De Lacy (1994)
82
Needs-based impact evaluation
Scriven (1967) offered a different perspective towards impact evaluation. He suggested that in some
circumstances objectives-testing evaluations have limitations. His argument was that testing the
goal (objectives) of a program does not tell the worth of that program for society in general, in that
it does not assess the goals themselves. In this thesis the term ‘needs-based’ evaluation will be used
to refer to evaluation where the orientation of the evaluation is directed toward the needs of society
or the stakeholders rather than a judgment of whether the given goals were reached. For example, if
the goal of a hypothetical program were to produce a million tonnes of beef as cheaply as possible,
a strictly goal-based program would work out whether this goal (and perhaps intermediate goals)
had been achieved. A needs-based evaluation, in addition to investigating whether the given project
goals have been met, might also look at whether program met the needs of the farmer’s and what
impact the program had on the environment, on the community and so forth.
The choice between a ‘needs-based’ approach to evaluation and a ‘goals-based’ approach should be
made with regard to the underlying purpose of the evaluation. Patton (1997) suggests that to be an
effective evaluator one needs to be able to evaluate with or without goals. Case Study 2, involving
the West Hume Landcare Group, gives an example of a needs-based, impact evaluation. It presents
quite a contrast to the previous case study and is much more participatory in orientation. Like the
intuitionist-pluralistic approaches discussed in Section 2.4, the evaluation in this case study attempts
to draw the farmers into dialogue with regard to what they consider to be important, rather than
testing ‘given-objectives’ as in Case Study 1.
This approach does appear to be able to meet some of the challenges discussed in Chapter 3. It
could go someway to providing accountability for group-based models of extension such as
Landcare that produce diverse, context-specific outcomes. Being a participatory approach it may
well be more commensurable with this type of project and foster learning of the program
participants. Again it does provide processes that can foster dialogue between multiple stakeholders.
It also goes some way to meeting demands for summative evaluation for accountability. It uses a
novel mix of approaches, including economic analysis. I applauded the evaluators for attempting to
address many of the challenges facing large participatory extension programs. However, it does not
provide any on-going information for learning, and was conducted while the program was in a
settled phase of development. It appears also to have been a highly resource intensive evaluation,
that might not be practical in every situation.
83
Case Study 3 : A case study undertaken with the West Hume Landcare Group as part of an international research project on Participatory Watershed Development
The program West Hume Landcare group was chosen to be one of two case-study evaluations of the National Landcare program. The group was established in 1989. There are 160 farms in the area and 85 families are group members. The main land degradation issues are rising water tables and salinity associated with a massive change in the natural vegetation, soil acidity, soil structural and nutrient decline. The evaluation was carried out when the group was well established. Nationally, there has been growing concern about how to evaluate local Landcare activities in a way that is useful to local groups, but which can also provide information to assist broader-scale policy development and help give a national perspective on Landcare achievements. This concern led to LWRRDC providing additional funding for a case study and to support the active involvement of a National Landcare facilitator.
Goal of the evaluation The overall goal of the evaluation was to appraise the success of Landcare, its shortfalls and future opportunities. More specifically the goals were to:
• enable Landcare group members to review the achievements, shortfalls and future opportunities for the group
• provide analysis of implications for Landcare policy • understand the economic outcomes of the group effort • provide an evaluation case study on Landcare to be presented at an international conference • provide a draft manual report for LWRRDC on how other groups could undertake a study such
as this, using a PRA (participatory rural appraisal) approach.
Form of evaluation This evaluation approach is ‘needs-based’ as opposed to ‘goals-based’ in that it attempts to evaluate against the needs of the community rather than against predefined goals. It is most similar to Owen’s Form 1, impact evaluation. However, it has both formative and summative elements: while it judges the performance of the project against community needs (summative), it also offers recommendations for the future (formative). This evaluation uses mixed methods ranging from participatory self-evaluation to economic analysis by external consultants.
Actors
External evaluators were used but their role was that of facilitation and analysis. There was a high level of community involvement in the evaluation. The audience was both internal (group and Landcare program) and external (IIED, LWRRDC).
Methods of evaluation
Four methodological approaches were used:
1. Qualitative data were gathered by means of a PRA. PRA is a method for rapidly gathering key information about a rural situation by drawing on local knowledge and integrating it with the knowledge and experience of professionals from a range of different disciplinary areas. The PRA was conducted for a period of seven days in 1994. Team members came from a variety of disciplinary areas. PRA techniques included semi-structured interviews, group histories, questionnaires, preliminary analysis of cost–benefit of some common land conservation practices, and an economic study of two local farms and two hypothetical best management practice farms on whether farmers could afford Landcare.
84
2. Some quantitative data were collected using a structured questionnaire, also during the PRA exercise, using semi-structured interviews.
3. Secondary information in the form of reports and previous studies were also examined and incorporated into the exercise.
4. Economic evaluation was carried out by external consultants prior to the PRA exercise. The purpose of that study was to determine whether farmers could afford Landcare. A detailed financial analysis was conducted of four typical farms of different sizes and enterprise mixes, and two hypothetical ‘best practice’ farms. Data for the analysis of the four farms were obtained by interview with four selected landholders.
PRA: during the day, team members conducted focus groups with a range of stakeholder groups. In addition, members travelled to farms and interviewed individual farm families. Groups included Landcare members, non-members, senior members of the community, a Landcare committee, and a women’s’ group. Each evening the team presented and analysed the findings during group discussions. At the end of the week the first results of the analysis were presented to the community and team during a dinner.
Use of findings
As well as being written as a report, this case study was presented at an international workshop on participatory watershed development in Bangalore, India, amongst 23 case studies from different countries. The West Hume case study was presented by the participating farmers and Landcare group members and was especially appreciated by people at the conference.
Source: Summarised from Woodhill et al. (1994)
Form 2: Monitoring and evaluation for program management
Monitoring and evaluation (M&E) for program management (based on Owen’s Form 2) is designed
to run throughout the life of a project and is generally set up during the design stage. This
distinguishes it from other forms of evaluation, which are more discrete studies. The main focus of
this form of evaluation is to provide continuous feedback to the program management information
system. While collecting material for the review several organisations reported that they were
initiating systematic monitoring systems, but I found little evidence of programs/projects that had
systematic, formalised, established monitoring systems at that time. This finding is confirmed by
the conclusion of Woods et al. (1993) that there was little formal M&E in Australia for the
management of extension.
In this section, two approaches to monitoring and evaluation are presented:
• planning-based approaches to monitoring using performance indicators
• evolutionary approaches to monitoring, that do not employ performance indicators.
85
M&E for program management is conventionally associated with input-output monitoring which
would be represented by levels 1 to 3 of Bennett’s Hierarchy (Bennett, 1977) (see Table 2).
Bennett’s Hierarchy is a widely used framework in extension evaluation, and is a postulated
outcome hierarchy for extension. Bennett describes a simplified chain of events assumed to
characterise most programs in extension. This chain of events is depicted as a hierarchy of
objectives and evidence for program evaluation. Bennett presents this hierarchy as a framework for
a goal-based approach to evaluation. The essence of this is that the more nearly the objectives of a
program are reached, the more positive the judgement of the program. The lowest level of Bennett’s
Hierarchy concerns the activities of implementing the program (such as contacting farmers, holding
meetings), while the highest level (Level 7 in Table 2) involves quite changes in farmer behaviour.
Bennett explains that it becomes increasingly difficult to attribute the changes to project activities as
the hierarchy is ascended.
Table 2 Bennett’s Hierarchy
Level Bennett’s Description Type of evaluation Level 7 Consequences for the target group Evaluation & monitoring relating to
social / economic impact Level 6 Behavioural changes in the target group
(Practice change, sometimes indicated by change in quality of products)
Evaluation & monitoring relating to behaviour change (also impact monitoring)
Level 5 Change in knowledge, attitude, skills, motivation and group norms
Evaluation & monitoring relating to intermediate indications of behaviour change
Level 4 The farmers’ opinion about extension activities
Evaluation & monitoring relating to farmers’ opinion
Level 3 Farmer participation in extension activities (participation monitoring)
Monitoring degree of participation
Level 2 Implementation of the program by extension agents (activities monitoring)
Activity monitoring
Level 1 Programming of the extension activities (inputs monitoring)
Input monitoring
Source: Modified from Bennett (1977)
In Australia, many extension programs do record inputs and outputs for accountability purposes.
But records of farmer perceptions of extension activities, changes in knowledge, attitude, skills,
motivation and group norms (KASA) and behaviour change (levels 3-6) are likely to be monitored
only informally, if at all. Sometimes observations are written in extension workers diaries and
internal files, or as verbal reports. A review of M&E of ‘Property Management Planning’ programs
in South-East Queensland showed that, while a significant amount of information was collected for
the program coordinators’ own use, few data were reported formally. In some cases the data were
86
collected for the team and reported verbally, but not documented. The most frequent time period for
reporting was six-monthly, and the type of information collected varied considerably from one
coordinator to another. It was most common to monitor levels of participation in the meetings and
farmer perception of workshop activities. Little information was routinely collected on changes
resulting from the workshops (Stewart, 1994).
Performance indicator approaches to monitoring
Traditionally, formal monitoring in program management records performance (or management)
indicators over a period of time. A performance indicator is a simple statistic recorded over time, to
inform managers of the success of some aspect of program performance. Examples of indicators in
agricultural extension might be the number of farmers contacted by extension agents per year, or the
number of members participating in discussion groups. Owen (1993) stresses the importance of
ensuring that evaluators use a full range of data collection and analysis techniques. In terms of
Bennett’s Hierarchy, a full range would include the above simple indicators, as well as some
measures of change at higher levels, such as number of farmers adopting a new practice, or decrease
in the number of farm families on social welfare.
The danger is that simplistic performance indicators can miss the bigger picture. In addition to this,
reliance on quantitative indicators alone can ‘lead’ the program: they are not neutral and can be
counter-productive and even run against the greater aims of the program. There is now a significant
debate about the past usefulness of performance indicators in government agency and corporate
planning. For example, Winston states that:
It appears that there is no evidence in the literature or in the practical experience of governments that
performance indicators have ever been successfully applied by governments to the evaluation of
human service programs, in the context of program budgeting. There is evidence to the contrary, that
exemplifies failures to get performance indicators to contribute as intended to management and
budget cycle decision making (Winston, 1991: 604).
Although many good examples of M&E exist in overseas development projects, no documented
evidence was found of comprehensive M&E in agricultural extension. On-going monitoring is
regarded to be an important part of the ‘basket of choices’ for extension evaluation. However, as
has been discussed, purely ‘indicator-driven’ approaches have serious limitations. In overseas
development projects a new approach to monitoring has been emerging that is referred to as
Participatory Monitoring and Evaluation (PM&E) (or monitoring without indicators). This approach
87
has similarities with pluralist-intuitionist approaches outlined in Section 2.4. One example of this
form of evaluation was found in Australia. Although the implementation was at an early stage at
the time of the review, it is nonetheless included.
Monitoring without indicators
Case Study 4 is an example of an innovative approach to ‘monitoring without indicators’. It seems
logical that an on-going process of data collection is necessary to allow on-going learning for a
project team. I applaud the approach referred to in this case study for managing to incorporate
qualitative data into the monitoring approach, and avoiding many of the problems of an indicator-
driven approach to monitoring. However, in comparison to many of the PM&E approaches being
practised in overseas development, the approach in this case study does not appear to provide much
opportunity for the participatory analysis of the data. It provides no guidance for how the data is
analysed. In PM&E in overseas development projects it is the stage of participatory data
interpretation that allows an opportunity for learning and dialogue concerning differences in
opinion. Despite the lack of examples available in Australia, on-going monitoring approaches are an
essential part of the ‘basket of choices’ for extension evaluation.
Case Study 4 FARMSCAPE (Farmer-Adviser-Researcher Monitoring, Simulation, Communication and Performance Evaluation) for improved management of dryland cropping
The project
FARMSCAPE is currently funded for three years and commenced in 1995. It is run by APSRU (Australian Production Systems Research Unit), a group made up of officers from the CSIRO and Queensland DPI. APSRU is directly interested in the geographical area from Northern New South Wales to Central Queensland. APSRU’s core technology is based on a computer program, Agricultural Production Simulator. APSIM uses meteorological, crop and soil data to simulate the consequences of management on system performance. FARMSCAPE’s decision support features a high degree of contextualisation and provision of insights rather than prescriptions. The notion of decision support has evolved from a sole emphasis on developing software as a ready-made product for use by farmers, to developing new relationships and learning processes withinin this project.
The FARMSCAPE project (that this evaluation concerns) was developed from a pilot project established in 1992 as part of a process to link APSRU with three farmer groups operating in the Dalby region. The aim was to demonstrate tools that were available to improve sound decision making, as well as providing feedback on those decision-making areas most likely to benefit from the products of research and modelling. The FARMSCAPE project had been operational for two years when this evaluation commenced.
Goal of the evaluation
The overall goal of the evaluation was to determine the degree to which various aspects of the FARMSCAPE RD&E approach was contributing to how managers think about management of their
88
production activities in the face of extra-ordinary weather uncertainty, changes in their practice, how this was happening and in what ways it could be enhanced. More specifically it aimed to:
• identify opportunities for contributing to new insights and better planning and decision-making • log evidence of the degree to which a project activity contributed to new insights and better
planning and decision-making.
Form of evaluation
This evaluation is best described as Owen’s Form 4: monitoring and evaluation for management. It is an on-going evaluation and is aimed at improving the delivery of the program as well as contributing to making summative judgements at a later stage. This evaluation process could also be described as Form 2, evaluation in program management.
Actors
Both internal and external evaluators carry out the evaluation. The audience appears to be mainly internal to the program, including the farmer clients.
Method of Evaluation
The central evaluation tool was the use of ‘logs’. In the context of this study, logs are iterative records/interviews of events, reactions, attitudes and action captured in a given framework, through a combination of:
• individual written notes/reports by key persons at selected intervals, or as an issue arises • structured debriefing sessions by selected groups themselves or facilitated by an outsider • semi-structured interviews with key persons.
Currently, 30 persons are ‘logged’ by telephone interview every three months and are interviewed face-to-face by an external evaluator every 6 months. The informants include farmers, consultants, DPI extension staff, researchers and modellers. This method of evaluation is still being developed. The intention was to trial various approaches and to select the most effective ones to use. The logs are to be used with each of the main stakeholder groups in the project.
Presentation/utilisation of findings
It is intended that the information gained in this process will be fed back to project members to assist in improving future actions.
Source: From information supplied by R. McCown, CSIRO (Australian Production Research Unit, Toowoomba) and J. Coutts (Rural Extension Centre, University of Queensland, Gatton)
Form 3: Process evaluation
The principal focus in process evaluation is on the improvement of a program and understanding
how it is being implemented. This form is generally carried out at or before program completion
and is distinct from monitoring in that it is a discrete activity. Examples of this model include
implementation studies, action research, and responsive evaluation. The Review found several
89
approaches to process evaluation, including:
• process evaluation that is carried out to understand how a program is being implemented
(illumination evaluation)
• process evaluation that is particularly responsive to the needs of the program’s clients and
aimed at improving the program while it is still developing (responsive evaluation)
• process evaluation that is carried out to provide guidance in the refinement of a program. This
approach is often carried out when reviewing pilot programs, and has a strong formative
element.
Process evaluation carried out to understand program delivery
Owen, (1993) explains how during the 1960s and 1970s the importance of examining
implementation in formal evaluation studies emerged. When programs failed to achieve desired
outcomes, practitioners started to examine how the programs had been implemented. It was found
that the type of implementation varied greatly from site to site and that the process of
implementation was an important factor in determining whether desired outcomes would be
reached. Studies of this kind are referred to as ‘process-outcome’ studies and are generally
summative in nature (Owen 1993).
Process evaluation that is particularly responsive to needs of program clients
(farmers)
Some process evaluations are designed to be responsive to the needs of those directly affected by
the program. Responsive evaluation is based on the view that those with a direct vested interest in
program delivery should control any evaluation agenda associated with the program. Often
intuitionist-pluralist approaches are conducted to in order to improve a program. Case Study 7
illustrates a process evaluation that was oriented to be responsive to the needs of the clients. This is
a small-scale formative evaluation, aimed at making the extension practice more useful for the
farmer clients. While this sort of evaluation could no doubt meet this need adequately, it might not
be appropriate for larger scale programs with many stakeholders, nor for a summative assessment of
impact. This case study illustrates the point that different sorts of evaluation are appropriate for
different purposes. However, this sort of rapid ‘formative’ evaluation based on farmer needs, is a
valuable component in the ‘basket of choices’ for extension evaluation.
90
Case Study 5 Improving the usefulness of the Pigpulse Report - an evaluation
The program
The Pigpulse Report was developed to improve the analysis of piggery data, to help decisions about interventions in piggery management. The Pigpulse project is a service provided to producers involved in the Sowtel-Pigpulse bureau operated under the project Pig Enterprise Management Services (PEMS). To generate the Pigpulse Report, a farm’s production figures and some financial records are processed using a computer package. Results of both individual farm productivity and state and district averages are sent back to farmers quarterly. The report lists how the farm production has deviated statistically from these averages. The report monitors traits such as pre-weaning mortality, average birth weights, and weaner weights.
In addition to the report, producers participate in discussion groups in a number of districts. These groups discuss issues raised by the reports, and the groups are encouraged to determine some of the topics. This evaluation concerns one aspect of the PEMS project; the Pigpulse project. This project was in the process of being developed at the time of evaluation.
Goal of evaluation
To find out if the development of the Pigpulse report was useful to producers in supporting decisions.
Form of evaluation
This evaluation best fits Owen’s Form 3, process evaluation. The evaluation information was aimed at improvement and development of an existing activity. It involved some assessment of how the activity is reaching its goals, but concentrated mainly on how the activity can be improved and made more effective. There was an opportunity to make changes in the software development as well as in the extension.
Actors
The evaluator was internal to the program, as was the primary audience for the evaluation.
Method of evaluation
The pre-test The evaluation was conducted by pre-testing an interview guide during a semi-structured interview with three producer clients. This guide contained open-ended questions and prompts that were developed from the key question and hypothesis. The interviews were conducted on the producers’ farms at a time to suit them. The interviews were arranged by telephone, the producers being informed of the interview content and purpose. The interview guide was used while the producer was asked to look at their last report and to recall how they used the report and what else was considered in making decisions in their piggery management. A copy of their Pigpulse report was provided as interviewees might otherwise say what they think they should consider instead of what they actually did consider.
During the interview, notes were written by the interviewer and were checked by the producers and modified to include their corrections. The results were coded according to a series of statements derived from the hypothesis. Examples of the statements in the coding frame were:
• whether producers read and thought through the Pigpulse report • whether trait type (of the pigs) was considered
91
• whether the magnitude of change was considered • whether targets were compared with the new level • whether the cause of change was considered • whether the effect on profit was important • whether producers changed their practice.
As a result of the pre-test, the interview guide was altered before the focus group (FGD) took place. Focus groups were chosen rather than questionnaires as the producers were feeling over-surveyed.
Use of findings
The results of the interviews were related to the level of Bennett’s Hierarchy that had been affected by the program. The report was an evaluation in process and did not include findings of the focused group discussions. However, it suggested that these results could be expressed qualitatively.
Source: Prepared from a report by Alison Spencer, Rural Extension Centre, U. Queensland, Gatton (Spencer, n.d. )
Form 4: Evaluation for design clarification
Evaluation for design clarification is generally carried out when the program is at the developmental
stage. Examples of methods used in this form are ‘evaluability assessment’ (Wholey, 1983; and
Smith, 1989), ‘program logic’ (see below). While no examples of ‘evaluability assessment’ were
found the review, two examples of program logic were found.
Program logic
Program logic is the rationale behind a program or project – i.e., what are understood to be the
cause-and-effect relationships between program activities, outputs, intermediate outcomes, and
ultimate outcomes. Usually represented in a diagram, program logic shows a series of expected
consequences, not just a sequence of events. It is best conducted as a team and is noted for enabling
staff to come to achieve consensus about the realistic outcomes and goals of a program. Ideally,
program logic would be mapped out by the project team before the implementation, and modified
and referred to throughout the life of the project. However, in many cases, program logic is
conducted later in the project life to help in the evaluation process, or to bring a program back on
track.
Program logic is a type of theory-driven approach to evaluation. It makes an attempt to open the
black-box, and understand what mechanisms underpin a program. Since this review was completed,
program logic seems to have become a much more widely used tool in extension evaluation. But at
92
the time of the review, only one such completed evaluation was found. Case Study 6 is presented
here as it is an example of the use of program logic in evaluation for the purpose of clarifying the
design of a program once it has commenced. Theory-driven evaluation approaches do appear to
have much to offer extension projects, as they help practitioners understand how programs work,
and thus can facilitate learning. When program logic is done in a team, it can help create a shared
vision of the program. Clarifying program logic can also be very useful for developing the key
evaluation questions that will guide the evaluation strategy for a large project. One example case
study was also found in which program logic was used as the starting point for an impact
evaluation.
Case Study 6: A workshop to refine the program logic for Coranagmite Salinity Program
The Program
The Victorian DNRE (Department of Natural Resources and Environment) decided to have a rigorous look at one of its programs directed against the problem of dryland salinity. This program had been running for about five years, and was considered to be at a settled stage. The specific project to be evaluated was the Corangamite salinity program.
Goal of the evaluation To gain consensus as to the project aims and to improve the implementation of the project.
Form of evaluation This evaluation best fits Owen’s Form 4, evaluation for design clarification.
Actors The group that requested the workshop was the DNRE staff responsible for this program, i.e. an internal audience. Other stakeholders included the steering committee and an implementation group, and landholders. These groups were not included at this stage. The facilitators were from the DNRE, and had recently received training in program logic.
Method of evaluation
Two workshop sessions were carried out over 1½ days. Eight people attended and the mode of inquiry was informal discussion and brainstorming. The steps taken are outlined below:
Step 1. Defining the problem: In the first session the group set about defining the problem. The facilitators encouraged participants to think critically. At this stage it became apparent that there was no common understanding of the underling salinity problem. The participants were encouraged to describe the ‘rich picture’, the political nuances, and all the stakeholders. Once they had agreed on what the problem really was, they moved on to discussing what outcome would be achieved if this problem was solved. When a consensus had been achieved on realistic outcomes, they moved on to the next stage.
Step 2. Mapping the first layer of the model. Participants were asked by the facilitator to try to forget that they were involved in the program in reality and to imagine that they were in the planning phase of a new program. They were then asked to think about what was needed in place in the hypothetical program for the outcome to be achieved. This formed the first level of the hierarchy in
93
the program logic map.
Step 3. Mapping out subsequent layers of the model, going downwards from the outcome: The participants were asked how these necessities were to be achieved, and thus the map of program logic was created, moving downwards from the desired outcome to activities necessary to achieve these. At each level of the map the facilitators would double check with the participants, by asking questions such as: “are you really sure?” and “if you do this and this, will that really lead to the outcome?” At each step of the model the cause-effect relationship was explored. The facilitators constantly questioned the participants with regards to any assumptions they had made concerning the links between cause and effect.
Step 4. Back into the real program: Participants were asked to return to the reality of the current program and to compare the conceptual model with that of current activities and state of play of the real program. It became obvious that certain things were not being done in the dryland salinity program that should have been done and also that certain inappropriate activities were on-going.
Step 5. Considering evaluation: This stage was to consider where evaluation should take place. The facilitators suggested that the links between cause and effect that involved ‘leaps of faith’ should be evaluated. Participants were asked to identify these weak links in the model. Once the locations or links had been established, the participants were asked to consider which data would answer their questions and what evidence was available to check that one link in the program led to the next intermediate outcome.
Presentation/utilisation of findings
The results of these workshops were not formally recorded. However, five days after the workshop ended, the DRNE members of the dryland salinity program were involved in an independently-organised workshop with the whole dryland salinity program team of Corangamite catchment area. The participants of the previous program logic workshop rescheduled the whole workshop to incorporate what they had learned in the previous workshop and to share it with the rest of the team. The program design was significantly altered to address the issues raised in the first workshop. This constitutes a high degree of utilisation, according to Patton’s framework (1997).
Source: pers. comm. with David Beckingsale (1997) Department Natural Resources and Environment, Melbourne.
Form 5: Evaluation for program development
Form 5, evaluation for program development, is for the synthesis of a new program. The program
will not yet have been implemented and will still be at the planning stage. The focus is on the
context of the program environment. Examples of models and methods include needs assessment,
economic evaluation, review of best practice, research synthesis, exploratory RRA (rapid rural
appraisal) or PRA (participatory rural appraisal), and the Delphi technique. While conducting the
review, I found few examples of evaluation for program development. In extension, needs
assessment is often considered as research rather than evaluation and some examples may therefore
have ‘slipped through the net’ of the study, when extension agents were asked to supply examples
of evaluation. However, Case Study 7 is an example of needs assessment using participatory rural
appraisal (PRA).
94
Rapid rural appraisal
Rapid rural appraisal (RRA) and participatory rural appraisal (PRA) are used extensively in social
and natural resource projects in developing countries. I have been involved in many PRAs and
RRAs during previous work that I conducted with development projects in India and Mexico – the
following description therefore, comes from my past experience. Both these techniques involve a
multidisciplinary team staying in the community for a period of around six days. The techniques
used in RRA include semi-structured interviews, diagramming, stories, histories, diagrams,
conceptual representations, maps, transects, seasonal calenders, time lines, flow diagrams, decision
trees, venn diagrams, pie charts and more. An important component of RRA is the reliance upon
‘triangulation’: this involves crosschecking information by talking to different stakeholders, using
different methods (interviews, diagrams, and observations) and having a multidisciplinary team
(men, women, insiders and outsiders). Triangulation is carried out to assure the validity of the
findings by strengthening/optimising the sources of information (Pretty et al., 1995). RRA and PRA
share common methodology, but the aims to which they are put differ. RRA is conducted by
researchers to capture the views of local people, whereas PRA is focuses on facilitating and
contributing to the empowerment of local people to take control over their own appraisal of
problems and opportunities, and to select appropriate action.
In the last decade, modified versions of RRA and PRA have begun to be used in industrialised
country contexts. Generally, the reliance is more upon multidisciplinary teams, and a strong use of
semi-structured interviews. Less emphasis is placed upon diagrams and pictures. In the context of
evaluation, RRA/PRA can be used to develop new programs, for process evaluation and for impact
evaluation. It is most appropriate when a high degree of client feedback is required in the
evaluation. In ‘evaluation talk’ it would be best described by naturalistic inquiry. Case Study 7 is an
example of an RRA that was carried out in order to contribute knowledge to the planning process of
new programs. Again, this evaluation is an example of how evaluation can be used for a very
specific purpose. RRA techniques appear to have good potential for drawing a range of stakeholders
into the negotiation process, and are an important tool for the ‘basket of choices’ of extension
evaluation. RRA approaches have much to offer participatory extension projects, as they can foster
ownership of the program at early stages of planning if conducted well. But RRA approaches have
their potential bias and limitations. RRA can be very expensive; as they involve resourcing a whole
team of people to work with a community. It has also been argued that the rapid nature of the
appraisals introduces bias, such as seasonal bias and spatial bias (often the people who are more
95
accessible are involved more).
Case Study 7 Understanding farmer-decision making on land use
The program
The Department of Primary Industries (DPI) and National Soil Conservation Program (NSCP) is funding a major project to hasten the exchange of knowledge and management skills amongst southern Queensland farmers on fallow management and viable farming systems. The VFSG is a DPI project team integrating extension and research personnel across southern Queensland. The project commenced in January 1991 and ran for 5 years.
To ensure that VFSG programs and products would be relevant to users, and that exchange of respective information would be efficiently conducted, farmers attitudes, knowledge and needs on farming system were researched. This was part of a research project set up because of concerns over excessive soil erosion, rainfall run-off, declining soil nitrogen fertility and soil structure, sub-optimal use of herbicides and fertilisers, and insufficient rotation of crops and pastures. The research was undertaken to ensure that consequent VFSG programs and products would be relevant to users and exchange of respective information would be efficiently conducted.
Goal of the evaluation
Was to:
• develop an understanding of farming systems and farmer attitudes towards farming systems in southern Queensland
• gain an overview of the important issues as seen by the people involved in the management of these relevant farming systems
• determine which issues required further investigation and who possessed knowledge that would assist in these activities
• provide a base of common understanding for further market research and action.
The concept of ‘triangulation’ was used. This involved the use of different methods to study the same problem or situation and provides a way of crosschecking data.
Form of evaluation
This evaluation best fits Owen’s Form 5, evaluation for program development. In evaluation terminology it can be described as a needs-assessment using participatory methodology.
Actors
The evaluation was carried out internally to an internal audience.
Method of evaluation
The research consisted of two phases:
• a modification of Rapid Rural Appraisal • focus groups to crosscheck issues raised in the RRA, investigate issues in greater depth and to
design a quantitative survey for future progress and impact evaluation.
The rapid rural appraisal consisted of 100 face-to-face semi-structured interviews carried out by two
96
multi-disciplinary teams. Each team consisted of four interview pairs with one member from the VFSG and one from outside. In this context, RRA was used to produce a wide overview of possible aspects, people and interactions and to obtain an impression of which issues were likely to be the important. RRA is meant to be heuristic, in that it will tell you what is there. The aims of the exercise were achieved by: reaching a wide cross-section of the relevant community, adjusting the definition of what was relevant during the exercise, and not excluding any related material from the report. The interviews were conducted in pairs (or small groups) with different backgrounds.
The focus groups were arranged into five districts and were segregated with regard to their tillage practice. Participants had to be located within 60 minutes travel time to the focus group venue and those farmers involved in the RRA were not involved. Farmers who met the above criteria were selected randomly from DPI and Grainco mailing lists. The prospective respondents were contacted by telephone and invited to attend the focused group discussions. Moderators were used in the discussions to focus the groups in a non-directional manner. Interviews were taped to assist analysis. A typist transcribed the transcripts from the focus groups and a computer package called Ethnograph was used to analyse the data from the focused group discussions and search for patterns and themes.
Use of findings
Implementation of VFSG sub-programs involving many of the recommendations from this research has commenced.
Source: Taken from an internal document by David Blacket and Gus Hamilton, DPI ,Goodndiwindi Queensland, 1992.
97
4.6 Summary of the Evaluation Samples
A summary of some main characteristics of the sample of 50 evaluation studies of extension
programs is provided in Table 3.
Table 3 Summary of main characteristics of the sample of evaluation studies
Approximate percentage in categories (n=50)
Criterion 0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
Owen’s form: F1- impact F2- monitoring F3- process F4- design F5- synthesis
F1
43%
F2
7%
F3
37%
F4
9%
F5
4%
Summative or formative
Summative 54% Formative 32% Both 14%
Stage of program Developing 25% Settled 58% Fin-ished 12%
Approach adopted
Largely objectives-testing approach 63% Other approaches 37%
Perspectives considered
Multi-perspective view taken 40%
Maintained a single focus-perspective 60%
Data type Qualitative 44% Quantitative 27% Both 11%
Surveys Structured survey 52%
no structured survey 48%
Rapid rural appraisal (RRA)
RRA 10%
RRA not used 90%
Focused groups Focused groups used 42% not used 56%
The case studies and the summary of all 50 samples (Table 3) illustrate that extension evaluation is
employed for several different purposes, and examples were found that cover all of Owen’s five
forms of evaluation (for program development, design clarification, process evaluation, evaluation
in management and impact evaluation). However, the most common form of evaluation was found
to be impact assessment, followed by process evaluation. Surprisingly, I found that very little
program monitoring is conducted in agricultural extension (Owen’s Form 4; evaluation in program
management). Only two examples were found during the entire review. Of the two cases of
monitoring, one was a study in the initial stages of developing a system of program ‘monitoring
without indicators’. This study had much in common with PM&E approaches seen in overseas
98
development projects. While it had potential to offer an approach to facilitate an on-going dialogue
with project stakeholders, it fell short in terms of including a participatory approach to interpret and
make sense of the data.
Several studies were found that fitted Owen’s Form 3: process evaluation. These were a mix of
small-scale formative studies to improve a program (generally by making it more responsive to
farmer needs), and summative-process studies aimed at checking to ensure the extent of program
implementation. It appeared from the time-scale of the reports that process evaluations are a
relatively recent practice.
One study was found that fitted into Owen’s Form 4: evaluation for design clarification (concerning
program logic). This approach offers insight to extension agents in understand how a program
works, and offers a process to draw a number of stakeholders into the task of clarifying real
objectives, and plausible outcomes. It is notable, that since the review was conducted in 1997, there
has been a large increase in this form of evaluation, with most evaluation training courses now
including it in their agendas. This perhaps reflects the increasing demand for extension evaluation
in general. While only one study was found that fitted Owen’s Form 5, evaluation for program
development generally goes under the name of ‘applied research’, rather than evaluation and for
this reason studies of this type may have slipped through the net of the review.
Overall only 23% of the evaluations in the study were solely formative. Indeed, the majority of the
evaluations reviewed were carried out to provide a report to justify spending and to understand
whether the stated objectives of the program had been met or not. This is reflected in that fact that
evaluations reviewed were most frequently conducted while the program was in a settled stage of
development.
In terms of methods of data collection, there appeared to be a tendency for evaluators to stick to a
handful of familiar methods and approaches in evaluating extension programs. The qualitative data
collection commonly took the form of focus groups and semi-structured interviews, while
questionnaires are by far the most frequently practised quantitative data collection tool used.
It is evident that a range of approaches to evaluation were employed in extension evaluation, with
objectives-testing approaches being most common. Most of the studies were limited to a summative
assessment of project impact, and there was a pre-dominance of objectives-testing approaches
99
(63%). However, there was a group of evaluations that went against the general trend of objectives-
testing approaches for justification of spending. These studies were much more focused on gaining
information to revise an existing project, for example Case Study 5 (process evaluation) and 6
(program logic).
Intuitionist-pluralistic approaches were to some extent, apparant in the studies that employed some
form of RRA. These studies took a pluralistic approach, and encouraged dialogue between different
stakeholders, they also included farmer participants in the interpretation of the data. While few
examples of pluralistic-intuitive were found, a multi-perspective view is apparently quite common;
this involves utilising people from different backgrounds and ‘worldview’ in conducting evaluation.
While only two theory-based approaches to evaluation were found, several studies used Bennett’s
Hierarchy of Outcomes to structure the data collection or analysis. Few examples were found that
referred to a management-decision approach, such as Patton’s (1997) ‘Utilization-focused
evaluation’.
This study did not set out to examine changes in the form or approach to evaluation. However, there
are strong indications from the sample of reports received that the range of approaches to evaluation
existing in agricultural extension is expanding. There are a number of new and innovative
evaluations under way that are generally more formative and qualitative than appears to have been
the norm in the past. Nonetheless I suggest that, while there is a range of forms and approaches to
evaluation in practice, the current suit of approaches needs to be more expanisive to meet all current
challenges and demands for extension evaluation in Australia. In particular no evaluation study was
found that facilitated on-going communication between multiple project stakeholders.
4.7 Examining the Evaluation Studies Against the Meta-
evaluation Questions
In addition to analysing the sample case studies with the help of Owen’s forms of evaluation, some
of the key meta-evaluation questions developed in Chapter 2 were applied. Other meta-evaluation
questions were not addressed due to the fact that the review was limited to an analysis of project
documentation. However, the full set of meta-evaluation questions is used to evaluate the central
case evaluation at a later stage of this thesis (Chapters 7 and 8). The questions that were addressed
100
in this review are:
• Is the evaluation relevant to the context and purpose in hand, and does it fulfil the evaluation
needs of the stakeholders?
• Is the evaluation guided by program theory?
• Was the design and data collection/analysis of the evaluations valid?
• Are the evaluation processes socially and politically just?
Was the evaluation relevant to the context and purpose in hand?
It was not often possible to determine from the evaluation reports whether the approach employed
had been appropriate to the project context. For the most part, little attempt was made to justify the
approach taken in the evaluation studies. The reporting suggests that many practitioners were not
aware of the range of evaluation models and methods and available, and therefore would not have a
full choice in selecting models/methods to best suit the context.
Were the evaluations guided by program theory?
Chen (1990) discusses the traditional neglect of program theory and the popularity of method-
oriented evaluation. He writes:
Until very recently evaluation literature has rarely been concerned with the importance of theory in
evaluating a program or with how to incorporate theory into evaluation processes (Chen 1990: 17).
The results of this review in the field of agricultural extension concur with Chen’s findings,
although there is indication of a move away from method-driven evaluation in extension. Many of
the evaluation studies examined appeared to be heavily methods-focused, and showed little
evidence of examining the underpinning causal mechanisms of the project – or indeed questioning
the project objectives themselves – Case Study 1 provides an example of objectives-based
evaluation. In other words they were ‘black box’ evaluations, which do not comply with the
premises of good evaluation as advocated in Chapter 2.
Was the design and data collection/ analysis of the evaluations valid?
There was a wide range of data collection/analysis employed in the different evaluation studies, and
it is hard to generalise about their validity. However, anecdotal evidence from discussions with
101
various organisations and individuals during this study suggested that there was a lack of
confidence as to how to select appropriate approaches and evaluation tools, to conduct evaluations
and utilise findings. The observations suggest that in agricultural extension there is a tendency to
stick to a handful of familiar frameworks and methods such as Bennett’s Hierarchy (Bennett, 1977)
questionnaires and focus groups. In some of the studies there seems to be a lack of understanding
of evaluation in its broad sense, and as a result practitioners confine themselves to familiar methods
that may not best fulfil their objectives.
An important factor contributing to a lack of expertise and confidence in evaluation is the fact that
most extension personnel are trained primarily in the physical sciences and few have a sound
background in social science methods. Nowadays extension agents are expected, on top of the
requirements of applied science-based aspects of their work, to cope with a range of concepts from
the social sciences. Most have never studied literature on program evaluation, and if they have had
training in social science methods it has been mainly from in-service training on short courses.
There is an obvious need to address this problem if extension organisations are to improve the
ability of staff and the quality of evaluation practice.
Were the appropriate levels of the outcomes hierarchy considered?
Bennett’s Hierarchy is a hierarchy of goals established for agricultural extension programs
(Bennett, 1977) (see Table 2). This framework is generally well known to agricultural extension
agents, probably because it is mentioned in a well-used extension text (Van den Ban and Hawkins,
1988: 205-208). Bennett lists seven levels of goals in extension and claims that it becomes more
difficult to evaluate at higher levels of the hierarchy, as it becomes more difficult to show that
changes at these levels are the result of extension activity and not of other factors.
102
Level 8 Consequences for society (an additional level to Bennett’s Hierarchy) Level 7 Consequences for the target group Level 6 Behavioural changes in the target group Level 5 Change in knowledge, attitude, skills, motivation and group norms (KASA) Level 4 The farmer’s opinion about extension activities Level 3 Farmer participation in extension activities Level 2 Implementation of the program by extension agents Level 1 Programming of the extension activities
Figure 7 Distribution of evaluation studies according to the level of investigation of Bennett’s
Hierarchy
Figure 7 shows the frequency of occurrence of studies in the sample at each level of Bennett’s
Hierarchy. The majority (89%) of the evaluations considered level 3; the degree of farmer
participation in farmer activities, while 60% of the evaluations considered changes in KASA (level
5), and only 22% measured behavioural changes. Using Bennett’s Hierarchy as a starting point, any
program evaluation of extension should include some reference to:
• ‘Level five’; changes in farmer knowledge, attitudes and skills, in addition to lower level
outcomes such as farmers participation in activities. An evaluation which only addresses issues
such as participation (level 3) or reactions (level 4) would provide little evidence of worth or
merit of a program.
• ‘Level six’; changes in behaviour. Despite the fact that the higher level outcome of behavior
change is harder to attribute to the program activities alone, evaluation of these programs
0
10
20
30
40
50
60
70
80
90
%of sample
1 2 3 4 5 6 7 8
Level of Bennett's hierarchy
103
should make an attempt to provide evidence concerning the extent to which these programs
contribute toward behavioural change (which is after all the ultimate aim of extension).
The majority of evaluation studies in the sample did not examine higher level outcomes as
categorised by Bennett’s levels 5 and 6 (changes in knowledge, attitudes, skills; changes in
behaviour), but were preoccupied with assessing farmer/ client reactions, and participation rates.
Are the evaluation processes socially and politically just?
In Chapter 2, I proposed that in good evaluation, attention is paid to who is given voice in
determining the indicators for the evaluation. I proposed that participatory extension projects
involve their farmer-clients in the decisions surrounding ‘what gets measured’, and in the valuing
process itself. In the model of Fourth Generation Evaluation, it is proposed that the ‘victims’ of a
program are identified and their view sought (Guba and Lincoln, 1989). In the 50 evaluation cases,
few of the potential ‘victims’ were consulted. The majority of studies tended to be situated within
the discourse of the project; non-participants, or protagonists of the project not being consulted.
Figure 8 shows the results of scoring the evaluation studies in the sample against a continuum of
participation based on a scale of participation modified from (Dart, 1997). In 54% of the
evaluations, the clients were not heavily involved in developing the indicators for the evaluation. At
the other extreme, 7.5% of the evaluations involved the clients being facilitated to carry out their
own evaluation.
104
Participation levels adopted
1 2 3 4 5 6 7
External evaluator, no consultation
External evaluator, staff consulted
Staff conduct evaluation, clients not consulted
External evaluation, staff & clients consulted
Staff & clients develop indicators together
Clients conduct facilitated self-evaluation
Clients conduct self-evaluation - alone
Figure 8 Level of participation in evaluation by farmer clients in the studies
In many instances in agriculture, evaluators have seen little reason for client participation. However,
the proponents of highly participatory evaluation stress the relevance in many situations in
agricultural extension, of involving clients in developing the evaluation framework, especially with
regard to the choice of indicators. Patton (1997: 101) writes that such participatory evaluation ‘is
most appropriate where the goals of the program include helping participants become more self-
sufficient and personally effective’. In such instances, participatory evaluation is also intervention
oriented in that the evaluation is designed and implemented to support and enhance the program’s
desired outcomes.
Insufficient data was presented in most of the evaluation reports to allow critical comment on the
appropriate level of participation in each individual evaluation study. However, few examples (7%)
were found where farmer clients were documented as having involvement in deciding the sort of
change to be measured, despite the fact that many of the extension projects in the sample espouse
0
5
10
15
20
25
% of sample
1 2 3 4 5 6 7
Level
105
the rhetoric of participation. In this thesis I encourage those responsible for evaluating participatory
extension projects to consider more suitable or expanded forms of evaluation that involve farmers in
determining what is included in an evaluation study.
4.8 Conclusions
Program evaluation in agricultural extension in Australia (1990-1998) is used to meet a
considerable range of tasks in agricultural extension. Examples were found of all of Owen’s five
forms of evaluation. However, the majority of the studies reviewed were limited to a summative
assessment of project impact. There was also very little monitoring of projects. There was a range
of approaches to evaluation, including objectives-testing approaches, theory-based approaches and
some forms of pluralist-intuitionist approaches. But the most common approaches were objectives-
testing approaches, and these do not attempt to ‘open the black box’ and understand how a program
works.
In terms of complying with the premises of good evaluation developed in Chapter 2, I found it
difficult to fully assess the evaluations, as the review comprised an analysis of only written
evaluation documents. However, it appeared that there was a tendency towards methods-driven
approaches, a lack of knowledge of program evaluation theory, a tendency to focus of the lower
levels of Bennett’s Hierarchy, and a tendency towards black-box evaluations – which are all
contrary to the derived premises for good evaluation. Method-driven approaches imply that the
methods dictate the evaluation design, rather the opposite. Many of the studies did not appear to
consider the context and evaluation needs when deciding on the evaluation design. This was
confirmed by the tendency to stick to a few methods of evaluation, especially focus groups and
questionnaires. In terms of Bennett’s Hierarchy of outcomes, only 22% of evaluations attempted to
address changes in behaviour, which after all, is the ultimate aim of extension.
While there is a considerable range of forms and approaches to evaluation in practice, the current
suite approaches is inadequate to meet all current challenges and demands for extension evaluation
in Australia. In particular no monitoring approaches were found that could facilitate on-going
communication between multiple project stakeholders.
Evaluation as practised in agricultural extension in Australia appears to be a largely atheoretical
activity. Most of the evaluations reviewed were conducted without reference to developments in
106
‘program evaluation’ – very few examples were found that were explicit in mentioning underlying
models of evaluation and very few reports referred to the literature of program evaluation. Indeed,
in most of the sample, the evaluation design appeared to be based on concepts borrowed from
extension practice and in few cases on social science approaches. It is important that, during the
process of the review, many extension agents expressed a lack of confidence with regard to their
skills in evaluation. Nevertheless, this trend is changing, and to meet the increased demand for
extension evaluation, several training initiatives and research programs are under way that pay
attention to developments in program evaluation theory and practice from other disciplines.
In the 50 evaluation studies examined, limited reference was made to popular models of evaluation
such as Utilization Focused Evaluation (Patton, 1997), Fourth Generation Evaluation (Guba and
Lincoln, 1989), Realistic Evaluation (Pawson and Tilley, 1997) and Participatory Monitoring and
Evaluation (PM&E) (that fill the journals concerning ‘program evaluation’ and development
evaluation). These models of evaluation appear to offer much to the practice of extension
evaluation, and have potential to address demands for evaluation discussed in Chapter 3. In the light
of the findings of this review, the next step is to conduct some research on the applicability of some
these contemporary evaluation models for extension evaluation. For this reason, in Chapter 5, I
examine these popular models in more depth, and select one model to implement as a case study
(Chapter 7).
107
CHAPTER 5
CONTENDING MODELS OF PROGRAM EVALUATION FOR
AGRICULTURAL EXTENSION
Four contemporary models of program evaluation were examined in terms of the extent to
which they presented a novel approach, able to address both the macro needs of extension
projects nationally and the micro needs of the case study project across which it was to be
tested. The models examined were Patton’s (1997) ‘Utilization Focused Evaluation’, Pawson
and Tilley’s (1997) ‘Realistic Evaluation’, Guba, & Lincoln’s (1989) ‘Fourth Generation
Evaluation’, and Davies’ (1996) ‘Most Significant Change’ model. All four models were
found to have much to offer extension evaluation at a national level. Davies’ (1996) Most
Significant Change (MSC) model was selected primarily as it appeared to complement the
existing evaluation practice of the Target 10 Project and to help address the evaluation needs
of the project. The MSC model also extends current methodology, employing the collection
and selection of stories of significant change. The use of stories could offer an effective way
of creating dialogue and sensemaking between the multiple stakeholders of large extension
projects.
5.1 Introduction
In Chapter 2 I argued that approaches to evaluation should meet certain premises for good
evaluation and should be grounded in a knowledge of the evaluation needs specific to the context.
An analysis of the current context in Victorian extension (Chapter 3) revealed there is a demand for
internal evaluation that can deal with multiple audiences and promote organisational learning and
communication. An analysis of the changing nature of extension revealed the need for evaluation
that can deal with participatory, group-based extension and diverse impacts. A review of current
practice indicated a limited spread of evaluation approaches in use, with a tendency to focus on
108
methods rather than theory, and a surprising lack of project monitoring and evaluation (Chapter 4).
To address these challenges a novel evaluation model was selected, modified, implemented across a
Victorian extension project and evaluated. The aims of this chapter are to describe the selection
process for choosing one promising model of evaluation, to test as part of the empirical work of this
thesis.
I begin the chapter with an explanation of criteria that were used to select an evaluation model.
Following this, the case study project is introduced along with an analysis of the evaluation needs
and gaps for this particular project. I then provide a summary of the demands facing evaluation of
extension projects at a nation level. Next, four contending models of program evaluation are
reviewed against the evaluation needs at both the micro and macro level. Finally Davies (1996)
Most Significant Change (MSC) model, that was selected, will be outlined in more detail, and the
ways in which this model has potential to address the micro and macro evaluation needs will be
elaborated.
5.2 Criteria for Choosing a Model to Pilot
New approaches to evaluations are more likely to be useful if they are appropriate to the project-
context, and complement the project’s existing evaluation practice. The case study for this thesis
involves the implementation of a novel model of evaluation across a large extension project.
However, the aim of this thesis is not only to develop a new model for the evaluation of the case
study project, but also to develop a new model to add to the ‘basket of choices’ for extension
evaluation in Australia. Therefore, in selecting an evaluation model, the challenges identified at the
national level will also be considered.
The review of current practice in extension evaluation (Chapter 4) revealed a limited spread of
evaluation approaches in use. For this reason, I felt that it was important to select a novel approach
to evaluation that had not been widely used in extension previously. Implementing a totally new
approach can stimulate new theory and debate, especially when a limited repertoire of tools is
currently in use. In summary the criteria for selection of an approach to test included that it should:
• be a novel approach that is not currently used in extension evaluation in Australia
• address some of the macro needs of extension projects in Australia
• address the micro needs of the case study project across which it is to be implemented.
109
Macro needs of Extension Evaluation in Australia
In Chapter 3, an analysis of the changing nature of extension organisations and the implications of
these changes for the practice of extension was presented. I suggested that these changes have led to
increased demands for evaluation that can:
• show the extent to which predetermined objectives have been achieved to satisfy accountability
requirements
• provide accountability for group-based models of extension that produce diverse, context-
specific outcomes
• involve appropriate evaluation processes for participatory projects and can foster learning of the
participants and augment the intervention effort
• can be conducted as part of an internal evaluation capacity and hence foster organisational
learning
• provide performance information for communication and between multiple stakeholders
• involve processes that can foster dialogue between multiple stakeholders.
Micro needs of the Target 10 Project
The Target 10 Dairy Extension Project is one of the most established of the new genre of ‘mega’
projects that operate under the purchaser-provider model for publicly-funded extension in Victoria.
It is largely outcome focused and has a productivity goal of increasing pasture utilisation by 10%
(hence the name Target 10). It has a complex structure, and is funded by various federal, state and
industry sources as well as receiving money indirectly from levies on the sale of milk. More details
are provided concerning the background to and aims of this project in Section 7.2.
In 1998, prior to the field work for this thesis, the Target 10 Project was conducting evaluation in
the following four areas:
• evaluation of the ‘logic’ of the five core programs using Bennett’s Hierarchy and subsequent
refinement of the valued outcomes for each core program
• evaluation of each of the five core programs against valued outcomes
• economic impact analysis of the whole project
• reporting against predetermined milestones.
110
The audiences for the whole range of evaluation activities were comprised of the project staff,
purchasers, and committee members (who represented the farmer-clients and other collaborators
from the university and dairy industry).
Yet, the project management still felt that certain areas of evaluation needed to be developed.
Firstly, they wished to learn how to evaluate the overall non-economic impact of the various
initiatives and activities. This included gaining an understanding of any unanticipated or unexpected
impacts. In particular, the project funders and stakeholders alike wanted more information about the
impact that the program was having on individual farmers’ lives. They also felt that an evaluation
approach that fostered some dialogue between the various different funders and stakeholders would
be advantageous. Therefore in 1997, the major evaluation needs of the Target 10 Project were to:
• gain an overarching understanding of the impact (including unexpected outcomes) of the
project on farmers’ lives
• adopt an evaluation process that was capable of fostering dialogue and negotiation between the
different project stakeholders.
These evaluation needs correspond with some of macro needs of agricultural extension projects at
the national level (see Section 3.5). Other evaluation needs at the macro level were already being
addressed by the existing Target 10 evaluation activities. Therefore, an evaluation model was
chosen on the grounds that it met all the needs of the case study project, and some of the evaluation
needs that were associated with the changing nature of extension institutions at the national level.
5.3 Contending Models of Program Evaluation to Test
Four models of evaluation will be examined against these criteria. These models are:
• Patton’s (1997) ‘Utilization Focused Evaluation’, a decision-management approach
• Pawson and Tilley’s (1997) ‘Realistic Evaluation’, a theory-driven approach
• Guba, & Lincoln’s (1989) ‘Fourth Generation Evaluation’, a intuitionist-pluralistic approach
• Davies’ (1996) ‘Most Significant Change’ model, an intuitionist-pluralistic approach (from
development studies).
These four models were chosen on the basis of the work presented in Section 2.4 in which six
111
different approaches to evaluation were examined for what they offered to the concept of good
evaluation (experimental, objectives-testing, judgemental, decision-management, intuitionist-
pluralistic and theory-driven approaches). Decision-management, intuitionist-pluralist and theory-
driven approaches are the most contemporary, and show potential to offer new ways of evaluating
extension projects in Australia. According to the findings of the review of current practice of
extension evaluation (Chapter 4), these approaches had rarely (and only partially) been
implemented previously. Therefore, one popular model was selected from each approach, with the
exception of intuitionist-pluralist approaches, from which two models were selected. This additional
model was selected because the challenges facing development studies have similarities with those
of extension evaluation, and that there might be the opportunity for transferable learning between
the two disciplinary areas.
5.4 Utilization Focused Evaluation (UFE)
The Utilization-Focused Evaluation model was developed by Patton and is presented in his book of
the same title (Patton 1997). According to Patton:
Utilization Focused Evaluation is a system the makes evaluations useful, practical, accurate,
systematic, and ethical. This involves learning to match the evaluation approach and the design to the
information and decision needs of primary intended users taking into account other stakeholders,
political factors, organisational constraints, project/program history, available resources, and cultural
factors of a specific evaluation context. This allows for situationally responsive evaluations (Patton,
2000: 1).
The model is premised on the ability of small co-operative groups to use evaluation information to
help make decisions in order to improve programs (Rogers, 1996). As implied by the title, this
model is primarily concerned with promoting evaluation to be used by decision-makers and project
stakeholders. Patton states that:
Utilization-Focused Evaluation begins with the premise that evaluations should be judged by their
utility and actual use; therefore, evaluators should facilitate the evaluation process and design any
evaluation with careful consideration of how everything that is done, from beginning to end, will affect
use (ibid, 1997: 20 emphasis in the original).
112
In summary, Patton’s Utilization-Focused Evaluation (1997) has three particular features:
• The intended users of the evaluation are personally identified and usually brought together
(1997: 50).
• These intended users decide the focus of the evaluation, the design and the research methods,
and are involved in analysis and interpretation (1997: 376).
• All these decisions are explicitly referred to intended Utilization of the evaluation, by asking the
identified users questions such as ‘How would you use that information if we had it?’ (1997:
376).
Rogers (1996) suggests that the UFE model is fundamentally a theory of evaluation use and does
not include a prescriptive theory of knowledge or valuing. Evaluation is seen as primarily a process
of producing information that will meet an identified need for knowledge. In Patton’s words ‘unless
a lack of knowledge and information is part of the problem, program evaluation will not help an
organisation’ (Patton, 1997: 72).
Definition of evaluation in UFE
Patton defines program evaluation as:
…the systematic collection of information about the activities, characteristics, and outcomes of the
programs to make judgments about the program, improve the program effectiveness, and/or inform
decisions about future programming. Utilization focused program evaluation (as opposed to program
evaluation in general) is evaluation done for and with specific, intended primary users for specific,
intended uses (Patton 1997: 23).
Underpinning paradigm in UFE
Patton appears to subscribe largely to a pragmatic paradigm, although he is rarely explicit about
this. Instead, he refers to a ‘paradigm of choices’. Patton states that the ‘paradigm of choices
recognises that different methods are appropriate for different situations and purposes’ (1997: 279).
This avoidance of prescriptive paradigms is characteristic of a pragmatist stance. Pragmatists see
the research question as being more important than the methods used, or the world view that is
supposed to underlie the method (Tashakkori and Teddlie, 1998). For this reason, the worldview
hardly enters the picture, except in the most abstract sense.
113
Patton, like other pragmatists, rejects the forced choice between positivism and constructivism with
regard to methods, logic and epistemology. Pragmatists advocate a choice of quantitative and
qualitative methods, depending upon the research question currently posed, and the phase of the
research cycle. Pragmatists contend that science inquiry is not ‘formalistic’ and that researchers
may be both objective and subjective in epistemological orientation over the course of studying a
research question (Tashakkori and Teddlie, 1998). Pragmatists believe that values play a large role
in conducting research and in drawing conclusions from their studies. Thus pragmatists decide what
they want to research, guided by their personal value systems, and they study what they feel is
important (Tashakkori and Teddlie, 1998). This is very much in line with Patton’s arguments
A critique of UFE
To whom is the evaluator responsible?
Patton’s description of a Utilization-Focused Evaluation suggests that evaluators are primarily
responsible for meeting clients needs: ‘the purpose of a Utilization-Focused Evaluation is to answer
stakeholders’ questions’ (Patton, 1997: 43). This view contrasts with that of Scriven (1967) who
contends that the evaluator is responsible to the society as a whole, while Wadsworth (1991)
suggests that evaluators should serve the people for whom the program is designed. Patton justified
his position by stating that he is selective about the clients he chooses (by implication, he does not
have to be concerned about the intended program outcomes, since the evaluation clients are well-
intentioned) and some evaluations are done for no charge (so it is not only the well-resourced who
can be assisted with evaluation (Rogers, 1996).
Despite Patton’s reassurances, there are those that still question the ‘cosy’ relationship between the
evaluator and the client (often the project manager) and question the ability of the evaluator to
present unpalatable findings to the client in this circumstance (for example Pawson and Tilley
1997:14).
Lack of judgement of intrinsic merit of the program
Scriven (1991) disputes Patton’s view that the evaluator should act as the provider of information to
clients to help them make decisions, urging Patton to consider the intrinsic merit of the project to be
evaluated. Scriven (1986: 19) states that ‘bad is bad and good is good and it is the job of evaluators
to decide which is which’.
114
Values inquiry
Patton deals with differing values of stakeholders in an evaluation by including them in the
evaluation process. The weakness of this model is likely to be in the representativeness of the
group. Some critique UFE because of the small number of people involved in the reference group.
Action researchers, for example would criticise the model for not explicitly aiming for fuller
representation.
Applicability of UFE for extension evaluation
On the surface UFE does appear to offer a way of meeting many of the micro and macro evaluation
needs outlined. UFE also offers a novel approach in that no examples were found in the review of
extension evaluation (Chapter 4) that explicitly followed Patton’s model (yet several studies did
refer to attempting to incorporate a utilisation focus).
In terms of meeting the macro needs of extension projects in general, the UFE approach can provide
accountability and be conducted as part of an internal evaluation capacity and hence foster
organisational learning. Getting the audience of the evaluation involved in the design and
interpretation of the data ensures an opportunity for dialogue between these people. However, the
UFE model does not suggest ways of involving the beneficiaries of the program in this process,
unless they are considered an audience of the evaluation, in which case they would be represented
in a limited manner in the reference group. The methods employed would be dependent on what the
reference group decided upon, therefore this model could include data collection methods that
would help determine the impact on farmers’ lives and whatever else the group felt was important.
Therefore, UFE does meet some of the needs outlined.
Nevertheless, as Rogers points out, this model provides a theory of maximising the use of
evaluation findings, rather than a comprehensive model of how to conduct an evaluation. This is
verified by Patton’s own words:
Utilization-focus evaluation does not advocate any particular evaluation content, model, method or
theory, or even use. Rather, it is a process for helping primary intended users select the most
appropriate content, model, methods, theory and use (1997: 22).
For example, it offers no guidance as how to deal with conflicting opinions within the reference
group. It offers no suggestions for how to involve the remaining stakeholders who are not members
115
of the reference group. While it can be considered a ‘participatory approach’ insomuch as it
involves staff and funders in the evaluation design, it does not appear to stress the widescale
involvement of the farmer-clients in the evaluation process.
While UFE meets some of the needs outlined at both micro and macro level, it does not extend
evaluation in the area identified as most needed. The existing evaluation of the core programs of the
Target 10 Project already attempts to adopt a partly utilisation-focused approach, and funders and
staff have been involved in developing the evaluation plans from the beginning. For this reason
UFE was not selected to implement as part of the empirical work for this thesis.
Nevertheless, I suggest that UFE is an important contribution to the ‘basket of choices’ for
extension evaluation in Australia. Patton’s UFE provides an important reminder to all evaluators,
that no matter what evaluation model is ultimately adopted, utilisation should be considered from
the onset. It is my belief that any model of evaluation can benefit from a careful consideration of
how the findings will be used. Patton’s (1997) UFE approach has greatly influenced the overarching
concept of an evaluation bricolage. The evaluation bricolage is designed to consider utilisation, in
that it is developed on the basis on the needs of those who will use the evaluation findings.
5.5 Realistic Evaluation
Pawson and Tilley’s ‘Realistic Evaluation’ is based on a realist philosophy of science. It seeks to
evaluate programs and policies within the realities of society. It acknowledges that these programs
take place in an open system, which consists of a ‘constellation of inter-connected structures,
mechanisms and contexts’ (Kazi, forthcoming). One of the central arguments inherent in this model
is that evaluators should go beyond addressing the question: ‘does a program work’, and instead
focus on which programs work in different situations, and why. To accomplish this, Pawson and
Tilley build a model that evolves around the notion of context-mechanisms-outcome configurations
(CMOs), or configurations of mechanisms (M) operating in contexts (C) to produce outcomes (O)
(ibid 1997: 55-83).
In essence, the realist model involves three iterative steps:
116
• develop a plausible explanation for regularities between the program and the expected
outcomes by elaborating the program theory of the intervention (using CMO configurations)
• develop testable hypotheses based on these CMO configurations and test them through multi-
method approaches (i.e., compare the model to reality)
• refine the theory in terms of plausible explanations for the regularities.
The research cycle for Realistic Evaluation begins by generating a theory that is framed in terms of
propositions about how mechanisms are triggered in particular contexts to produce particular
outcomes. The realistic research cycle is presented in Figure 9.
Figure 9 The realist evaluation cycle, source: Pawson and Tilley (1997: 85)
Pawson and Tilley (1997: 216) state that ‘evaluators need to penetrate beneath the surface of
observable inputs and outputs of a program’. They suggest that observable program outcomes are
brought about by a complex range of generative forces that may not be immediately observable, and
it is the evaluator’s job to uncover these mechanisms and contexts. They focus heavily on the need
to consider context, stating that ‘evaluators need to understand the contexts within which problem
mechanisms are activated and in which program mechanisms can be successfully fired’ (ibid: 216).
Theory
Observations
Program
specification
Hypotheses
Mechanisms, Contexts, Outcomes
What might work for whom in what circumstances
Multi-method data collection and analysis on M,C,O
What works for whom in what circumstances
117
Evaluators are encouraged to seek to understand ‘for whom and in what circumstances’ a program
works through the study of the context in which the program works.
Realistic Evaluation also emphasises the need to understand the multiple outcomes of program
initiatives and how they are produced. The model suggests that programs cannot be understood as
undifferentiated wholes, but as having different effects on different subjects in different situations to
produce multiple outcomes. It is the evaluator’s job to understand how these multiple outcomes are
produced.
Definition of evaluation under Realistic Evaluation
In answer to the question of what is evaluation supposed to do, Pawson and Tilley (1997: xiv) state
that they want ‘to return evaluation to its roots in examining the effectiveness of particular social
programs targeted at specific social problems’. Thus evaluation is conducted to improve programs
and policies, to better meet stakeholder needs in order to ultimately ameliorate social problems.
Underpinning paradigm of Realistic Evaluation
Realistic Evaluation is grounded in a realist ontology. A realist ontology implies that there is a real
world that exists apart from human constructions. Realism attempts to incorporate the critique of
positivism without abandoning the concept of a knowable reality. In doing so, realism rejects the
ontological viewpoint posited by the radical constructivism approach, that suggests that there is no
objective reality. Positivism also subscribes to a realist ontology, yet there are some fundamental
differences in the notion of causality and the epistemological stance between the two paradigms.
Realism incorporates the notion of a stratified reality (i.e., a hierarchy of embedded causal orders)
with real underlying generative mechanisms (Julnes et al., 1998). Harre (1986) made a distinction
between successionist and generative theories of causation, stating that the basic task is not to
hypothesise or demonstrate the constant conjunction whereby action X produces outcome Y
(successionist causation), but rather to understand the conditions required for causal potential to be
released in practice. In contrast, positivism asserts that the world is actually equal to what we
observe, that is our experiences gained through sense impressions. Positivism incorporates a
successionist theory of causation or (Humean theory of causation) that stresses that all we can ever
observe is the constant conjunction of events, and that this is all we need to know for empirical
science to be possible.
118
Yet, realism shares with constructivism an epistemological belief that our experience of the world is
not direct, that our perception of reality is mediated by what we bring to the experience, and thus
our experience of reality is constructed (Julnes and Mark, 1998). Realism rejects the foundationalist
epistemology of positivism that sees all scientific propositions as founded on data and facts and in
which hypotheses are to be tested against the facts. Instead realism contends that knowledge is a
social and historical product, facts are theory-laden and the task of science is to invent these
theories to explain the real world and to test these theories by rational criteria developed within
particular disciplines (Manicas and Secord, 1983).
Methods in Realistic Evaluation
Pawson and Tilley (1997) talk of a need to break down the paradigm wars that have raged between
quantitative and qualitative methodologies. They advocate theory-guided evaluation, with the stress
away from a method-guided approach. They see themselves as ‘whole heartedly pluralists when it
comes to the choice of methods’ (ibid: 85). Despite all this, they give few examples of a theory-
guided evaluation that utilises qualitative approaches, although they do not rule this out.
Realistic Evaluation is presented as theory-driven approach rather than method-driven. In the case
of Realistic Evaluation, the theory relates to the CMO configurations, and the task of the evaluator
is to postulate conjectured CMO configurations drawing on folk wisdom of practitioners and formal
knowledge in literature.
A Critique of the Realistic Evaluation
In the following sections Realistic Evaluation is critiqued on the grounds:
• that it does not focus sufficiently on values, or stakeholder concerns
• that is does not focus sufficiently on utilisation of the evaluation findings
• the emphasis on testing hypothetical theories may overshadow an examination of unanticipated
program outcomes
• that insufficient guidance is given on how to choose methods, and how to generate conjectured
mechanisms
• that in some cases, a realist model may not be appropriate: (1) when the cost of conducting a
theory-driven evaluation is not justifiable; (2) when the nature of the evaluation places more
119
emphasis on answering specific stakeholder concerns.
Values
An important component of good evaluation concerns the knowledge that we construct about the
values of those affected by and interested in the programs being evaluated. I suggest that evaluation
in addition to identifying mechanisms in contexts, should include some understanding of the values
of the stakeholders. Indeed, Julnes et al. (1998: 499) argue that evaluators should consider ‘how it is
that the results of evaluative inquiry are to be translated, through the lenses of values, into decisions
about social programs and policies’. Pawson and Tilley (1997) argue that formal theories of values
do not provide a path to deciding what values should be incorporated into a particular evaluation.
This may be the case, but realistic evaluation does not offer any alternative position on how to
incorporate values into the practice of evaluation. Julnes et al. (1998: 484) suggest that ‘realist
theory must grapple with values issues and move beyond the fact-value dichotomy’.
Leviton (1994) states that theory-driven evaluation is most seriously deficient in assignment of
value to community-based programs. Realistic Evaluation does not specifically address how
stakeholders concerns will be addressed. The development of conjectured theories is directed in an
analytical manner, conducted by expert evaluators. This process of theory building does not address
the fact that, even within one sub-group of people, there may be multiple program realities (in
addition to context) and therefore multiple theories of how the program is actually working. An
alternative approach to developing a program theory during evaluation is the Action Science
approach, proposed by Argyris and Schon (1996). In this model Agyris and Schon discriminate
between practitioners’ espoused theories (what they say, or write) and theories of action (what they
do). They also highlight the fact that individual actors may hold quite distinct ‘theories of action’,
which when shared can help organisation learning. Unlike the theory advanced by Pawson and
Tilley, in Action Science these theories are not ‘collected’ but are constructed by practitioners to
understand and improve their practice.
In certain program contexts, Realistic Evaluation could perhaps be more useful if values and
alternative theories of action were addressed more fully.
Utilisation
Realistic Evaluation is not overtly concerned with issues such as how to foster ownership of an
120
evaluation study. Patton (1997) emphasised that the findings of many evaluation studied seldom get
utilised, and suggested that the stakeholders of projects must be incorporated into the evaluation
process, for the evaluation findings to be owned and ultimately acted upon. Realistic Evaluation
does not fully address Patton’s concerns (Patton, 1999).
Unanticipated/ unexpected outcomes
Realistic Evaluation does not specify how unintended or unanticipated outcomes are to be
investigated. This is common to approaches of evaluation that are based on testing key elements of
a conjectured program theory. In my experience of constructing program theory models (or program
logic) there is a tendency to get drawn into the micro mechanisms that have been postulated,
neglecting to look at the data afresh for new emerging patterns or mechanisms. Hamilton-Smith and
Hopkins (1998) suggested that in the Realistic Evaluation, no overt demand is made to include
unintended consequences once plausible and rival explanations are developed.
Clear guidance for implementation
Pawson and Tilley (1997) are vague about how to generate the conjectured theory and whose
theories should be included. There are a large number of mechanisms that could be responsible for
an outcome, and a vast number of potential outcomes. Perhaps at this stage, stakeholders could
become actively involved in deciding which theories should be tested. In three case study examples
presented in their book, researchers assembled the key hypotheses by drawing on the folk wisdom
of practitioners as well as the formal knowledge of the academy. But Pawson and Tilley do not
suggest whose views should be sought, or who should be given priority. Thus, the Realist model
does not provide clear guidance on how to generate conjectured mechanisms; it does not elaborate
clearly how evaluators should identify conjectured mechanisms nor prioritise which mechanisms
should be investigated.
In addition, Realistic Evaluation does not provide clear guidance on how methods should be
strategically selected in relation to the theory that is to be tested.
Contexts when Realistic Evaluation may not be appropriate
In some cases, depending on the nature of the evaluation questions, Realistic Evaluation may not be
appropriate. Firstly, as argued by Scriven (1994), there are situations when the cost benefit of
conducting a theory-driven evaluation may not be justifiable in terms of the benefits gained.
121
Secondly, in highly participative programs there may be instances when the nature of the evaluation
questions rightfully places more emphasis on answering specific stakeholder concerns than on
investigating carefully crafted analytical suppositions.
Summary of review of Realistic Evaluation
Despite the criticisms levelled at Realistic Evaluation, I suggest that Realistic Evaluation is a very
important contribution to program evaluation theory, offering a comprehensive and internally
consistent model for conducting evaluation. However, despite the claims that Realistic Evaluation
offers a holistic approach to evaluation, there are situations where it might be less practical and
helpful than other models of evaluation.
I suggest that the way in which the initial postulated CMO configurations are developed could be
strengthened. Pawson and Tilley suggest in an ad hoc manner that folk wisdom be consulted in
addition to literary knowledge. It is my suggestion that this step of Realistic Evaluation could be
enlarged, and directed largely by the stakeholders who are affected by program being evaluated.
Some of the work by Argyris and Schon (1996) could be incorporated at this step, and an effort
made to prioritise between the rival conjectured hypothesis (that will be tested later in the research
cycle) and that this stage consider values.
Applicability of Realistic Evaluation for extension evaluation
Realistic Evaluation is a novel approach to evaluation for extension evaluation, and no examples of
this model were seen in the review of extension evaluation in 1997 (Chapter 4). This model has
potential to meet many of the needs of agricultural extension projects in Australia. However, at the
micro level, it does not extend evaluation in the area identified as most needed.
Realistic Evaluation is able to meet many of the identified evaluation needs of agricultural
extension projects. It provides guidance for evaluation that can provide accountability for group-
based models of extension that produce diverse, context-specific outcomes. It could also potentially
be conducted as part of an internal evaluation capacity and hence foster organisational learning.
Indeed, it seems to have much to offer in terms of making sense of different outcomes in different
program contexts. Realistic Evaluation also has potential to provide performance information for
communication and between multiple stakeholders. Nevertheless, while it involves several different
sub-groups of stakeholders in testing conjectured theory, it does not provide a process for
122
facilitating this diologue, and provides no process for involving the beneficiaries or service users in
guiding the evaluation process.
At the micro level of the case study project, Realistic Evaluation does not offer evaluation processes
identified as most needed. The articulated evaluation needs of the project management were to gain
an overarching understanding of the project impact (including unexpected outcomes) and to adopt
an evaluation process capable of fostering dialogue and negotiation between the different project
stakeholders. Firstly, while Realistic Evaluation does go some way in providing information to help
stakeholders gain an overarching understanding of the impact of the project on farmers’ lives, I
have argued that it is not necessarily good at investigating unexpected outcomes. Secondly, as
mentioned above, Realistic Evaluation does not provide a process for fostering dialogue and
negotiation between project stakeholders. Realistic Evaluation is not intended to meet these
evaluation purposes. Thirdly, the existing evaluation of the Target 10 Project already included a
partly theory-driven approach – the logic of the core programs was analysed and the objectives
refined using Bennett’s hierarchy of outcomes. For these reasons the Realistic Evaluation model
was not selected for implementation across the case study project.
Nevertheless, I suggest that Realistic Evaluation would add much to the ‘basket of choices’ for
extension evaluation nationally and is a fairly comprehensive model to guide evaluation. Despite
the fact that Realistic Evaluation was not chosen, it has influenced this thesis at two levels. Firstly,
the notion of moving beyond whether a program works and focusing instead of what ‘works for
who in what situation and why’ is a central feature of the conceptual framework for good evaluation
that was developed in Chapter 2. Secondly, Realistic Evaluation was used to guide the actual meta-
evaluation (described in Chapter 8).
5.6 Fourth Generation Evaluation (FGE)
Fourth Generation Evaluation is a pluralist-intuitivist approach to evaluation presented in a book by
Guba and Lincoln of the same name (1989). Guba and Lincoln advocate a constructivist approach
to evaluation, and contend that realities are social constructs of the mind, and that there is no one
objective reality. The key emphasis of their model is on the process of negotiation, incorporating
various stakeholders more centrally into the evaluation process. Guba and Lincoln (1989) lay out
several premises for their model.
123
Firstly they state that evaluation outcomes are not descriptions of some ‘true’ state of affairs, but
instead ‘present meaningful constructions that individual actors or groups of actors form to make
sense of the situations which they find themselves in’ (1989: 8). As these constructions are shaped
by the values of the constructors themselves they are ‘inextricably linked to the particular physical,
psychological, social and cultural contexts within which they are formed and to which they refer’
(1989: 8). Therefore, colleagues who work in the same context may share the same constructions of
an event. But, reaching consensus does not imply that these constructions are any more true or valid
than other constructions.
Guba and Lincoln (1989: 9) contend that ‘evaluations can be shaped to enfranchise or
disenfranchise stakeholder groups in a variety of ways’. This includes who gets consulted in the
evaluation, who designs the evaluation, who interprets the findings and decides what is included in
the report, and to who the evaluation findings are disseminated. The evaluator should ‘interact with
humans in a manner that respects their dignity, their integrity and their privacy’ (1989: 10). In this
respect, stakeholders ‘are accorded a full measure of political parity and control’ in the evaluation
(1989: 11). In terms of commitment to utilise the evaluation findings, it is suggested that
‘evaluation must have an action orientation that defines a course to be followed’ (1989: 10). This
course of action should be negotiated between the stakeholders and should include a commitment to
follow-up the findings of the evaluation. The evaluator, therefore, plays the role of ‘orchestrator of
the negotiation process’ (1989: 10).
Definition of Evaluation under FGE
Evaluators employing Fourth Generation Evaluation judge evaluation by the extent to which it
succeeds in involving a range of participants in sharing their views. Guba and Lincoln (1989)
advocate that evaluators ought to strive for, not validity, but increasingly sophisticated constructions
of reality.
Underpinning paradigm of FGE
Fourth Generation Evaluation (FGE) employs a relativist ontology, asserting that there exist
‘multiple, socially constructed realities ungoverned by any natural laws, casual or otherwise’ (Guba
and Lincoln 1989:84). Constructivists deny that there is any reality separate from experience.
Instead, individual interpretations are thought to be all that exists. In contrast to an empiricist’s
objective stance, FGE subscribes to subjectivist epistemology. This implies that the ‘inquirer and
124
the inquired into are interlocked in such a way that the findings of an investigation are the literal
creation of the inquiry process’ (Guba and Lincoln 1998: 84).
Methodology of FGE
Guba and Lincoln (1989) advocate a ‘hermeneutic’ methodology that involves a ‘continuing
dialectic of iteration, analysis, critique, reiteration, reanalysis’, leading to a joint construction of the
case. They state a preference for qualitative methods to be employed.
A Critique of FGE
In the following sections Fourth Generation Evaluation will be critiqued for a number of reasons,
which include:
• limited documented examples of practice
• not applicable to all program contexts
• methodological dogmatism and buying into the ‘paradigm war.’
Lack of practical examples
Fishman (1992: 267) points to the lack of documented case studies demonstrating the Fourth
Generation Evaluation model. Despite all the literature, it appears that there have been few
publications documenting instances of these approaches actually being implemented. This raises
questions of how practical the FGE model really is.
Not applicable in all program contexts
To conduct a Fourth Generation Evaluation, evaluators must caution those who pay for the
evaluation that there are many realities. They will produce no recommendations, and will be
hesitant at drawing singular conclusions that could be construed as a ‘totalising narrative’. Guba
and Lincoln (1989: 45) state:
Evaluation data derived from constructivist inquiry have neither special status not legitimation; they
represent simply another construction to be taken into account in the move toward consensus.
For many clients, this sort of output from an evaluation would be unacceptable. I have seen few
tenders for the conduct an evaluation that could accommodate these stipulations. Kaplan (1994)
125
suggests that the emphasis must be on making the methodology fit the needs of the society, its
institutions and its citizens, rather than the reverse. There would be many instances when it would
not be acceptable to conduct an evaluation under the Fourth Generation Evaluation model. This is
especially true for the evaluation of large-scale extension projects that are the context for this thesis.
Indeed, Pawson and Tilley (1997: 20) suspect that Fourth Generation Evaluation ‘belongs to the
cosy suburban work of the meeting between school governors, teachers and parents in which they
thrash out homework policy for the fifth grade’. While this criticism may be unduly harsh, it serves
to point out that not all program contexts and evaluation requests could accommodate this model.
Another view on this is that models such as FGE could be usefully combined with other approaches
to meet the needs of those who request an evaluation study to be undertaken. However, Guba and
Lincoln argue against the combining of paradigms.
Methodological dogmatism and the paradigm war
The field of program evaluation, along with many of the social sciences, has become embroiled in a
‘paradigm war’. The opposing sides have variously been referred to as logical
empiricism/positivism, versus radical constructivism (Caracelli and Greene, 1997). While this war
may have created some important insights, it has also led to apparently unnecessary and unhelpful
conflicts (Julnes et al., 1998). Pawson and Tilley (1997) review the history of evaluation in terms of
this paradigm war and conclude that each of the opposing extremes was based on assumptions that
most evaluators (and others) would reject. Indeed, the battle of the paradigms seems to be fought in
academia, rather than between those who actually implement and practise evaluation. Many
evaluation textbooks encourage the would-be evaluator to decide which side they will take,
constructivism or positivism? For example, Guba and Lincoln cautions the reader:
No accommodation is possible between positivist and constructivist belief systems as they are now
formulated. We do not see any possibility for accommodation if that accommodation is to occur by
having one paradigm overwhelm the other by the sheer power of its arguments, or by having
paradigms play complementary roles, or by showing that one is simply a special case of the other
(Guba and Lincoln 1989: 17).
Despite these strong words, very few evaluation theorists advance a positivist stance, and very few
examples of radical constructivism (i.e., Fourth Generation Evaluation) have ever been
implemented (Fishman 1992). In reality most evaluation exhibits qualities of an intermediate stance.
In search of ‘peace’ in this paradigm war, some authors have suggested that a new paradigm is
needed, such as realism (Julnes et al., 1998; Pawson and Tilley, 1997) or pragmatism (Tashakkori
126
and Teddlie 1998).
Applicability of FGE for extension evaluation
FGE offers a relatively novel approach to extension evaluation that is able to address the micro
needs of the case study project, and addressing some of the needs at the macro level.
No examples of FGE were found in the review of evaluation Australian agricultural extension
(Chapter 4), although since then there have been a few studies that have certainly been influenced
by this model. For example, Roberts (1998) ‘coexistive evaluation’ of a Landcare initiative in
Queensland, and an evaluation of the Human Resource Development across the dairy industry in
Victoria (Virtual Consulting Group, 1999). However, as far as I am aware, to date the FGE model
has not been implemented in its entirety in agricultural extension. For this reason it represents a
novel approach for extension evaluation.
FGE is able to address some of the macro needs of extension evaluation. This evaluation model is
not so much focused on accountability but on negotiation and dialogue between the different
stakeholders in the evaluation. Therefore it offers processes for negotiation, but the focus is on
investigating stakeholder concerns, rather than conducting verifiable data for accountability
purposes. FGE can be conducted as part of an internal evaluation capacity (perhaps with the
addition of an external facilitator) and has the potential to foster organisational learning. FGE offers
a participatory approach to evaluation in that many stakeholders are involved in articulating their
concerns. This generally includes the beneficiaries or services users as well as specifically seeking
views of antagonists of the program.
In terms of the micro needs of the case study project, FGE is the first model that has been examined
here which appears to fully specify a process capable of fostering dialogue and negotiation between
the different project stakeholders. In terms of offering an approach that can help stakeholders
understanding of the impact, (including unexpected outcomes) if this was one of the articulated
concerns of the stakeholders then this could be investigated. In terms of capturing unexpected
negative impact, the inclusion of antagonists of the program into the evaluation, ensures that
negative views are heard. FGE offers a constructive way to involve the views of antagonists into the
evaluation, and co-opt them into the evaluation process itself.
127
In summary, the FGE model is able to meet the criteria of novelty and answering all the micro
needs of the case study project, and some of the macro needs of extension evaluation nationally. It
is suggested that FGE would contribute to the basket of choices for extension evaluation in
Australia.
However, in this instance, it is questionable whether the adoption of this approach would combine
with other approaches used in the Target 10 Project. If it was implemented wholeheartedly, and the
underpinning paradigm adopted, then it could be argued that the findings would be incommensurate
with the more objectives-based and economic models that are already practised in the case study
project. It is also questionable whether this is a practical model of evaluation, as very few instances
of this model have ever been implemented in any disciplinary field. Despite the fact that FGE meets
all the selection criteria, it was placed second to the MSC model of evaluation for applicability to
the case study project.
5.7 Most Significant Change (MSC) Model
Davies’ (1996) ‘Most Significant Change’ (MSC) model was originally named the ‘Evolutionary
Approach to Organisational Learning’. It is a qualitative approach to PM&E that involves the
collection and systematic interpretation of stories of significant change. The MSC model aims to
facilitate learning between the ‘layers’ of a large project and is based on a purposive sampling
system. Davies developed the model in an attempt to meet some of the challenges associated with
evaluating a social development project with much diversity. The structure of the MSC model was
built upon concepts of evolution theory, such as ‘survival of the fittest’, through the promotion of a
system of competitively selecting ideas (in the form of stories).
The MSC model is participatory, as many of the project stakeholders are involved in deciding the
sorts of change to be recorded. Unlike conventional approaches to monitoring, the MSC model does
not employ quantitative indicators, but is a qualitative approach. Davies suggests that quantitative
approaches to monitoring extension programs are heavily influenced by a planning ethos that places
substantial emphasis on prediction and control and that MSC is in many respects, the opposite. In
essence, there appear to be four main facets of the MSC model:
• A subjective iterative process: in which the identification of differences in interpretation is
central to the process; difference are brought to the surface and explored, not ruled out.
128
• Participatory definition of indicators: those closest to the experience being monitored (e.g.,
the field staff) are given the right to pose a range of competing interpretations of those events
without being constrained by pre-determined indicators.
• Participatory analysis of data: information is not stored or processed on a centralised basis
but is distributed throughout the organisation and processed locally. Staff not only collect
information about events, but also make their own evaluation of that information, according to
their own local perspective.
• Dynamic, rather than static indicators: Events reported reflect both a changing world and
changing sets of perceptions within the members of the organisation about what is important
within their world. For this reason, the indicators are ‘dynamic’, changing as the organisation
learns and focuses on different aspects.
The research cycle involves three main phases of action:
• establishing several ‘domains’ of change
• developing a of process based on the existing hierarchical structure of the project to collect and
review stories of change
• implementing an on-going process of collection and selection of stories.
In the first stage of the process, the evaluation audience identifies the ‘domains’ of change they
think need to be monitored at the project level. This involves a process whereby senior staff in the
project hierarchy identify broad domains, for example ‘changes in people’s lives’. According to
Davies (1996) unlike performance indicators these domains of change are not precisely defined, but
are left deliberately loose and it is initially up to field staff to interpret change belonging to any one
of these categories.
The second stage of the process involves the collection of stories of significant change (according to
the defined domains of change). The stories are collected from those most directly involved (i.e.,
farmers, extension staff and field workers). The stories are collected with the help of a simple
question in the form: ‘during the last month, in your opinion, what do you think was the most
significant change that took place in the lives of people participating in the project?’ The respondent
is then asked to give an answer in two parts. The first part is descriptive, answering the questions
‘what happened?’, ‘who was involved?’, ‘where did it happen?’, ‘when did it happen?’ In the
second part the respondent is asked ‘why do you think the change is the most significant out of all
129
the changes that took place in that month?’ and ‘what difference has it made, or will it make in the
future?’ The response to the question ‘what happened?’ generally comes in the form of a story.
These stories are reviewed on a monthly basis (see Figure 10). Each level of the program hierarchy
(i.e., field workers, regional staff, state level staff and funders) are involved in reviewing a series of
stories and selecting those that they think are the most significant accounts of change. They are then
required to document which stories they selected and what criteria they used. Davies (1996: 4)
states that ‘the monitoring system should take the form of a slow but extensive dialogue up and
down the project hierarchy each month’.
Level 3 Project funders
Level 2 State level
Level 1 Region 1 Region 2 Region 3 Region 4
Stories collected by those working directly with farmers and brought to monthly staff meetings
Represents a single story
Figure 10 Diagram to represent monthly flow of stories in MSC model for a hypothetical project
Quantification can take place at two stages. Firstly, when a ‘story’ of change is being described it is
quite possible to include quantitative information as well as qualitative information. Secondly, it is
possible to quantify the extent to which changes identified as the most significant in one location or
zone have taken place in other locations, within a specific period. However, no need is seen to
repeat this particular question every month thereafter, as in conventional monitoring systems.
At the end of each year a document is produced containing all the stories selected at the state level
over the period. The stories are accompanied by the criteria that were used to select the stories. The
document contains four chapters, each presenting three to four stories of each of the ‘domains’ of
130
change that are to be monitored by this process. However, this document is not intended to be a
final product, merely a point at which to include the funders in the process of review.
Definition of evaluation in the MSC model
Davies (1996) does not explicitly specify how the MSC model is purported to bring about better
programs. But it is clear that the MSC model was designed to enhance organisational learning.
Davies is a proponent of evolutionary theory to guide organisational learning. Campbell (1969) and
Nelson and Winter (1982) draw from the Darwinian language of evolution, adaptation, and natural
selection to explain organisational change. Under the evolutionary view of organisational learning,
whole organisations or their components adapt to changing environments by generating and
selectively adopting organisational routines.
Underpinning paradigm of the MSC model
While Davies is very much influenced by an evolutionary epistemology, he does not make the
underpinning paradigm (in terms of what can be know about reality) explicit. However, the MSC
model has much in common with constructivists’ subjective epistemology, in that the focus is on
the process of increasing stakeholders’ understanding of both the program and others’ views about
it. For example, in the MSC model, project stakeholders interpret their experiences with the project
and select instances of what they consider to be the most significant change to record as a story.
They are also required to record why this change is significant to them. Thus, when a beneficiary
tells a story of significant change, she/he interacts with the world and draws meaning from it, and it
is in the telling of the story that meaning is constructed. Then, when reviewers read and evaluate
the story, they engage with it and construct a further new meaning. When this is done in a group,
this construction may be shared. In the MSC model the criteria that are used to interpret the story
are clearly documented, made transparent and attached to the story itself. It is this transparency that
makes the whole process even more open to new and more sophisticated constructions of meaning,
because in the MSC model we know who selected each story, in what context, and for what reason.
However, Davies (1996) also subscribes to a verification stage, whereby the stories can be ‘policed’
to see if the event described in the story really occurred. This suggests that Davies does not
subscribe to a radical constructivist ontology, where ‘facts’ are considered to be a function of
multiple realities as much as values.
131
The MSC model employs inductive logic, where indicative events are abstracted out of recent
experience, and this process is renewed with each new reporting period of the monitoring system.
Instead of being deductive, it is open-ended. In terms of causation, Davies makes little attempt to
analyse the causality of the outcomes presented in the stories, or to analyse the stories en masse.
The analysis is purely based on the interpretations of the participants in the process.
Methods of the MSC model
When conventional monitoring data are analysed, they are in a form and location that strips them of
context. Typically, few text comments accompany statistics sent up from field workers. As in many
qualitative approaches, the MSC model makes use of ‘thick description’ (Geertz, 1973); closely
textured accounts of events, placed in their local context, and where the role and subjectivity of the
observer is visible. In the world of ordinary people, these often take the form of stories or
anecdotes. Within the MSC model to monitoring outlined here, these stories are accompanied by
their reviewers’ interpretations.
Within the MSC model, experience is summarised by selection rather than by inclusion; it focuses
on the exceptional rather than the common experience; and it seeks to differentiate rather than to
homogenise. Davies (1996) suggests the approach attempts to define the meaningful edges of
experience rather, than to identify central tendencies.
Critique of MSC model
The MSC model can be critiqued on the grounds that:
• in some contexts it would be inappropriate (a) where the project is small and does not have
‘layers’ (b) where the information produced by the process does not satisfy the evaluation
client’s needs
• under a realist epistemology the approach could be criticised for presenting a biased
representation of reality
• aspects of the process are overtly biased towards the view of project staff and the views of
antagonists of the project are not deliberately sought.
132
Limited contexts where the MSC model would be feasible
The MSC model aims to facilitate learning between the ‘layers’ of a large project. Clearly for
smaller scale projects, with just one hierarchical layer, this model would not be applicable in its
current form. This model is also very time consuming and requires high levels of participation that
might not be appropriate for all evaluation endeavours.
In addition the MSC model does not describe the average experience of the project participants. For
some evaluation clients, sampling the ‘outer edges of experience’ might not provide sufficient
information for accountability.
Biased representation of ‘objective’ reality
Under the lens of positivism, the MSC model can be criticised strongly for providing a poor, biased
representation of objective reality. Therefore, it cannot be seen as a valid stand-alone model to
objectively assess the overall impact of the project. However, this criticism depends very much on
the epistemological stance the critic adopts. Alasuutari (1995) makes a useful distinction between
two perspectives in qualitative research – the factist perspective (realist epistemology) and the
specimen perspective (relativist epistemology).
• Factist perspective (realist) – makes a clear cut division between the work or reality ‘out there’
and the claims made about it. A researcher with a factist perspective will need to consider how
truthful the words that the informants use are. The characteristics of language and of the
situation are only taken into account as possible noise in the channel through which information
about the world is conveyed, or as distortions in the lens through which the reality is observed
(paraphrased from Alasaturri 1995: 47).
• The specimen perspective (relativist) – the material is scrutinised in its own right. Truthfulness
and honesty are not relevant notions within which to approach the material. A specimen as a
form of research material is not treated as either a statement about or a refection of reality –
instead a specimen is seen as a part of the reality being studied. Thus honesty is an irrelevant
concept to be used in assessing the material (paraphrased from Alasaturri 1995: 63).
Under what Alasatuuri calls the ‘specimen approach’, the stories represent reality in their own right,
and it is in the telling of the stories that the storytellers and reviewers construct reality. In addition
133
to this, the interpretations that the tellers and the reviewers provide give strong clues as to how these
actors construct reality. The sharing of the stories and of the interpretations made of these stories
allows a new, shared and (hopefully) more sophisticated reality, to be jointly constructed by the
storytellers, reviewers and the story readers. In this sense then, the process helps stakeholders
apprehend reality (in its variously constructed forms) in order to make decisions.
Yet under the factist perspective, the stories themselves require crosschecking and may not
represent the best way of accurately determining what events have occurred.
Biased towards the perspective of project staff
The MSC model is situated within the existing structures of power that make a difference on a day
to day basis – not in an egalitarian setting. The sampling system privileges people participating in
the program, so it excludes the views of the antagonists of the program who might not be
participants. The interpretation of the stories is also deliberately biased towards the views of those
who interpreting the stories. Another criticism of this model (as levelled on all methods that strive
for consensus), is that particularly harsh or unpopular views may be silenced by the majority vote.
5.8 Selection of Evaluation Model to be Tested
Davies’ Most Significant Change (MSC) model for participatory monitoring and evaluation
(PM&E) was selected as the basis for developing a new model to evaluation for extension. In the
following section it will be argued that the MSC model:
• Addresses macro needs - has potential to address some of the identified evaluation needs of
Australian agricultural extension projects
• Addresses micro needs - has potential to adequately address all the evaluation needs of the case
study project
• was developed from the field of development projects which faces similar challenges common
to those of Australian extension projects
• is a highly novel in approach and methods, particularly in the collection and selection of stories
• involved the participatory interpretation of stories which is a novel way of helping stakeholders
to make sense of impact.
134
Addresses macro needs
The MSC model addresses some of the macro needs of extension projects nationally. It does not
directly attempt to verify whether valued objectives have been achieved. However, it can provide
accountability in a more ‘goal-free’ manner which could be particularly appropriate for group-based
models of extension that produce diverse, context-specific outcomes. Of all four models presented it
offers the most participatory approach which is directly orientated to foster learning by the
participants and augment the intervention effort. It was also designed to be conducted as part of an
internal evaluation capacity. The stories themselves provide performance information for
communication between multiple stakeholders.
Addresses micro needs
In terms of addressing the micro needs of the case study project, the MSC model seems very well
suited. Firstly, it offers ample opportunity for stakeholders to gain broad understanding of the
impact (including unexpected outcomes) of the project on farmers’ lives. The stories specifically
focus on the impact of the project on individual farmer lives, and the selection process is orientated
towards discussing and interpreting these stories. The MSC model also offers a process capable of
fostering dialogue and negotiation between the different project stakeholders. As in FGE, many
stakeholders from farmers to staff and funders of the project are involved in interpreting the stories
and providing feedback on these interpretations to all the other stakeholders.
Development projects face similar challenges to extension projects
International development projects include projects that concern agricultural development, as well
as health, social community development. Unlike the majority of the literature from program
evaluation per se, the evaluation literature of development studies tends to focus on internal
evaluation of externally-funded, participatory, group-oriented projects. This appears to have much
structural commonality with Australian agricultural extension projects. Many of the challenges
facing the evaluation of development projects appear similar to those facing extension projects in
Australia.
Davies developed the MSC model in an attempt to meet some of the challenges associated with
evaluating a complex micro-credit project in Bangladesh, which had both diversity in
implementation and outcomes. The project concerned was run by the Christian Commission for
Development in Bangladesh (CCDB), a Bangladeshi non-government organisation with almost 550
135
staff and an annual budget of approximately US$4 million. According to Davies (1996) the main
program CCDB runs is the Peoples’ Participatory Rural Development Program (PPRDP), that
involves more than 46000 people in 785 villages in 10 districts. Approximately 80% of the direct
beneficiaries are women. Development assistance is made available to participants in three forms:
group-based savings and credit facilities used to meet the needs of individual households, grant
assistance given to the same groups on a pro-rata basis and intended for community level
developments, and skills training, mainly for livelihood purposes. Davies (1996: 1) states that ‘the
large scale and open-ended nature of these activities poses a major problem for the design of any
system intended to monitor process and outcome’.
Thus the model was designed to be implemented across a large complex project with hierarchical
structures such as are emerging in Victorian extension under the purchaser-provider model (Section
3.4). The MSC model directly addresses the issues of how to involve many different stakeholders in
evaluation, which was one of the issues emerging as problematic for extension evaluators. The
MSC model also has built-in mechanisms for capturing unexpected change, another factor that was
stated as important by investors in extension. In addition, the participatory nature of the model is
important, as extension projects in Victoria place considerable emphasis on farmer ownership and
participation in extension projects.
Novelty
Implementing a new approach can stimulate new theory and debate, especially when a limited
repertoire of tools is in use. The MSC model represents a novel approach to test for several reasons.
Firstly, as a ‘monitoring’ approach, it represents a form of evaluation that is very rarely practised in
Australian extension (see Chapter 4). Julnes et al. (1998) suggest that monitoring is an important,
and growing form of evaluation, largely ignored by evaluation theorists. Secondly, the MSC model
is particularly novel as it is an ‘indicator-free’ form of monitoring, and challenges many of the ways
that conventional monitoring is conducted. Thirdly, the collection and selection of stories offers a
radically different approach to monitoring, that it offers much in terms of helping stakeholders
make sense of both negative and positive impact.
Stories can help stakeholders make sense of diverse outcomes
The third premise for selecting the MSC model was that it involved the collection and collective
interpretation of stories, which has been shown to be particularly effective as a ‘sensemaking’ tool.
136
Stories are used widely for co-operative inquiry and for discourse analysis, feminist research and
cultural studies. In organisational learning literature, stories are valued and studied as the preferred
sense-making currency (Weick, 1995). However, there appears to be little systematic, formal use of
stories in program evaluation. Nevertheless, program evaluation frequently involves the collection
and interpretation of stakeholder stories to make some decision or other regarding a program.
These ‘stories’ emerge during interviews (often embedded in transcripts) and in written documents
such as diaries or open-ended responses to questions. But paradoxically, there is a dearth of
literature that specifically cites the use of stories for evaluating programs.
A possibility is that the under-use of the term ‘story’ in program evaluation relates to the dubious
value in terms of revealing the truth that storytelling implies. As children we are asked by adults
whether we are telling stories – there is an implicit notion in the term ‘telling stories’, that links it to
telling fibs. In addition to truth, accuracy can also be called into question; storytelling can conjure
up the vision of the game of ‘Chinese whispers’: where a story moves around a group of people,
constantly changing and being reinterpreted as it passes from mouth to mouth. Understandably,
evaluators may question the value of collecting stories told in casual conversation, for eliciting
factual content and accurate description of events.
However, stories told in casual conversation can harness another sort of information; they provide
insight into how storytellers construct reality, and to what they attach importance. In the
organisational learning literature, stories told in casual conversation are recorded and studied by
researchers to understand organisational culture (Boje, 1991). However, they are not usually used to
drive change, or to evaluate the effectiveness of an intervention. I suggest that when the collection
of stories is coupled with a process of systematically and collectively interpreting these stories,
(including documentation of these interpretations) then storytelling can be effectively harnessed for
participatory evaluation. The interpretations themselves tell another story, and the process of
collective interpretation can have several beneficial outcomes for evaluation utilisation. Through
adding the extra step of collectively sharing and interpreting stories of program impact, a whole
new dimension to the use of stories in program evaluation is added.
Another factor contributing to the lack of use of the term ‘stories’ in program evaluation literature
probably relates to a lack of clarity with regard to what constitutes a story. In the wider literature,
stories are frequently defined by their structure. Most scholars treat narratives (or stories) as
discrete units, with clear beginnings and endings, as detachable from the surrounding discourse
137
rather than as situated events (Riessman, 1993). In Poetics, Aristotle said that stories have a
beginning, middle and end. Since that time there has been general agreement in the literature that
sequence is necessary for stories (Polanyi, 1985).
Across the literature, differences between the terms ‘story’ and ‘narrative’ are blurred and these
terms appear to be exchangeable. There is disagreement about the precise definition of a story or
narrative. According to Prince (1973), the simplest story contains three interrelated events, the first
and third state a certain situation, while the second is active. The third event depicts a change in the
state of affairs as compared to the first event. An example of this type of story is: John was a
confident driver, until he was in a crash, then he lost his confidence. However, there is a general
agreement that stories are used by the tellers to make sense of a situation. Riessman (1993) suggests
the ‘story metaphor’ emphasises that we create order and construct texts in particular contexts.
Lessons about the value of stories from the sensemaking literature
Organisational researchers emphasise the value of studying stories told in casual conversation
within organisations as part of understanding sensemaking. Louis (1980) suggests that sensemaking
is a process that uses retrospective accounts to explain surprises and make sense of change. Weick
(1995) one of the most widely cited authors in this field, suggests that the central questions for
people interested in sensemaking are how they (the agents) construct what they construct, why and
with what effects.
If storytelling is considered to be an indigenous sensemaking system in organisations, then this
medium would also seem to be suited to collectively making sense of the impact of program
interventions. Just as staff use stories told in casual conversation to make sense of surprises
(Weick, 1995), such as someone being fired in a business firm, the natural storytelling process can
be harnessed in a structured way to help practitioners make sense of the complex nuances of impact
associated with program intervention.
Storytelling is also important in terms of organisational memory. Shaw et al. (1998) contend that
stories are central to human intelligence and memory – that a good story defines relationships, a
sequence of events, cause and effect, and a priority among items – and those elements are likely to
be remembered as a complex whole. If stories about the impact of interventions can infiltrate the
collective memory of an organisation, practitioners will gain and retain a more deeply shared
understanding of what is being achieved. This could create a common base to enter into dialogue
138
about what is desirable in terms of expected and unexpected impact. Boje (1991) asserts that in
complex organisations, part of the reason for storytelling (in casual conversation) is the working out
of those differences in the interface of individual and collective memory.
5.9 Considering all Four Models
The analysis of the four contemporary models, revealed that while all the models make important
contributions, they all have certain limitations. In all cases the models would not be applicable for
every evaluation context and all contain inherent bias. This supports a central argument of this
thesis, that no single model of evaluation is sufficient for all the evaluation needs of large complex
extension projects and that combining models and methods can help off-set the bias of individual
tools. Each large project may need a ‘tailor-made’ bricolage of different methods and approaches to
meet all the needs of the people who request and use the evaluation findings. At a national level, I
argued that what is needed is a ‘basket of choices’ of evaluation models and methods from which
extension professions can pick and choose. All four models reviewed in this chapter have potential
to add to this ‘basket of choices’ and improve the quality and usefulness of program evaluation as it
is practised in Australian agricultural extension institutions.
While only one of the four models examined will be investigated empirically, it should be pointed
out that all of the models have influenced the thinking and framework of this thesis in some form or
other. Patton’s (1997) UFE approach has greatly influenced the overarching concept of an
evaluation bricolage. The evaluation bricolage is designed to consider utilisation, and was
developed on the basis on the needs of those who will use the evaluation findings. Pawson and
Tilley’s (1997) Realistic Evaluation has also influenced this thesis at two levels. Firstly, the notion
of moving beyond whether a program works and focusing instead of what ‘works for who in what
situation and why’ is a central feature of the conceptual framework for good evaluation that was
developed in Chapter 2. Secondly, Realistic Evaluation was used to guide the actual meta-
evaluation (described in Chapter 8). Guba and Lincoln’s (1989) FGE model was used to help
understand how Davies’ Most Significant Change model was intended to bring about an improved
program. At the time of the fieldwork (1997-1999), little was written on the MSC model, as Davies’
thesis was not published until 1998. As both the models are based an intuitionist-pluralist approach,
the fuller explication of FGE model contributed to my understanding how the MSC approach was
intended to work.
139
5.10 Conclusions
Four promising models of program evaluation were analysed for novelty, for their potential to
address the macro evaluation needs of extension projects in Australia, and the micro needs of the
case study project. All four models appeared to have much to offer the ‘basket of choices’ for
agricultural extension evaluation at a national level. However, Davies’ Most Significant Change
(MSC) model was selected as it seemed the best model to address the micro evaluation needs of the
Target 10 Project, across which it is to be implemented. The MSC model was also considered to be
a novel approach for extension evaluation, as the review of current practice (Chapter 4) revealed a
surprising lack of project monitoring. It was also a highly novel approach in terms of methodology,
involving the collection and participatory interpretation of stories of significant change. Indeed,
there appears to be little systematic, formal use of stories in program evaluation, yet they are
frequently studied by organisational learning specialists. I suggest that the participatory
interpretation of stories as advocated in the MSC model has much to offer stakeholders in terms of
helping them make sense of impact.
140
CHAPTER 6
METHODOLOGY FOR THE CASE EVALUATION
In this chapter the research design and methods used to evaluate Davies’ (1996) Most
Significant Change (MSC) model of monitoring and evaluation are described. The research
design involved conducting a single case study of the implementation of a modified version
of MSC model across the Target 10 Dairy Extension Project for a period of 12 months. The
MSC model was then evaluated using two different research approaches. Firstly, ‘action
research’ was conducted whereby project staff implemented, reviewed and adapted the MSC
model. Secondly, Pawson and Tilley’s (1997) ‘Realistic’ model was used to guide a meta-
evaluation and to help understand how the MSC model worked and in what situations. This
involved developing an ‘evaluation theory model’ of how the MSC process attempted to
bring about an improved project. This model was refined through an iterative process that
involved collecting and analysing empirical data. The data for the meta-evaluation included
semi-structured interviews and group discussions; observational data, project documents, and
a questionnaire.
6.1 Introduction
The empirical research for this thesis involves the modification, implementation and evaluation of
this model across an extension project. This aim of this chapter is to describe the research design
and methodology that was employed in evaluating Davies’ (1996) Most Significant Change (MSC)
evaluation model.
I begin this chapter with an explanation of the overarching research design for the case study. The
case project is then presented along with the rationale for choosing this particular project. Following
this, I describe the participatory action research component of the design. Next, a description is
provided of methodology adopted for the meta-evaluation. Lastly, the procedures that were taken to
ensure the confidentiality of the informants are outlined.
141
6.2 The Overarching Research Design
The research design consisted of a single case study of the implementation of the MSC model
across a Victorian extension project. The case study was guided by two main research approaches:
1. Participatory Action Research: whereby I facilitated the project staff to implement, evaluate and
adapt the MSC process over a period of 12 months across the Target 10 Dairy Extension
Project.
2. Meta-evaluation: whereby I evaluated the MSC model, guided by a Realistic model of meta-
evaluation. This involved developing theory about how the MSC model worked, in what
situations and why.
Key differences between the two approaches were the degree of participation of project
stakeholders and the purpose for which the information was collected. The PAR component was
conducted by the project stakeholders to meet the immediate needs of the project, while the meta-
evaluation was conducted largely by me to meet the requirements of my doctoral research. Figure
11 illustrates how the two research approaches draw data from different aspects of the
implementation process.
Another key difference between the two approaches is that the PAR component was used to provide
context-specific findings, rather than generalisable findings. The meta-evaluation was aimed at
providing both context-specific findings and developing theory concerning the worth of the MSC
model for extension projects more broadly.
142
Figure 11 A systems diagram to illustrate case-study design including concurrent action research
and meta-evaluation components
Participatory Action research component (PAR): Participants implement, evaluate and modify the MSC process in an iterative manner. This was facilitated by the researcher. The outputs of the process are used by the project stakeholders to improve the process and to make sense of project impacts. The findings of this research related to the worth of the MSC approach specifically for the Target 10 Project.
Meta-evaluation component Researcher investigates how the MSC process worked, in what situations and why. The idea was to develop theory about the worth of the MSC model for extension projects in general.
Two research approaches and purposes
Target 10 Dairy Extension Project operating in four regions of Victoria
Context
12-month implement-ation of MSC Approach
Process
Changes in knowledge and skills of participants that contribute to an improved project
Results/outcomes Outputs of Process
Information (in the form of stories and interpretations) produced by participants
Systems diagram for Implementation of the MSC process
143
6.3 Rationale for Choice of Project
The project chosen for the case study was the Target 10 Dairy Extension Project. It was selected for
several reasons that are outlined in the following section. These reasons included that the project:
• has a mega-project, with multiple-stakeholders, and was experiencing an increasing demand for
evaluation
• was one of 10 projects that had volunteered to work with the ‘Evaluation Pilot’, a group within
the Agriculture Division that aimed to facilitate projects in developing an internal evaluation
capacity
• had an established evaluation culture, making entry easier
• was staffed by people who were enthusiastic about implementing the trial
• was in a transitional stage, with plans to redevelop the project in the following year
• had a three tiered management structure.
This project has been referred to as one of the most established of the new mega-project structures
that are now being encouraged within the Agriculture Division (Kefford, B pers. comm., 1999).
Because of the challenges faced in evaluating complex ‘mega-projects’, a case project with these
characteristics of a mega-project was seen to be important. The project operates in four regions of
Victoria and has a total of 50 extension workers operating from 10 different locations. Four
‘regional committees’ and one ‘central executive committee’ oversee the project. These committees
comprise farmers, industry representatives, representatives from the universities and project staff
(elaborated more fully in Chapter 7). This constitutes a very large extension project by Australian
standards.
In the Agriculture Division of the Victorian Department of Natural Resources and Environment, a
project was initiated in 1996, entitled the ‘Evaluation Pilot’ and the Target 10 Project was working
with this Pilot project. The mandate of the Pilot Project was to facilitate over 40 mega-projects in
the Agriculture Division to develop comprehensive evaluation plans and to implement them. It was
through this project that I first gained access to extension projects in Victoria. When the fieldwork
for this thesis was being planned, the Pilot Project was currently working with only ten of these
mega-projects, all of which were part of the Agriculture Division. I was invited to informally join
the Pilot Project, and attended many of the meetings and activities of this project between 1997 and
144
2000. Several of the initial 10 projects were interested in collaborating in this research project.
From these I selected the Target 10 Dairy Extension Project as a case study.
At the time, agricultural extension in Victoria was split between two Divisions: the Agriculture
Division and the Division of Land and Water. The Division of Land and Water includes
participatory environmental initiatives such as Landcare. At the time, this division did not have an
evaluation initiative such as the ‘Evaluation Pilot’ project, therefore my access was limited to
projects within the Agriculture Division. At the time of selection of the case study project, there was
some concern that the Target 10 Project, being largely production orientated and not necessarily
‘typical’ of extension projects, might not be the most appropriate project with which to test a novel
evaluation model. However, ultimately the Target 10 Project seemed like the best choice available,
as prior to 1997, the project already had some culture of evaluation, and was probably conducting as
much if not more evaluation than any other agricultural extension project in Victoria. This allowed
a swift entry time into the project, without a need for spending time convincing management and
staff for the need of evaluation, which may have been encountered in other projects with a less
developed evaluation culture. In addition, the project managers of Target 10 Project expressed
commitment and enthusiasm for the trial.
An important aspect of the case study was the extent of change occurring at all levels. The project
was undergoing changes unrelated to the evaluation during the period studied. The project was in its
last year of the first funding round and in addition to my research an external review of ‘Human
Resources, Development and Extension, was conducted across the dairy industry. Rogers (1996)
suggests that given the rapid rate of change which is now a feature of organisations, including the
Australian public sector, it is important that an evaluation approach not be tested in a situation
where the only change occurring is that under the control of those controlling the evaluation. She
suggests that for an evaluation approach to be useful in contemporary public sector evaluation, it
must be feasible during periods of rapid change. Certainly in the case of MSC model, it was
implemented in a time of considerable change and project re-development. The fact that the project
was in re-development was also beneficial as it meant that things learned from the MSC evaluation
could more easily be assimilated into project planning.
6.4 Participatory Action Research Component
Initial reviews of published and non-published literature concerning the current practice of
145
evaluation in agricultural extension revealed that very little systematic monitoring is being
publicly reported in Australia (Chapter 4). Therefore it was not possible to conduct empirical
research into the effectiveness of an existing system of monitoring in Australia. Action research
allowed the implementation and modification of the MSC model to suit the context of the Target
10 Project in Victoria.
I felt it necessary to develop some system to ensure that the project staff had considerable
freedom in adapting the MSC model to meet the regional and project needs. Firstly, the MSC
model was designed by Davies to be implemented across a savings and credit program in
Bangladesh, and had not previously been implemented in a ‘Western’ context. I anticipated that
it would require modifications in order to adapt it to the context of a Victorian extension project.
Secondly, the Target 10 Project is also partially regionalised, with each of the four regions
having considerable autonomy with regard to project activities and organisation – so it was
important that the process could be modified to meet the regional needs. Thirdly, the MSC
model itself is designed to ‘evolve’ to the local context, with each successive round of story
review affording opportunity to refine the process itself. Consequently, I felt it was important
that the project staff themselves had freedom to evaluate and modify the process as they saw fit,
and therefore adopted a Participatory Action Research approach.
Participatory Action Research (PAR) pursues both action and research outcomes. It is most
commonly conducted with high levels of participation. Its purpose is to bring about planned
change in such a way that there is also a contribution to understanding of the system that is being
changed (Cunningham, 1993). Whyte (1991) suggests that in PAR some of the people in the
organisation under study participate actively with the researcher throughout the research process,
from the initial design to the final presentation of results and discussion of their action
implications. Whyte’s description of PAR is congruent with the research that I facilitated under
the label of PAR.
PAR is an especially participatory form of action research. Hart and Bond (1995) selected seven
criteria to distinguish action research from other methodologies, i.e., it:
• is educative
• deals with individuals as member of social groups
• is problem focused, context-specific and future-oriented
146
• involves a change intervention
• aims at improvement and involvement
• involves a cyclical process in which research, action and evaluation are interlinked
• is founded on a research relationship in which those involved are participants in the change
process.
The participatory action research that I was involved in facilitating fits with the above criteria. It
was educative in that it exposed participants to a new approach to evaluation, and to new
understandings of the impact of the project. It involved action in terms of actually implementing the
MSC model, modifying the process to meet needs, and modifying extension practice as a result of
knowledge gained through the process. The MSC process was conducted with the participation of
over 70 extension staff and collaborators, who had input into evaluating and modifying the process
as it was implemented.
Most action research is specific to the study and its findings are not necessarily generalisable. For
this reason it was conducted alongside a ‘realistic’ meta-evaluation, which is less context-specific in
orientation.
Data sources for the PAR
The MSC model was facilitated by me, but implemented across the Target 10 Project by the project
staff and committee members. Each region set in place a system for collecting and reviewing stories
of significant change on a monthly basis. The implementation process is described in Chapter 7.3.
The main forms of data that were available to the participants to modify the MSC process and to
reflect on their own extension practice are described in the following section. These were:
• stories of significant change that were collected from each region
• their own experiences of being participants in the story review process
• informal, verbal feedback detailing interpretations of the stories at the regional level meeting-
some of which was documented
• feedback documents detailing interpretations of the stories at the Statewide meetings
• a feedback document from a panel of purchasers and key influencers of the project, detailing
interpretation of their stories (but only at the end of the 12-month trial).
147
The method of developing both stories and feedback documents evolved as the process was
implemented. Therefore details of the data collection approach for each of these stages will be
discussed at greater length in Chapter 7, which includes a chronological description of the
implementation process itself.
Stories
Over the 12-month trial, 134 stories were collected and reviewed. The stories themselves were
collected in one of two ways. Some were hand-written on proformas. Others were directly narrated
into a tape recorder – this tended to happen frequently at regional committee meetings (the telling of
one story by a participant would frequently result in memory triggers for other participants). In the
latter case, mostly I transcribed the recorded stories (word for word). In a handful of cases this
process resulted in very lengthy stories, and the selection committee suggested that they be
shortened. In these cases, the stories were abbreviated and given back to the storytellers to check
and validate. The wording of the stories was never modified without the subsequent examination
and approval of the storytellers. For this reason the stories maintained a conversational tone.
However, in the final report (that was submitted to the purchasers) the project staff decided that the
stories should be edited for ease of reading, but colloquialisms and slang were not removed.
A master copy of all the stories was kept by the project management, and a spread sheet was
established on Excel to keep track of where the stories originated, who told them and whether or not
they were selected at the various review panels.
Feedback reports
At the regional meetings, stories were read aloud by the committee members, and a discussion was
held with regard to the interpretation of the stories. A summary of the discussion around the stories,
and a list of the selected stories, was included in the minutes of the meeting.
After each statewide meeting, (every three months) I compiled a feedback report. These documents
consisted of the stories that were selected plus a list of the comments that people had made about
the stories. During the meeting at which the stories were reviewed, these comments were scribed
onto a white board for all to see, and the committee members were asked to verify the comments at
the end of each meeting. Discussion at all of the state executive meetings where story interpretation
occurred, was audio-recorded using a multi-directional microphone. This was done with the consent
148
of the committee members. Where there was some confusion or lack of clarity the transcripts were
consulted, but generally the feedback documents consisted of the points made on the white board
(more detail is provided in the description of the implementation of the process in Section 7.3). The
feedback document was disseminated across the entire project – by email to all the project staff, and
to all committee members by hand or mail.
At the end of the 12 months, a meeting was held with a panel of project funders and key purchasers.
This meeting was also audio-recorded. The outcomes, and main discussion points form this meeting
were written into a formal evaluation document (Target 10 Evaluation Stories, Dart, 1999c), and
circulated to all project staff and stakeholders. This document was sent to over 250 people.
How the data were used by participants to modify the MSC model
All project staff and committee members had access to the stories and to the interpretations of the
stories from the various meetings (referred to as feedback documents). Using this information the
regional committees made decisions about how the process should operate in their regions. This
process was also facilitated by ‘regional champions’, who were responsible for encouraging the
collection of stories, and facilitating the selection process during the regional committee meetings.
In addition, three group discussions were held between all ‘regional champions’ of the process.
These meetings concerned the MSC implementation process, rather that the business of reviewing
the stories per se. This group acted as a quasi ‘action learning group’. Through discussion, this
group shared their experiences of the MSC model in the different regions to suggest modifications
to the process so that it would better meet their needs. These suggestions were then taken back to
the four regional committees for discussion, modification and in some cases were implemented.
This process is fully described in the Section 7.3.
6.5 Meta-evaluation
The Participatory Action Research (PAR) component provided context-specific findings and helped
the project staff to tailor the MSC process to meet their needs rather than generalisable theory. As
this thesis is concerned with developing a new approach to evaluation for agricultural extension
projects nationally, I felt it was also important to evaluate the implementation in a manner that
would produce some theory about the applicability of the MSC model for extension projects in
149
general. For this reason, I also conducted a less participatory ‘meta-evaluation’ of the MSC model.
Scriven (1991) describes meta-evaluation as the evaluation of evaluations; indirectly the evaluation
of evaluators. He states that meta-evaluation is the professional imperative of evaluation: it
recognisees that ‘evaluation begins at home’. Scriven states that an evaluation should be
conceptually clear, comprehensible in all detail, comprehensive, cost-effective, credible, ethical,
and explicit about the standards of merit or worth. Conducting a meta-evaluation (as in any
evaluation) is a value-laden process and will hinge upon the epistemological approach and the
criteria selected by the evaluator. Worthen et al. (1997) state that there are over 60 different models
of how evaluation should be conducted. With the same logic, there are also many models that could
inform the criteria (and the epistemology) by which an evaluation is judged (i.e., meta-evaluation).
However, Rogers (1996) states there is little guidance available for the practice of meta-evaluation:
…evaluation approaches have rarely been formally evaluated. Instead they have tended to be adopted and promoted without convincing evidence of their effectiveness or appropriateness, nor information about how they work (Rogers 1996: 8).
Because of the scarcity of guides to meta-evaluation, I developed a series of meta-evaluation
questions to guide this process (Section 2.6). All of these questions are addressed in the meta-
evaluation and presented in Chapter 9. However, in Chapter 8, the first question (did the evaluation
lead to an improved project?) was examined in detail as this was seen as a central concern.
This first question is based on the premise advocated by Rogers (1996) that good evaluation should
contribute to improved projects; that is projects which meet needs. Improved projects can come
about in one of three ways; improving existing projects, developing alternative projects, or
developing new projects. In this case, I am dealing only with the first. To address this question, I
felt it was necessary to conduct some research to help explain not only whether the MSC model
contributed to improvements in the project, but how and in what situations these ‘improvements’
occurred. Gaining an understanding of the contexts in which the model was most successful (and
where it was not successful) is also an important question to answer in order to be able to generalise
about the applicability of the model to extension projects more widely (Chapter 9).
Theory-guided approaches to evaluation involve the construction of ‘program theory models’; these
are essentially casual models that elaborate how a program is intended to meet needs. A theory-
driven approach to meta-evaluation was adopted for this research as it appears to offer great
potential in gaining a deeper understanding of how an evaluation approach is working, and whether
150
it would be applicable to other contexts. This seemed particularly appropriate for a novel approach
to evaluation such as the MSC model that has only been implemented once before in a different
culture, for a different program.
Theory-guided meta-evaluation
Shadish (1992) defines theory-driven evaluation and theory-driven meta-evaluation:
Theory-driven evaluation suggests constructing program theories to describe how programs work (Chen & Rossi 1981, 1983, 1987). These theories are usually embodied in program models. Such models draw on substantive hypothesis from various disciplines, from past research and from ordinary knowledge about program inputs, processes and outputs…. Theory-driven meta-evaluation parallels this logic, but models evaluations themselves rather than programs. Models of how evaluation operates draw from theories of evaluation, from past research, and from ordinary knowledge about the inputs, operating processes and outputs of evaluations. Theory-driven evaluation tests the plausibility of such models (ibid: 29, emphasis in original).
Several authors have suggested recently that theory-driven models of evaluation are particularly
relevant for meta-evaluation (Rogers, 1996; Shadish, 1992). In this thesis I elected to use Pawson
and Tilley’s (1997) ‘Realistic’ model of theory-driven evaluation to guide the meta-evaluation. One
of the central arguments inherent in this model is that evaluators should go beyond addressing the
question: ‘does a program work?’, and instead focus on which programs work, and in what
situations and why. To accomplish this, Pawson and Tilley build a model that evolves around the
notion of context-mechanisms-outcome configurations (CMOs), or configurations of mechanisms
(M) operating in contexts (C) to produce outcomes (O) (more fully elaborated in Section 6.5).
It should be noted that Pawson and Tilley’s realistic model was designed for evaluating programs,
not evaluations. However Shadish, (1992) suggests that logic for meta-evaluation parallels that of
evaluation, the difference being that the evaluation is modelled rather than the program. Using
Realistic Evaluation to guide the meta-evaluation appeared to be advantageous. Nevertheless, I
found no instances of a realistic model being used for meta-evaluation.
151
A realistic model was adopted for several reasons that are outlined in the following sections. These
reasons were to:
• ensure that the meta-evaluation was not a black-box venture.
• learn how the MSC model attempts to contribute to project improvements
• understand how the MSC model works and in what situations and why
• implement another promising approach to evaluation
• develop a theory about the applicability of the MSC model for extension projects in Australia.
At the time of implementing the trial, little guidance was available with regard to how the MSC
model is supposed to contribute to project improvements. This was because Davies’ thesis was not
published until 1998, which was after this trial had been conducted. The original instruction for the
process was limited to a 10-page publication (Davies 1996) that outlined the method but did not lay
out a theory of how the model was intended to bring about an improved project. Therefore, it
seemed important to develop some plausible theory as to how the MSC model attempts to bring
about change, and the Realistic model provides guidance for doing this.
An analysis of four contending models of program evaluation revealed that Pawson and Tilley’s
(1997) Realistic Evaluation had much to offer the ‘basket of choices’ for extension evaluation in
Australia (Section 5.5). Using Realistic Evaluation to guide the meta-evaluation also affords some
new knowledge with regard to how this model can be implemented.
As the aim of this thesis is to develop a model of evaluation for agricultural extension projects in
Australia, any assessment of the MSC model should not be confined to an investigation of how
successful the model was for the Target 10 case project. It should also consider how applicable it
could be for extension projects more broadly. A realistic approach to meta-evaluation facilitates
understanding of the underlying mechanisms of the MSC model and under which contexts these
mechanisms are likely or unlikely to be fired. Following this logic, a realistic approach to meta-
evaluation affords a description of how appropriate the MSC model may be for other extension
projects.
152
Method of applying Realistic approach in the meta-evaluation
Pawson and Tilley’s description of Realistic Evaluation was followed were possible. Conjectured
CMO (context, mechanism outcome) configurations for the MSC process were developed in an
iterative manner then tested against empirical evidence.
Developing an evaluation theory model and the conjectured CMO
configurations
The CMO configurations were developed after the 12-month trial, and the process of developing
them involved five main steps:
1. I developed a preliminary ‘free-form’ evaluation theory model for how I felt the MSC process
was intended to bring about project improvements.
2. 10 semi-structured interviews were conducted in which informants explained how they
constructed the MSC process. At the end of each interview, the informant and myself modified
the ‘evaluation theory’ model. At the end of 10 interviews, the final ‘free-form’ conjectured
evaluation theory model was drawn (see Figure 12).
3. I identified nine key conjectured CMO configurations from the evaluation theory model and the
interview transcripts.
4. A questionnaire was administered to all staff, with regard to the extent to which they felt that
the conjectured outcomes had been achieved.
5. A causative theory of how the model works was developed, using data from the questionnaire,
interviews, group discussions, and observational data.
I developed the first draft of the ‘evaluation theory model’ using the experience and knowledge I
had gained during the 12-month implementation of the MSC process. This was in the form of a
visual diagram ‘free-form’ diagram consisting of text boxes and arrows, similar to the final model
presented in Figure 12. I have termed this model an ‘evaluation theory model’, in reference the
equivalent ‘program theory model’. ‘Program theory models’ are developed to understand how a
program (or project) is intended to bring about intended outcomes. In theory-guided meta-
evaluation, it is the evaluation which is modelled, rather than the project, the term ‘evaluation
theory model’ is intended to reflect this difference.
153
The evaluation theory model was substantially revised over the course of conducting 10 semi-
structured interviews with key informants (interview guide is provided in Appendix 1.1). In doing
so, I tried to place my construction of how the model operated to a side - letting the informants talk
freely of how they constructed the MSC process. At the end of the interviews, I showed them my
preliminary evaluation theory model and discussed differences in the way I constructed the process
compared to how they constructed it. Together, we modified the evaluation theory model based our
mutual insights. Therefore, as the interviews proceeded the model was refined. The final
conjectured ‘evaluation theory model’ is presented in Figure 12.
The evaluation theory model depicted in Figure 12 was organised with the help of Bennett’s
Hierarchy of outcomes – a hierarchy of goals established for agricultural extension programs
(Bennett, 1977). Bennett lists seven levels of goals, which can also usefully be applied to goals of
an evaluation initiative:
• Level 7 – Consequences for the target group
• Level 6 – Behavioural changes in the target group
• Level 5 – Change in knowledge, attitude, skills, motivation and group norms
• Level 4 – The farmers’ opinion about evaluation activities
• Level 3 – Participation in evaluation
• Level 2 – Implementation of the evaluation by extension agents
• Level 1 – Programming of the evaluation activities
In Figure 12 these goals are listed down the left-hand margin. (Patton, 1997: 236) also used this
hierarchy to structure an evaluation theory model for his Utilization Focused Evaluation Model.
This evaluation theory model did not give reference to context, and was not presented in the CMO
format as suggested by Pawson and Tilley (1997). Nevertheless, during the interviews, the 10 key
informants discussed project contexts where outcomes were more likely to be realised, and why
these outcomes had occurred. Using the transcripts from these interviews, alongside the evaluation
theory model, I identified nine conjectured CMO configurations, which are presented in Chapter 8
(Table 14). In Figure 12, the numbers in the text boxes correspond to the nine key outcomes in the
CMO configurations. Some of the outcomes represented in Figure 12 were not selected to test in the
meta-evaluation. The rationale for selecting some key outcomes over others concerned the purpose
of the meta- evaluation, which was to gain some idea of the sorts of project contexts in which the
154
MSC process might bring about project improvements.
Finally, I developed a simple questionnaire to send to all stakeholders to test and refine the
conjectured CMO configurations. I developed the final model without further reference to the key
informants. The model-building process was therefore iterative and the final model was grounded in
empirical data from several sources.
In hindsight it may have been easier to develop the CMO configurations directly with the key
informants. However, both the informants and I were accustomed to drawing ‘free-form’ project
theory models, and it was natural for us to display the evaluation theory model in a visual, ‘free-
form’ manner. This enabled us to explore the connections between different conjectured outcomes.
This part of the process deviated from that laid out by Pawson and Tilley (1997). In the analysis I
also tended to focus on whether the conjectured outcomes had been achieved in different contexts,
rather than looking for evidence as to whether the mechanisms had lead to the outcomes in certain
contexts. I believe the analysis might have been richer if I had designed the questionnaire to
specifically test the CMO configurations, rather than the context to outcome relationships.
155
Figure 12 An evaluation theory model showing how the MSC process was thought to bring about improvements in the project
Purchasers reviewthe stories andreflect on desirableoutcomes
Storiescollected byfield staff andfarmers
Staff andfarmers reflecton individualpractice
Stories reviewedby committees
Committeesreflect onpractice
Discussionfrom reviewprocess isrecorded
(Feedback)
h) Short-termplanning &beneficialoperationalchanges
1. Either an improved project that can better meet needs, or improvements in the next projectUltimateoutcome
f) A moreshared visionbetween allstakeholders,making moreefficient work
h) Strategicchange andproject re-development
g) Evolution &improvementin the way theMSC processis conducted
i) Projectlearns howto continueto learnfrom pastexperience
h) More informeddecision-makingby purchasersresulting ineffective fundingallocation
Changes inbehaviourbrought aboutby the MSCprocess
e) Project learnswhat unexpectedimpacts arevalued and whichare not valued bythe differentstakeholders
c) Moraleincreasesas staffreceivefeedback
d) Management &steeringcommittee gainknowledge &change attitudesthrough theirinvolvement in theprocess
d) Deliverystaff gainknowledge &change theirattitudes frominvolvementwith theprocess
Enhancedinter-organisationalcommunicationdialogue
d) Purchasers gainnew knowledgeand attitudes aboutproject’sachievements(both expectedand unexpectedoutcomes)
KASA changes(knowledge,attitudes, skills &aspirations)brought about bythe process
Reactions tothe MSCprocess
Stories and the feedback are interesting somore stakeholder read/listen to the storiesand interpretations
a) More people pay attention to evaluativeinformation and participate in theevaluation process
Activities in theMSC Process
g) Projectlearnshow to runthe MSCprocess
The discussion is seen as valuablein terms of working out what isconsidered to be desirable impact
Stakeholders are willing to participatein the review session & commit toparticipation in the process
Farmers members also enjoythe process and feel they cancontribute to the evaluation
b) Farmers gaina voice in theevaluation & helpsteer the projectto better meetfarmer needs
As the MSC process involves existingcommittee structure farmer-committeemembers participate in the evaluation
Changes inparticipation
156
Data sources for meta-evaluation
In addition to the data collected during the trail, I also collected additional post-trial qualitative and
quantitative data. This second phase of data collected was entirely to inform the meta-evaluation; it
was not to reinforce the implementation process. The data sources are elaborated in the following
section and comprised: project documents, transcripts from story review sessions, field notes, focus
group with the purchasers, group discussions, semi-structured interviews and an internet
questionnaire.
Project documents
Relevant project documents included the output of the evaluation process that comprised 134
stories, and over 20 feedback documents, which were circulated to all staff from meetings at which
the stories were reviewed and selected. I also had access to previously-conducted evaluation reports
and the final evaluation document from the MSC model.
Transcripts from story review sessions
I systematically recorded and transcribed discussion at over 15 meetings at which stories were
interpreted by the project staff and committee members. The discussions were audio-recorded. I
later transcribed the tapes (word for word). As no discourse analysis was planned, pauses and non-
verbal sounds were not transcribed. In compliance with the ethics committee regulation all tapes
and transcripts were kept in a secure location.
157
Field notes
I kept field notes on my reflections of the difficulties and issues encountered while trying to apply
the principles and procedures of Davies’ model. After each selection committee, I recorded my
immediate reactions to the session on a tape recorder. These reactions included contextual factors
that I believed might have affected the process at that particular meeting. For example, instances
when I felt that I had done a poor job of facilitation due to my own fatigue, instances when I had an
impression that a particular person had dominated the review session. These field notes were
transcribed and entered into the computer program ‘NUD.IST’ revision 4 (Non-numerical
Unstructured Data Indexing Searching and Theorising) under the memos, and kept near the
transcripts to help make sense of any anomalies at later stage. I also noted down any insights I had
on the process as I went along, which formed part of the iterative theory building process.
Focus group with the purchasers
In August 1999 a round-table meeting was held with eight participants who were considered to be
‘purchasers’ or ‘key influencers’ of the Target 10 Project. These participants represented the Dairy
Research and Development Cooperation (DRDC), NRE, the University of Melbourne and the
DRDC Regional Development Boards. The round-table meeting took the form of a facilitated group
discussion in which all participants were asked to give their reaction to the stories. After the session
concerning the stories, the participants were taken through an ORID focus group (the guide used for
the discussion is presented in Appendix 1.2). Another person facilitated this process, to ensure that I
did not ‘lead’ the discussion.
Stanfield (1997) describes the ORID process as a technique for structuring a facilitated and focused
group conversation based four types of question:
• Objective questions
• Reflective questions
• Interpretive questions
• Decisional questions
This order of questions is designed to follow natural processes of the human mind of which we are
usually unconscious. ORIDs are used for a variety of purposes in organisations and workshops, and
is commonly used to evaluate a workshop and decide what more information is needed, to develop
158
a new vision, for strategic planning purposes or to collectively make a decision.
The reason for adopting this approach was mainly practical. Experience showed that it was difficult
to gain access to this particular group, due to their heavy time commitment to other matters. The
round-table meeting afforded a rare opportunity to speak with the seven purchasers at the same
time. The ORID method of focus group was particularly appealing in that it is good approach of
focusing conversation in a limited period of time.
Group discussions
Informal discussions were held with all the committees during the regular meeting times. The main
focus of these discussions was to identify the strengths and weaknesses of the MSC model and to
come to a decision as to whether to continue the MSC model after the 12-month trial period. In
addition a discussion was held with the regional-champions of the process.
Semi-structured interviews
Semi-structured interviews were held with eight project staff and two farmer representatives.
(interview guide is provided in Appendix 1.1). These interviews were conducted to understand how
project staff understood the MSC process to work. The data from these interviews informed the
Internet questionnaire. The interviews themselves were an important stage in developing the
conjectured CMO configurations (as described previously).
Internet questionnaire
The questionnaire (see Appendix 2) was based on conjectured CMO configurations (Table 14) that
were developed during the implementation of the MSC process and through analysis of the semi-
structured interviews. The questionnaire consisted of a mix of structured questions based on a
Likert scale and open-ended questions. In every instance the respondents were given an opportunity
to explain their ranking in comment boxes.
All project staff were sent an email alerting them to an Internet site where the on-line survey was
located. Links were placed on the survey to an ethics declaration (see Appendix 3) and to a
publication detailing the MSC model (Dart, 1999a). The surveys were anonymous, and each
informant was asked to provide a PIN number, so that any accidental duplication could be
identified. After seven and 14 days the informants were sent an email to remind them to complete
159
the survey. The survey was filled out directly onto the computer, and when the respondent had
completed the survey they pressed the ‘submit’ button, and a copy of the response was
automatically sent to me by email. The responses to the survey were copied and pasted directly into
an Excel spreadsheet.
This approach was used as I thought that it would be an interesting and efficient way of conducting
a questionnaire. All the project staff have access to a computer and Internet, and use email as a
regular part of their work.
Data analysis
Much of the data analysis was conducted in an iterative manner; as one piece of data was collected
and interpreted, this informed the design of the subsequent phase of data collection. For example,
the knowledge of the implementation of the MSC model led to the semi-structured interview guide
and preliminary theory, and the findings of this led to the formulation of questions for the
questionnaire. However, all the qualitative data was finally entered and organised using the
computer program ‘NUD.IST’ revision 4. I read and re-read the data as it was collected. I then
coded text extracts under the emergent and predetermined categories. The predetermined categories
were based on the key meta-evaluation questions developed in Chapter 2, and as the model was
constructed, the conjectured outcomes of the MSC model were also added into the coding frame.
The quantitative data collected from the on-line questionnaire was organised into spread-sheets
using the computer program Excel and statistical analysis was conducted using the computer
program SPSS. Analysis was conducted using cross-tabulations with the Gamma measure of
correlation. In order to test the level of significance of the associations, the Gamma inferential
statistic was used. Gamma is a PRE (proportional reduction of error) measure. De Vaus (1995: 168)
suggests the Gamma measure is the most common statistic when both variables in a crosstabulation
are nominal and is an appropriate test to use when there are few categories as was the case with the
data.
160
6.6 Ethical Considerations
The study was approved by the University Ethics Committee, and endorsed by all five steering
committees of the Target 10 Dairy Extension Project. In all cases, where face-to-face interviews,
group discussions or telephone interviews were conducted, the informants were presented with an
information sheet that detailed their rights to confidentiality (Appendix 4). After providing this
information, the informants were asked to give their consent for the interview to be used in this
study, this verbal consent was recorded on tape along with the interview or group discussions.
Achieving written (signed) consent in this research was seen as an unnecessary hindrance to the
researcher obtaining freely given information in a relaxed, informal environment. All the
interviewees have been involved with this process for over a period of one year, and were very
familiar with the process and the aims of the assessment of this tool. It seemed ‘overly formal’ to
obtain signed consent for reviewing a process in which they had been so extensively involved in
implementing.
Interviews were conducted in private rooms, with no other people present. The interviews were
recorded with an audio tape recorder. The tapes were transcribed word for word by me. The
transcripts and tape copies have been kept in a locked drawer in the Institute of Land and Food
Resources. The files on the computer are protected by a password. In accordance with what was
said to the interviewees, quotations are used in the thesis, but the authors of the quotation will not
be individually referred to. The quotations were sourced in terms of the job of the informant (ie.
quote: extension staff). In order to maintain the confidentiality of the few managers and regional
coordinators all were referred to as ‘project co-ordinators’.
161
CHAPTER 7
CASE EVALUATION OF THE MSC MODEL AS
IMPLEMENTED ACROSS THE TARGET 10 PROJECT
In this chapter the case evaluation is presented, comprising a description of the Target 10
Project and the implementation of a 12-month trial of the Most Significant Change model.
The output of each step of the process is described and examples are provided to illustrate
the points made. The output included 134 stories and feedback from the review of the stories
during 15 project meetings. The output also included an analysis of the 134 stories, which
formed part of the process itself. The problems and issues that arose while implementing the
model are discussed and include issues concerned with competition, time, culture and ethics.
7.1 Introduction
The previous chapters established the need for new approaches for evaluating extension projects in
Victoria, and that forms of PM&E practised in overseas development may offer promising
approaches for adoption in Australia. Davies’ (1996) Most Significant Change model (MSC), in
particular, appears to have potential to meet these challenges. In this chapter the 12-month
implementation of the MSC model across the Target 10 Dairy Extension Project is described. This
case evaluation was conducted for two reasons: firstly to investigate the potential of the MSC model
to bring about extension projects that can better meet needs; secondly, to determine what
modifications are needed to adapt this approach to suit the context of Australian extension projects.
The chapter begins with a description of the Target 10 Dairy Extension Project and the historical
background to the project. I then explain the process used in introducing the MSC model to the
project and facilitating its implementation. I draw attention to the ways in which the application
deviated from that of Davies, and the reasons for these modifications. The output of each step of the
162
process is described and examples are provided to illustrate the points made. Lastly, I discuss some
of the problems that occurred during implementation concerning competition, time, issues of culture
and ethics. In Chapters 8 and 9, an evaluation of the MSC model is described. This meta-evaluation
attempts to determine the effectiveness of MSC model and its ability to address the challenges
associated with the evaluation of the new genre of agricultural extension projects in Australia.
7.2 Description of the Project
The Target 10 Project is one of the most established of the new genre of ‘mega’ projects that
operate under the purchaser-provider model for publicly-funded extension in Victoria. It is largely
outcome focused and has a productivity goal of increasing pasture utilisation by 10% (hence the
name Target 10). It has a complex structure, and is funded by various federal, state and industry
sources, as well as receiving money indirectly from levies on the sale of milk. It is seen by some as
an example of how the mega-project structure can succeed. For example, the director of the
Agriculture Division in NRE, Victoria pointed out that the ‘success of Target 10 was the precursor
to the move to mega-projects’ (Kefford, B. pers. comm., 1999). Other industries are now adopting
similar models.
Aim of Target 10 Project
The Target 10 Dairy Extension Project was initiated in 1992, with the aim of enhancing the viability
of the dairy industry through programs that profitably increase consumption of pasture by cows. It
operates across four regions of the State of Victoria in Australia. In 1996 the project focus was
broadened to include other areas that were of high priority to the industry. Information about these
areas (grazing management, business, dairy cow nutrition, soils and fertilisers and natural resource
management) is extended to farmers through courses, discussion groups, newsletters, comparative
analysis, field days, focus farms and demonstrations and other media. Target 10 is a collaborative
industry project that aims to deliver a customer-focused change program in a coordinated and
timely manner and with community ownership. The project has a documented conceptual
framework for the delivery of the project components (see Figure 13).
163
Figure 13 The service delivery model of the Target 10 Dairy Extension Project (Boomsma et. al.,
1996)
Figure 13 depicts individuals involved in the project participating in a learning process, surrounded
by a peer group and a community supportive of change. The model recognises that, in the adoption
process, the creation of the environment for change is as important as providing the technical
information upon which the change is based (Boomsma et al., 1996). The framework involves
interaction at three levels:
• At a community level, the project provides opportunities for stakeholders to participate in the
development, organisation, and delivery of project elements, as well as in communication
strategies and collaborative alliances at state and regional levels.
• At a farmer peer group level, the project provides opportunities for creating awareness of the
issues being addressed, together with structured opportunities for peer group support and for
new learning.
164
• At an individual farmer level, the project provides opportunities to adapt management practices
to suit individual circumstances by integrating management changes into the whole farm
business and providing ongoing reinforcement for practice change.
The model represents a move away from the notion of extension being simply a technology transfer
activity, to one of industry development involving the management of both the technical
information and the participatory environment of the project. This movement is characteristic of
many newer extension projects in general and is discussed in Section 2.3. An internal Target 10
report stated that:
Combining educational components (courses) with on-farm adoption (discussion groups and visits),
follow-up (newsletters and evaluation) provides a potentially greater impact than any of these
activities in isolation or partial combination (DRDC, 1994).
Historical development of the Target 10 Project
Target 10 was originally conceived as a regional project, focusing on the benefits to dairy farmers
of maximising the utilisation of pasture grown on their farms. In 1990 Agriculture Victoria
prepared a submission to the DRDC to establish a statewide extension project based on grazing
management. In 1991 a meeting of industry representatives supported the concept and established
an executive committee to guide the implementation of the project. This meeting comprised
representatives of Agriculture Victoria, milk manufacturers, and processors, Victoria College of
Agriculture and Horticulture, Herd Improvement Organisation of Victoria, United Dairy Farmers of
Victoria and farm management consultants. In 1992 agreement on resourcing was reached between
the key financial backers, Agriculture Victoria and DRDC, and the project was initiated (Boomsma
et al., 1996).
It was anticipated that increases in production and productivity worth $50 million a year after five
years would flow to farmers and hence to the state economy and all sectors of the dairy industry as
a result of the project (Boomsma et al., 1996). In the 1980s, farm gate prices for manufacturing
milk had been static in real terms and the manufacturing sector had entered a period of declining
export prices. Domestic support for the manufacturing milk process was expected to decrease
further as the sector continued to experience increased exposure to fluctuations in the international
market place. Industry agreed that the best opportunity to increase farm profitability was through
165
improved consumption of pasture. O’Brien and Hepworth (1993) showed that pasture consumption
could be substantially improved through improved management practices.
The project that evolved proposed to extend results of pasture research and farmer experience in
grazing management. The aim was to achieve a substantial improvement in pasture consumption
and it was estimated that a 10% increase in pasture consumption on a typical farm could increase
gross returns in the order of $16,000 (Boomsma et al, 1996).
Project architecture and complexity
The organisational structure under which the Target 10 Project operates is complex. The project
has a ‘mega-project’ structure, considerable collaboration and co-delivery of programs, and is
operating under the new environment of the purchaser-provider model. It is also a large project
with over 50 staff and an annual budget of over $1,000,000. Several purchasers and co-providers
interact in a dynamic project environment (McDonald and Kefford, 1998).
Farmer ownership
From the onset, emphasis was placed on ownership of the project by the dairy farmers that it was
designed to benefit. An internal Target 10 report states that:
Target 10 was developed as a regionalised project, bringing the development, management, and
implementation and evaluation processes closer to farmers through regional committees (Boomsma
et al, 1996).
From the beginning it was envisaged that evaluation procedures would be regionalised and involve
farmers as much as possible.
Marsh and Panel (1997) point out that that a uniquely Australian situation exists in the Research and
Development Corporations: whereby Australian farmers contribute through industry-levied funding
to agricultural research, development and extension, and so have expectations of a degree of
control. The co-funding of the Target 10 Project by the DRDC with the Agriculture Division of
NRE was designed to encourage farmer participation.
166
Outcome focus
Critics of the Target 10 extension project suggest that it is overly focused on production at the
expense of community, environmental, and social issues (e.g. Beilin, 1998). An external review of
the project in 1998 recommended that it become more ‘learner-focused’ (The Virtual Consulting
Group, 1999). As described in Section 3.3, the Target 10 Project combines seemingly contradictory
aspects of both ‘outcome-focus’ and ‘learner-focused extension’. As well as having a mandate to
increase production, it attempts to be highly participatory in planning, delivery, and evaluation.
The contract to implement the MSC process within the Target 10 Project
Between May 1998 and May 1999 I introduced the Target 10 Dairy Extension staff to the MSC
process, which was then implemented, as far as I know, for the first time in Australia. The MSC
model became referred to as the ‘Story Approach’ by project stakeholders. Through implementation
of this approach the project hoped to fulfil two purposes:
• to supplement other evaluation approaches by collecting qualitative data about the impact of the
project as a whole
• to promote organisational learning within the project team and between the myriad of
stakeholders.
The audience of the evaluation comprised project staff, purchasers and committee members (who
represented farmer clients and other collaborators from the university and dairy industry).
Since its inception in 1992, Target 10 had been experimenting with evaluation. Around 1992 an
evaluation consultant introduced the project team to Bennett’s Hierarchy of (Bennett, 1975). A
specialist in evaluation was appointed to the project team who had a background in marketing and
as a result considerable work was done on developing evaluation plans at the program level.
However, in 1997 the project management still felt that certain areas of evaluation needed to be
developed. Firstly, they wished to learn how to evaluate the overall non-economic impact of the
various initiatives and activities, ie. was Target 10 more than the sum of its parts? They sought to
understand the unexpected impacts. Messages and questions of help were placed on Evaltalk1 and
1 EVALTALK’ is an international e-mail discussion group that is affiliated with the American Evaluation
Association and has over 1,500 subscribers.
167
considerable research was done. Many evaluation experts seemed to come up with different
answers, but non-of these answers seemed appropriate (McDonald, B. pers. comm., 1996).
Secondly, the project funders and stakeholders alike wanted more information about the impact that
the program was having on individual farmers’ lives. In 1992 extensive benefit-cost analysis had
been completed and individual programs had been evaluated against their objectives. However, in
1998 the project manager informed me that there was still a feeling that some of the project impact
and outcomes were not being captured. He explained that when the economic analysis came back
with a figure of $62 million net economic benefit to the dairy industry (Appleyard, 1996a) a typical
stakeholder response was ‘well that’s great, but I want to know who has got that money, and what
effect this is having!’
Around about the same time, I was searching for an extension project with which to carry out a case
study. I was interested in the Target 10 Project, as it was a well-established mega-project that
already had culture of evaluation. After discussing the evaluation needs of the project with the
management I proposed to trial Davies’ approach to participatory monitoring and evaluation
(PM&E). The Project Manager gave provisional agreement on the condition that all four regional
committee meetings and the state executive committee endorse the approach. I attended all these
committee meetings over the next three months, presented the approach, and gained the
endorsement from all the committees. In May 1998 an agreement was made that the MSC model
would be implemented across the whole Target 10 Project for a period of 12 months.
The MSC process was to be an internal, formative process of monitoring and evaluation. It formed
one part on an overall evaluation strategy that included other forms of evaluation. The Target 10
Project is conducting evaluation in the following five areas:
• internally conducted evaluation of the ‘logic’ of the five core programs using Bennett’s
Hierarchy and subsequent refinement of the valued outcomes for each core program
• internally conducted evaluation of each of the five core programs against valued outcomes
• externally conducted economic impact analysis of the whole project
• internally conducted reporting against predetermined milestones
• internally conducted evaluation of the overall non-economic impact of the project.
This MSC process addressed only the evaluation of overall non-economic impact of the project.
168
While the idea of implementing the MSC process was mandated by the project management, the
project funders had previously stipulated that this sort of evaluation should occur. Internal
documents from the Agriculture Division’s purchasing group suggest that formative evaluation
occur mainly under the proviso of continuous improvement and assessment of unanticipated
outcomes (NRE, 1997).
7.3 Implementation of the Approach
Chapter 5 provides a description of the MSC model according to Davies (1996). While parts of the
MSC model remained unchanged, the implementation in this case evaluation did involve some
modification of the process to fit the context of the Target 10 Project. However, Davies (1996)
main three steps remained central in the process:
• establish domains of change
• set in place a process to collect and review stories of change within the organisational hierarchy
• hold an annual round table meeting with the project funders.
In the Target 10 case, three additional steps were added. Firstly, as staff were unsure of the sort of
stories required, an additional process referred to as the ‘taste test’ was developed prior to
establishing the domains of change. Secondly, for the approach to evolve to meet the regional
requirements a reference group was established consisting of myself and the ‘regional champions’.
Thirdly, the Target 10 management also requested a secondary analysis of the stories en masse,
which appears to be absent from the Davies case in Bangladesh. Thus a final step was added which
involved a secondary analysis of the stories. The resulting steps were:
1. ‘Taste test’ and become familiar with the approach.
2. Establish four domains of change.
3. Establish a learning set, that includes ‘champions’ from each region.
4. Set in place a process to collect and review stories of change within the organisational
hierarchy.
5. Hold an annual round table meeting for the project funders to review the stories.
6. Conduct a secondary analysis of the stories en masse.
In the following sections, each of the above steps, as implemented in case evaluation, are described.
169
I describe how the process differed from the Bangladesh case, including the output at each stage
and provide some examples of the output to illustrate the main points.
Step 1 - ‘Taste testing’ the process
After the Target 10 regional committees had endorsed the trial, I had the task of catalysing the
implementation of the process. Davies (1996) provides no concrete guidance on how the approach
should be initiated nor gives detail as to how the first stories were collected. He explains that MSC
model was built upon previous systems of monitoring, and developed in consultation with many of
the project staff. In the Target 10 Project there was no history of monitoring or participatory
evaluation, although considerable evaluation work had been completed. Part of Davies lack of
prescription can be explained by the ethos of the approach. He suggests that the MSC process
should evolve organically to meet the needs of the organisation, and thus deliberately avoids
providing a normative framework (Davies, R., J. pers. comm., 1999).
During early attempts to initiate the process, it became apparent that staff and committee members
were not sure what sort of stories were required and asked for guidance on the length of the story,
the subject matter and the form it should take. In line with the evolutionary ethos of the approach, I
felt that the project stakeholders themselves should determine these things. Staff were
understandably nervous about putting pen to paper with so little guidance, and my initial attempts to
collect stories failed. Staff explained to me that they had no idea what was expected, and thus were
reluctant to attempt relating a story. On the recommendation of several staff, I developed a
proforma (see Figure 14) to help collect the stories, and then held a pre-trial workshop in which we
‘taste tested’ the approach with the whole project team, which consisted of over 50 extension staff
and managers.
To develop the proforma I asked three staff to help me generate a sample of stories. They agreed
that I could tape this conversation. In line with the process that Davies described (1996), I asked
the staff the following question:
‘During the last month, in your opinion, what do you think was the most significant change that
took place as a result of project activities?’
After they had described the event, I asked the following two questions:
170
Why do you think the change was the most significant?
What difference has it made, or will it make in the future?
I then transcribed their responses from the audio recording, and thus generated the first three stories.
On the basis of these stories, I developed a provisional proforma along the lines of the above
questions, but including some questions concerning things such as who was involved, where did it
occur (see Figure 14). Davies suggests that enough information should be collected so that the
actors in the story could be located and that the events could be verified. This transparency proved
important in terms of credibility of the stories, but posed ethical issues (discussed in Section 7.4).
The process of developing a proforma was iterative and was conducted in conjunction with the
reference group (see below).
Story title: ‘…………………………………………...……………. ‘ Domain:
ο changes in decision-making skills ο changes in on-farm practice ο changes in profitability/ productivity ο other significant changes
Name of person recording story: ……………………………………………………. Region: ……………………………………………………. Date of narration: … /… /…… Where did this happen? ……………………………………………………. When did it happen? ……………………………………………………. ********************************************************************************** What happened?
Why do you think this is a significant change?
What difference did it make already/ will it make in the future?
Figure 14 Items contained in the proforma for collection of stories (normally more space is
allocated for responses)
In May 1998, I held a workshop at which all the project staff were present. During this workshop, I
asked everyone to submit one story about the most significant change that they knew about, that
occurred as a result of project activity. They were all given the blank proforma. About 30 stories
were collected (hand written). These stories were placed on a wall and every staff member was
171
asked to select three stories that they considered to represent the most significant change. They did
this by placing sticky labels next to the favoured stories.
I then read aloud the highest scoring stories to the whole group. This was followed by a facilitated
discussion concerning why they had selected certain stories over the others. This discussion was
audio-recorded, summarised, and sent to all staff as a memory jogger. This process was used to get
the ball rolling. In this taste test I did not include the domains, but left it entirely up to the staff what
sort of stories they wanted to record. However, during the discussion, staff explained that it was
very difficult to select one story over another as they were about such entirely different things. At
this stage I re-introduced the concept of domains of change, which the participants recognised as an
important addition to the process.
Step 2 - Establishing the domains of change
‘Domains of change’ are loose categories that are used to distinguish different types of stories. For
example, one of the four domains used in Bangladesh was ‘Changes in People’s Participation’.
Davies suggests that, unlike performance indicators, the domains of change are not precisely
defined but are left deliberately fuzzy; and it was initially up to field staff to interpret what they felt
was a change belonging to any one of these categories. Davies (1998: 279) explains that the term
‘domain’ was borrowed from Spradly’s (1979) The Ethnographic Interview. Davies borrowed the
concept of fuzzy from the mathematical term ‘fuzzy sets’.
In Bangladesh, the domains were established by a group of project managers. In the case of the
Target 10 Project, the domains were established using the Delphi technique (see Section 6.4), and
involving over 100 stakeholders of the project. Delphi is a form of interactive (postal) surveying
that utilises an iterative questionnaire and feedback and provides participants with an opportunity to
revise earlier views based on the response of other participants, until some desired level of
consensus is reached (Cary and Salmon, 1976). Part of my rationale for using the Delphi approach
was concerned with balancing the need to have ownership of evaluation by those participating in
the process.
When evaluation questions are not established in a participatory manner, insufficient attention may
be paid to addressing projects’ felt-information needs (Leviton, 1994). During wide consultation
with the project staff, it was put to me that developing the domains of change in an analytical
172
manner, without widespread consultation could have led to a lack of ownership and the feeling that
the evaluation was being done to them, rather than being done by them. In the Target 10 case it was
extremely important for the project stakeholders to have the say about what sort of changes should
get measured. Feedback from the process indicated that stakeholders were keen to be involved in
establishing the domains.
An open-ended questionnaire was sent to 150 stakeholders asking what sort of things they felt were
important to monitor (see Appendix 5). The results of the preliminary survey are presented in
Appendix 6. In the second iteration of the questionnaire, participants were asked to rank the
domains of change in terms of relative importance (Appendix 7). Table 4 illustrates the results of
the second round of the Delphi survey. In the first round of the process, 104 replies were received
(70% response rate) and in the second round 76 replies were received (50% response rate).
Table 4 Results of the Delphi process used to establish domains of change
Domain of Change Number of Votes
Changes in on-farm practice 60
Changes in profitability 57
Changes in productivity 24
Changes in decision-making skills 16
Changes in participants’ skills 15
Changes in participants’ ability to deal with changing industry conditions 15
Changes in quality of life 14
Changes in goals and aspirations 8
Changes in on-farm environmental conditions 6
Changes in farmer confidence 5
Changes to the industry in general 2
Changes in attitudes towards Target 10 2
Because several respondents pointed out that productivity is a part of profitability and none of the
respondents selected both ‘productivity’ and ‘profitability’, it was decided to select ‘changes in
profitability’ and ‘changes in decision-making skills’ (the fourth highest ranking) rather than both
‘profitability’ and ‘productivity’. During the last round, several people pointed out that all 12
domains of change offered were important to monitor. For this reason, the fourth domain of change
has been left ‘open’ to allow stories that concern issues or events that are not covered by the first
three specific domains. Davies (1996) also used an open domain in Bangladesh.
173
The domains of change chosen for monitoring are listed below, followed by discussion of the
reasons for choosing these domains:
1. Changes in on-farm practice (received a total of 60 votes).
2. Changes in profitability (57 votes).
3. Changes in decision-making skills (16 votes).
4. Any other significant types of change.
As we used the selected domains, we came to realise that their logical order was ‘changes in
decision-making skills’, ‘changes in on-farm practice’, and ‘changes in profitability’. Thus, the
domains were usually presented in this logical order rather than in the order given above. At every
Central Executive Committee meeting over the 12 months, each region was be encouraged to
submit four stories, one for each domain of change.
Step 3 - Establishing a reference group
A reference group was established to capture learning, to encourage the adaptation of the process to
local conditions and to co-ordinate the process. While Davies describes a project member who
acted as his counterpart and states that each region developed their own process; he does not refer to
having such a reference group. After consultation with the project management it was agreed to
appoint regional monitoring ‘champions’. During the first three months of the trial, I facilitated the
collection and selection of the stories at the regional level, after which, these regional monitoring
champions took over the process. However, I continued to facilitate the selection process at all five
of the state executive meetings during the 12-month trial.
The monitoring champions volunteered for the role of co-ordinating the story collection and
selection in their region and formed the basis of a reference group for the implementation of the
process across the project. The group met on three occasions during the 12-month trial and I
communicated with them frequently on an individual basis. Modifications to the process were
discussed and decided upon during communication with these individuals. In some cases, we
decided to test an idea in one region, before recommending the practice to the other regions. A
formal learning set appeared to be an important addition to the MSC process, and is congruent with
the ‘Action Learning’ approach that characterises Davies’ work with evolutionary theory. In the
174
following sections when I use we I am referring to this group of people.
However, there were difficulties in getting the group to meet regularly due to their disparate
geographic locations. In addition there was flux in the membership of the group as people changed
roles within the project. I am of the opinion that this group could have played a much stronger and
important role in the process if it had been more stable and accessible.
Step 4 - Collecting and reviewing the stories of change
Collection of stories
In June 1997, all staff and committee members were supplied with blank proformas and were
encouraged to generate stories of significant change. We intended that these initial stories would be
shared at the staff meetings at each of the four regions. However, as very few stories were
generated using this method, we encouraged staff to share the stories verbally during the meetings.
These impromptu stories were later recorded and transcribed. For some people, this was their
preferred form of storytelling; thus each regional champions elected to purchase a tape recorder and
to record stories at staff meetings. However, some participants continued to write the stories onto
the blank proformas (see Figure 14) and the mode of initial recording was left up to personal choice.
Use of the domains
Initially we asked the storytellers to classify their story according to the four domains of change.
The domains were placed on the top of the proforma, the idea being that the storyteller would tick
the appropriate box (see Figure 14). We had envisaged that the domains would guide the story
collection process. In practice this did not occur; and the domains were not seen to be helpful in
terms of collecting the stories. However, we found that when the stories were ‘shared’ at the staff
or committee meetings, the group could categorise the stories by domains. This enabled the voting
process to run much more smoothly. Nominating domains also encouraged the story reviewers to
consider the story in some depth, and this appeared to be a useful process.
Structure of the review process
We decided at an early stage of the implementation that the MSC process should ride on the back of
the pre-existing project structure. This was considered important, as stakeholders did not want to
schedule any additional meetings. My initial proposal for the story selection design was a copy of
175
the Bangladesh structure, within the limitations posed by the pre-existing Target 10 hierarchy. I
proposed that selection would occur at four hierarchical levels (staff meetings, regional committee
meetings, state executive meetings, and purchaser meetings) and that stories were to be reported on
a monthly basis. In the Bangladesh case, the approach was implemented at four hierarchical levels,
on a monthly basis.
However, fairly early in the implementation process modifications were made to suit the local
context, particularly issues of logistics, participation, and timing. My initial proposal was for the
stories to be collected primarily by staff, based on their own experience, or second hand from
farmers and other stakeholders. The storytellers were to nominate the appropriate domain for their
story. At staff meetings, participants were to review all the stories collected over the month and to
select four, one for each domain, that represented the most significant change from their
perspective. The four selected stories were then to be sent to the corresponding regional committee
meeting. As these committee meetings were held every three months, the idea was that 12 stories
(four from each of the three monthly meetings) would be sent to the respective regional committee
meeting. Each of the four project regional committees, was then to select four stories (one from
each domain) to send to the central executive committee (CEC) meeting. They in turn would select
a further four stories at each executive meeting, that would be sent to an annual round table meeting
with the purchasers of the project. This proposal is illustrated in Figure 15.
In practice we found that the first level of selection; monthly staff meetings, was problematic in
that:
• not all regions had monthly staff meetings, and the committee structure varied considerably
• project staff were keen to include stories directly from the committee members
• collecting stories at every monthly staff meeting was too frequent
• some felt that the stories should be selected by the regional committee members, rather than at
‘staff only’ meetings.
As a result, in most cases the selection process began at the regional committee, and thus occurred
at three, rather than four hierarchical levels. Each region developed its own system of selecting and
collecting stories, depending on the existing committee structure. The only condition was that the
three large regions should submit four stories to each state-executive meeting and that they should
document how they had selected these stories.
176
.
a) Four domains of change are agreed upon using a postal process
b) People working in the field collect stories that they consider to be the most significantaccounts of the agreed domain of change
c) Stories are reviewed on a monthly basis at regional staff meetings. Four stories, one for each ofthe four domains of change, are selected and sent to the respective regional committee meeting
d) The regional committee reviews the stories selected in the region since the last meeting(usually three months, so there would be 3 x 4 =12 stories). Then four stories, one for each of
the four domains of change are selected and sent to the Central Executive Committee
e) At the Central Executive Committee meeting, four stories from each region are presented.The Central Executive Committee will select one story for each domain of change
g) Key influencers and purchasers read the document and scorethe stories in terms of the extent to which the stories represent thesort of outcomes that they wish to purchase
f) At the end of the year, a document is written containing all the stories thathave been selected by the Central Statewide Executive Committee. Thedocument will also contain the reasons for selection of these particular stories
At each stage of selection, the reason that the particular stories were selected must be recorded
Figure 15 Proposed main steps for implementation of MSC process with Target 10 Project
177
In the North East, which is a particularly small region, committee meetings are held every six
months. Only two staff work in this region, so the story selection and collection process was only
conducted at the regional committee level on two occasions over the 12-month period. However,
the two staff did submit at least two stories to every State Executive Committee, even though these
stories had not always been reviewed by all of the committee.
In the Northern Irrigation Region, the process stayed fairly faithful to the original proposal. The
only change was that stories were shared, but not selected, at the staff meetings. In the first two
staff meetings, staff tried to vote on what were effectively each other’s stories. Some felt that this
competitive element was not helpful. This point is further discussed in Section 5. After discussion
with the reference group, the system was changed so that only the regional committee level actually
reviewed and selected stories. Regional committee members were also encouraged to submit
stories directly to the regional committee. After successfully piloting this modification we agreed
that the process would be adopted in all regions.
In the region of Gippsland, the project structure is quite distinct from other regions in that it has
three sub-regional committees. These sub-regional committees became the main collection points
for the stories. The stories were not shared at staff meetings, but were reviewed and selected solely
at committee meetings dominated by farmers.
In the South West, the committee structure is again different. They favour large (over 40 members)
regional committee meetings which all the staff attend, in addition to other farmers and
representatives. These meetings are held every two to three months. As all the staff are present, it
would have involved considerable duplication to share the stories at the staff meeting. Thus the
stories were collected and reviewed at the regional committee level, and no activity occurred at the
staff meeting.
Selection process
The process by which the stories were actually selected is not detailed in Davies’ publications. In
this case we developed a facilitated process. Firstly, if the stories had not all been allocated
domains, which often happened at the regional level, each story was read out aloud and
immediately allocated a domain. The titles of the stories were written on a white board under the
178
respective domains. When all the stories had been read out, all the stories within one domain would
be considered together. The facilitator would then ask a series of questions to prompt discussion
(see Appendix 8), before moving on to a vote by hands. Each committee member was given one
vote for each domain. When the vote was done, if there was no consensus, then further discussion
was facilitated until an agreement had been reached as to which story should be selected.
Occasionally no agreement could be reached, therefore either two stories were selected, or no story
was found to be suitable. The idea was to come to an agreement as a group. As well as selecting a
story, the committee members were also asked to state why the story had been selected above the
others. Much of the discussion revolved around explanations of why they thought one story was
particularly valuable or particularly misleading (see Text box 1 for an example of discussion).
Similar discussion was held about each story prior to selections being made. This discussion was
recorded on tape, or by a note taker. The intention was that these notes would be promptly
summarised and circulated to all stakeholders in the region.
179
(Background: The story concerned a farmer who had started using a project recommended practice but then reverted to his original practice. The discussion was held at a regional committee meeting at which five farmers, one dairy representative, three staff members and the facilitator were present.)
G: It’s a story in reverse, an actual thing that happened. It’s a really important story and we need to think about the implications of this.
Facilitator: What are the implications of what happened in this story? G: But….. It is not a feel good story, that’s what I am saying C: The whole point of evaluation is that we should hear stories like this – both positive things and
negative things. M: Yes, I think this is a really valid story, I think that we need more stories like this. Facilitator: But I need you to think about what are the implications of a story like this? F For the farmer or us? Facilitator: For both S: Target 10 pushed him too far.. G: No no no no no, he’s got to be confident to carry it out. [all talk at once] C: He believes it enough in the first place, but he had a lack of confidence to carry on G: He couldn’t get over the first hurdle – he had a brick wall in front of him – he didn’t have the
confidence to stay with it. C: Its not just confidence – it’s the back up and the skills. G: Oh No – but he’s got the skills there – cause he did it for the first 2 months, then he had a
shower of rain or a flood or something and he couldn’t get over that next hurdle. C: But if he had had a 1:1 visit from an extension officer, perhaps a week before, the decision to
go back? G: But how are you going to know to do that? C: He needed support to do it. G: You have got to find out, he may have had an inability to speak within a discussion group as
well, and that would probably be the area that he needed to get support from. C: Or a mentor of some form. S: Or he may not have had the ability or even realise to say… G: I’ve heard that same thing more than once, I suppose that everyone else has heard it too:
when things get hard, people give up. C: When it gets hard you often go back to what you did before, I think this is why this is an
important story. G: I think people should know that these stories should be written, as they need to be heard. I
don’t think people do know. D: Jessica said at the start that the stories could be negative or positive. CM: But they are not necessarily negative stories – like it is bad for the farmer for the individual
farmer maybe. F: I think that some of the farmers read these stories and think that these are about me…[laughs] C: this is what I always thought about Target 10 – works in spring but not in summer.
Source: Transcript from the MSC process at a regional committee meeting
Text box 1 Segment of discussion at a regional committee meeting during story selection
The process of selection was similar at the CEC. However, at this forum the process tended to be
more formal and the stories were typed, titled, and had been allocated a domain. The stories were
circulated with the agenda before the meeting, along with a voting form (Appendix 9) to allow
participants to jot their comments down before coming to the meeting. Despite the fact that most
180
people had read the stories before the meeting, the committee still favoured the process of reading
the stories out aloud, in fact this became almost a ritual that people did not want to give up.
Feedback process
The various committees were required to document which stories they selected and what criteria
they used. The idea was that this information would be fed back to the project stakeholders on a
regular basis so that they could learn from the previous round of stories and interpretations. It was
intended that the monitoring system should take the form of a slow but extensive dialogue up and
down the project hierarchy each month (Davies 1996). It was unclear in Davies’ thesis how this
feedback process had occurred in Bangladesh, so again the process was arrived at through
experiential learning.
In the first three months I tape-recorded the entire story review session held at each of the regional
committee meetings. The discussion was then transcribed, and the reasons for selecting particular
stories were summarised and circulated in the minutes of the meeting. The idea was that the criteria
for selection of the stories would be appended to the story itself, so that the state executive
committee could consider the RC member’s interpretation of story, in addition to the story itself and
the storytellers’ interpretation. However, when the regional champions took over this process, they
found it to be too time consuming to tape and transcribe the whole meeting, so the main points were
jotted down and included in the minutes in bullet point form. Due to timing issues, it was
frequently not possible to attach this feedback to the selected stories that were sent to the state
executive committee meeting. However, feedback concerning the reason for selecting the stories
was generally verbally relayed by the respective regional co-ordinators who attended the CEC
meeting. We felt that this was an acceptable trade off.
However, at the CEC the feedback was much more systematic. The review session was tape
recorded at all five CEC over the 18-month period. The details of the selected stories and the
criteria by which they were chosen were systematically reported to all project staff and
stakeholders. This was conducted largely by means of an email list-serve immediately after the
meeting. In addition, the booklet Target 10 Evaluation Stories (Dart, 1999c) was produced
containing all the stories that had been selected by the CEC over the period of the year. Each story
was accompanied by the interpretation of the storyteller, comments from the CEC, and comments
from the purchasers. Thus the reader of the document could make a judgement on the story, and
also have access to information about how the project committees valued it, and the purchasers of
181
the project. The booklet also detailed an outline of the MSC process, and the findings of the
secondary analysis of the stories. Approximately 250 copies of the booklet were distributed to
project stakeholders.
Step 5 - Annual round table meeting
At the end of the 12-month trial period a round-table meeting was held with eight participants who
were considered to be ‘purchasers’ or ‘key influencers’ of the Target 10 Project. These participants
represented the Dairy Research and Development Cooperation (DRDC), NRE, the University of
Melbourne and the DRDC regional development boards. The round-table meeting took the form of
a facilitated group discussion in which all participants were asked to give their reaction, in general,
to the stories. They were then asked to nominate certain stories as being the most representative of
the sort of outcomes that they were interested in purchasing.
Step 6 - Secondary analysis of the stories
In total 134 stories were collected, transcribed and entered onto a database. On request of the
project, I conducted an additional step of analysing the stories en masse, the findings of which were
included in the booklet Target 10 Evaluation Stories (Dart, 1999c) and circulated to all
stakeholders. This analysis was done by examining the origin of the stories, the main themes, and
differences between the stories that were selected and those that were not. These findings are
detailed in the following section concerning the process outputs.
7.4 Analysis of Outputs
The 134 stories collected between May 1998 and May 1999 originated from all of the four regions
of Victoria where the project operates. These stories were written by staff from the Department of
Natural Resources and Environment (NRE), farmers, industry representatives and educators.
Twenty-four stories from the total were selected on the basis that they were considered by the
Central Executive Committee to be the most significant accounts of change in the specified
domains.
The story collection process was monitored and the stories were examined for overall trends in
content and origin. Firstly, the total group of 134 stories was considered. Secondly, the 24 stories
that were selected by the Central Executive Committee over the year along and the criteria with
182
which they were selected, were examined. Table 5 lists the questions that were used to structure the
following section and to describe the ‘results’ of monitoring the Story Process.
Table 5 Questions asked to help describe the stories
Data sets Qualitative questions Quantitative questions
Total 134 stories
that were collected
across the state
over the 12-month
period.
24 stories selected
from the total set by
the Central
Executive
Committee, and
scored by a group
of purchasers.
• What were the stories about?
• What unexpected farmer outcomes are described in
the stories?
• Did issues covered in the negative event stories get
acted upon?
• What is different about the 24 stories that were
selected and those that were not?
• What criteria did the project team use to select the
stories? What does this tell us about what the project
team values?
• What criteria did the purchasers use to judge the
stories? What does this tell us about what the
purchasers’ values?
• Did these stories change over the year as a result of
the feedback ie? Was there evidence of learning?
• How many stories
were collected and
when?
• Who were the
storytellers?
• Which programs
were mentioned in
the stories?
• Which domains of
change were
covered?
• What percentage of
stories concerned
negative news?
• To which level of
Bennett’s Hierarchy
do the stories
correspond?
A description of the stories
Number of collected during the process
Over the year, 134 stories were recorded and documented. The story review process has occurred at
five Target 10 Central Executive Committee meetings and over 10 regional committee meetings.
Table 6 presents the number of stories collected in each round. One round represents the time span
between two consecutive Central Executive Committee meetings. There is no clear increasing or
decreasing trend and in each round of the process between 22 and 32 stories were collected.
183
Table 6 Number of stories collected each round
Detail Total number of stories
Round 1 – May to July 1998 29
Round 2 – July to September 1998 26
Round 3 – September to November 1998 22
Round 4 – November 1998 to February 1999 25
Round 5 – February to May 1999 32
Total for 12 months 134
Origin of the stories
The three larger regions of the project all contributed approximately the same number of stories to
the process (see Table 7). The reason for the smaller number of stories originating from the North
East Region, is due to the fact that this region has very few staff. When the mean number of stories
per team member for each region is considered, it becomes apparent that there is little difference
between the four regions.
Table 7 Distribution of stories per region
Region from which the stories originated: Number of stories collected (n=124)
Mean number of Stories per staff
• The Northern Irrigation 38 3.5
• The South West 39 3.9
• Gippsland 46 3.8
• The North East 9 4.5
The majority of the stories were written by staff from NRE (who were mostly Target 10 extension
staff). However, of the selected stories, a higher percentage were written by farmers than in the
total group (see Table 8). Feedback from the Central Executive Committee explains that farmer
stories were perceived to be more powerful than stories written in the third person. It is also
interesting to note that three out of the five stories scored most highly by the purchasers were also
written by farmers. Clearly, stories told directly by farmers are valued by many of the project
stakeholders. This is reflected in the proportionally higher number of stories about farmers who
have attended multiple programs.
184
Table 8 Distribution of the story tellers
Detail Percent of total Percent of selected
Written by:
• NRE staff 77% 62%
• Farmers 13% 30%
• Other collaborators 10% 8%
Project programs covered in the stories
The stories cover all the programs, and 10% concern farmers who had attended multiple programs
(Table 9). However, there is a higher proportion of stories about farmers who had attended multiple
programs in the selected stories than from the total group. Feedback from the Central Executive
Committee suggested that stories that were about consolidating learning from the various different
programs were especially valued.
Table 9 Project ‘programs’ mentioned in the story
Detail Percent of total Percent of selected
Target 10 Program Involved:
More than one program 10% 20%
Discussion groups 15% 08%
Grazing Management Program 18% 17%
Nutrition Program 17% 13%
Dairy Business Focus Program 16% 21% Soils and Fertilisers Program 15% 08% Dairy Farm Performance Analysis 09% 13%
Subject material of the stories
En masse the stories present a picture of many farmers implementing part or all of the Target 10
message, and of farmers gaining from the programs in unexpected ways. Some of the common
themes running through the stories are issues such as feeling more in control of the business, feeling
empowered to challenge the consultant, gaining increased family communication after the
identification of mutual goals. There were also multiple stories about the far reaching benefits those
185
new farmers to the area or industry can gain through the Target 10 activities and networks. Table
10 lists some of the themes that have been the subject of more than two stories. The most frequent
theme described in the stories (11%) concerns how farmers experienced an increase in production
after adopting Target 10 practice.
Table 10 Description of themes that have been the subject of multiple stories
Main theme %
Change to Target 10 practice leads to increased productivity (pasture growth/utilisation, milk production or condition score)
11
Positive reaction to the Target 10 Project (useful information, unbiased information or meeting farmer needs)
9
The importance of discussion groups and sharing information 6
Change to Target 10 practice leads to saving money 6
A change of practice after making systematic decisions about operational issues rather than using rule of thumb
5
Empowered to make a strategic long-term decision 5
Identification of operational problem using skills acquired on program 5
Systematic operational decisions lead to feeling more in control of business 5
Empowered to deal with external experts through skill acquisition 5
Evidence of farmers taking up on-farm practices that were recommended by the project
5
Overcoming negative attitudes toward the project or project messages 4
New farmers to the industry gain valuable network through the project activities 3
Gaining a new understanding of the importance of the basic principles of the pasture management
3
The importance of learning to think rather than following a recipe approach 3
It is also clear from the stories that the storytellers value the concept of empowerment and of
farmers thinking for themselves rather than using a recipe book approach. In total, 34% of the
stories make some mention of increased control, independence, or empowerment.
186
Levels of Bennett’s Hierarchy to which the stories correspond
The stories cover a whole range of topics and subjects. To gain a picture of the spread, the stories
have been categorised into broad themes and arranged using Bennett’s Hierarchy of outcomes2 see
Table 12. However, it should be pointed out that each story may contain several themes, so the
stories have been categorised by the theme that rates highest in the hierarchy of outcomes. For
example, a story categorised as practice change will generally also contain themes such as changes
in knowledge and attitude which rank lower in Bennett’s Hierarchy of outcomes.
It is clear from Table 11 that stories that are situated higher in Bennett’s Hierarchy of outcomes are
more likely to be selected by the Central Executive Committee. The purchaser group also allocated
high scores to stories that contained themes relating to the higher levels of the hierarchy (Level 7 or
6). Thus, it can be inferred that most stakeholders of the project value the stories that are about
behaviour change and consequences of this change.
Table 11 Level of Bennett’s Hierarchy
Bennett’s level Percent of total Percent of selected 7 – Consequences 27% 50%
6 – Behaviour change 14% 21%
5 – KASA changes 35% 17%
4 – Reactions 15% 4%
2 Bennett describes a chain of events assumed to characterise most programs in extension (Bennett 1977). He
uses this chain of events to depict a hierarchy of objectives and evidence for program evaluation. Bennett lists
seven levels of goals in extension and claims that it becomes more difficult to evaluate at higher levels of the
hierarchy, as it becomes more difficult to show that changes at these levels are the result of extension activity
and not of other factors.
187
Table 12 Grouping of stories by theme using Bennett’s Hierarchy
Level of Hierarchy - Description of theme and number of stories belonging to each category (N=128) totalQuality of life increases
1 (1 selected)
Goals achieved in career steps
3 (2 selected)
New farmers to the industry gainvaluable network4
Level 7Consequencesof change
Improved pasturegrowth/consumption7 (1 selected)
Increased milkProduction6 (2 selected)
Production maintainedin harsh conditions1
Increased conditionscore of herd3 (1 selected)
Money is saved
8 (5 selected)35(12)
Level 6
Changebehaviour
Evidence of farmers takingup T10 recommended on-farm operational practices5
Empowered to make astrategic business decision
7 (2 selected)
Changed practice in terms ofcommunication within family
2 (1 selected)
Changed operational practiceafter systematic analysis offigures rather than rule ofthumb 5 (2 selected)
18(5)
Level 5 –KASA
Skills andempowerment
Identifyingoperationalproblem throughacquisition of newskills
6
Calculationsdoneindependently
3 (1 selected)
Calculationsbased on accuratefigures rather thanapproximation –lead tobreakthroughs4
Empowered todeal with externalexperts throughskills acquisition
6 (1 selected)
Systematicoperationaldecisions madelead to feelingmore in control ofbusiness6 (1 selected)
Women gainingmore recognitionin family throughskill acquisition
227(3)
Knowledgeandawareness
New KnowledgeBecomes incorporatedinto the prevalentculture2 (1 selected)
Gained a newunderstanding ofimportance pasturemanagement4
Gained deepertechnical knowledge ofoperational process
1
Gained deeperunderstandingstrategic thinking andbusiness planning2
Importance of learninghow to think ratherthan following recipe
413(1)
Attitude Overcoming negative attitudes toward the program2
Attitude towards an element of the Target 10 message3 5
Level 4Reactions
Project providesuseful, goodinformation4
Project providesunbiased advice
2
Project is meetingfarmers needs
2
Importance ofteam work
2
Importance ofon-farm days
2
Importance ofdiscussion groups
7- (1 selected)19(1)
Lessonslearned
FarmerOwnership
1
Confidentiality
1
Limits of theprogram
1
Needingfigures to solveproblem
1
What canhappen withonly partialunderstanding1
Comingunstuck inharsh seasonalconditions-2
The need forone to onesupport
18(2)
Other Evidence of the spread of Target 10 messages and practices to non-Target 10 participants3 3
188
Level 7 Consequences for the target group – 26% of stories
In total, 26% of the stories concerned Bennett’s higher level consequences. The largest proportion of
these stories concerned increases in production after adopting Target 10 practices. This was
expressed in terms of increased milk production, improved pasture growth, improved pasture
utilisation, or increases in the condition score of the herd. Several stories told of how farmers had
made financial savings after putting into practice knowledge that had been gained during Target 10
programs.
In addition to issues of productivity and profitability, there were some stories that related to overall
improvement in family quality of life, and the achievement of life or career goals. Another
unexpected consequence told in three stories was that the project had provided an invaluable social
and business network for farmers new to the industry or geographical area.
Level 6 Behavioural changes in the target group – 14% of stories
Stories categorised into Level 6 concerned behaviour change, but did not explicitly mention the
consequences of these changes. The changes described in this category ranged from operational
changes to long-term strategic changes based on considerable planning. There were also stories of
changed practice with regard to communication between family members.
Level 5 Changes in: knowledge, attitude, skills, motivation and group norms –
34% of stories
The most frequent themes in this category referred to issues such as empowerment and feeling more
in control of the farm business as a result of increased knowledge and skills. There were also several
stories which demonstrated skill acquisition in terms of how farmers had been able to solve technical
problems using skills that they had gained from attending Target 10 programs. Several stories were
about farmers gaining a much deeper understanding of why certain basic operational practices
(especially pasture management) are so important.
Level 4 The farmer’s opinion about extension activities - 14% of the stories
Stories that were not specifically about change, but concerned farmers’ positive opinions of project
activities fall into this category. It is striking that seven of these stories were independently written
on the importance of discussion groups – discussion groups are clearly a most valued event.
189
Other 9% of the stories
Some of the stories did not fit easily within the framework that was adapted from Bennett’s
Hierarchy. This was especially true of some of the stories about lessons learned. Three of the main
messages that came out of these stories were that:
• farmers may need more support in implementing new practices during difficult seasonal
conditions
• farmers with only partial knowledge of the Target 10 practices can run into serious problems
• farmers should be consulted before major changes to the programs are made.
Characteristics of stories selected by Central Executive Committee
The most frequent theme in the stories selected by the Central Executive Committee was that of
money being saved after a farmer adopts a recommended Target 10 practice (see Table 13). This is
reflected in the feedback that was given from the committee to the storytellers – that stories could be
improved by including some element of the ‘bottom line’ and should contain some tangible outcome.
The stories did change over the year and the regions obviously tried to take note of the feedback
given by the Executive Committee. After about six months, feedback indicated that novel stories
were more likely to be selected than stories that contained themes that had been heard before. Again,
this is illustrated by the fact that stories that had a unique theme, such as new knowledge becoming
incorporated into the prevalent culture, were selected despite the fact that they represent lower level
outcomes. Another factor that affected selection was that stories were more likely to be selected if
they clearly showed evidence that directly connected the occurrence of change to a Target 10
activity.
190
Table 13 Characteristics of stories selected by the Central Executive Committee (n=24)
Theme Level of Bennett’s Hierarchy
Number of stories selected
Percent selected from
sub-group Quality of life increases 7 1 100% Goals achieved in career steps 7 2 67% Change to Target 10 practice leads to saving money 7 5 63% Change to Target 10 practice leads to increased productivity (pasture growth/utilisation, milk production or condition score)
7 4 27%
Changed practice in terms of communication within family 6 1 50% A change of practice after making systematic decisions about operational issues rather than using rule of thumb
6 2 29%
Empowered to make a strategic long-term decision 6 2 28% New Knowledge Becomes incorporated into the prevalent culture
5 1 50%
Calculations done Independently
5 1 33%
Systematic operational decisions made lead to feeling more in control of business
5 1 17%
Empowered to deal with external experts through skills acquisition
5 1 17%
The importance of discussion groups 4 1 13% Lessons learned Other 2 22%
Feedback given by purchasers
Prior to the round table meeting, the purchasers were asked to score the stories individually (out of
10). When the scores for all 24 stories were examined, it became apparent that the purchasers-
respondents had very different reactions to the stories. Two stories were allocated the lowest score
by one purchaser, and the highest score by another! The purchaser group did not have a unified
vision as to what is desirable when given a choice of these stories. It was also apparent that most of
the stories were considered to have merit by at least one of the participants. This finding supports
the concept that evaluation is conducted in a value-pluralistic context; that the various stakeholders
hold differing values. Thus, negotiation and dialogue between the various evaluation stakeholders
(including between the purchaser and the provider) is essential.
There was general consensus at the meeting that overall the stories demonstrated: • Technical skill improvement after attending Target 10 programs.
191
• An array of unexpected positive outcomes achieved by the project and specifically with regard to
changes in attitude, achievement of personal goals, and changes in the farm business and family
situation.
Statements by two of the purchasers are provided in Text box 2 as examples of these comments.
I was taken by those stories that tended to indicate that not only had the program delivered the technical outcomes that are being sought by the industry, but also that, for a number of participants, it had brought about a change in attitude and achievement of personal goals as well.
Some of the description of the logic that flowed through stories (the technical logic) was very, very good and showed that these farmers in question had really grabbed this technology by the throat and were really milking it. The other sort of stories that got to me were the life-changing ones. And those two types of stories really got to me.
Source: Transcript from Round table meeting
Text box 2 Examples of comments from the purchasers concerning the stories
Why specific stories were valued highly by the purchasers
The discussion held at the round table meeting revealed that specific stories were highly valued by particular participants when: • the change described in the story was clearly attributable to the project
• the participants could relate to the story from their own experience
• the change described showed a diversity of learning
• the story demonstrated project reflection on negative events and subsequent learning
• the change described in the story demonstrated teamwork in action.
In most of the cases, there were considerable differences in interpretation of the stories; however, a
full consensus was achieved that the story ‘knowledge is power’ presented the most significant
account of change (see Text box 3). Because of this, considerable time was spent discussing why
this story was considered to be so significant. Comments from the meeting included that the story
represented the sort of outcomes that they were looking for from the project because:
• the event described in the story was the combination of the farmer having an articulated need and
the training program being available at the right time
• the program provided a stepping stone to another job and also into further education
• it shows a congruence of personal goals and business goals
192
• this story illustrates an example of how the program provided a springboard for farmers to go on
to what they want to do without prescribing a single pathway
• the farmer in this story has the right attitude
• how can we harness the learning from this story to learn how to encourage other young farmers to
have similar success?
The stories themselves
Texts boxes 3, 4, 5 and 6 provide four stories to illustrate points that have been made in the
discussion. The purchasers of the project gave Story 1 the highest score. The first story entitled
knowledge is power, was categorised as being concerned with ‘goal achieved in career steps’, an
outcome relating to Bennett’s level 7. This story concerned a farmer who had attended several
programs of the project. The feedback notes from the selection process often stated that the stories
written by farmers were more powerful, as they were first-hand accounts of farmer experience.
The second story entitled I’ll not be milking cows when I’m 55, was categorised as a story
concerning changes to the ‘quality of life’, an outcome relating to Bennett’s level 7. This story has
the characteristic conversational tone that was often present in stories that were recorded orally, and
were valued for this characteristic. It is notable that Stories 1 and 2 were written by farmers, and
were concerned with the highest level of Bennett’s Hierarchy, and that both were valued highly by
all the committees and staff.
However, the majority of stories in the process were written by staff, and were written in the second
person. Story 3, empowered, was written by a member of staff. Several stories with this theme of
‘empowerment’ were collected over the year, and this outcome was categorised as a Bennett’s level
6 outcome. This particular story generated lengthy discussion when it was reviewed. The outcome
of ‘empowerment’ is not highly related to the overarching production goal of the project – this is the
sort of story that ‘surprised’ some of the purchasers.
Story 4 entitled Saved me $2,000 is a concise story written by an extension officer and details short-
term savings gained from Target 10 activities. This story was particularly valued by the project as it
was a short story with tangible profit-based outcomes. Few stories were collected that concerned
long-term profitability or productivity increases, and stories collected under the domain of ‘changes
in profitability and productivity’ were generally short term in nature.
193
Selected at the round-table meeting as the most significant story
Story 1 Title Knowledge is Power Name of person recording story: MJ, Dairy farm employee Region: Gippsland Date of narration: Round 5 – 15 March 1999 Where did this happen: XXX When did it happen: 1996
What happened? In 1996, I was working on a 500-cow dairy farm and had no input to the day-to-day running. But I wanted to have more input. I decided to do something about it, so I attended the Target 10 Grazing Management Program followed by the Nutrition Program. After completing these courses, I had enough knowledge to start making decisions, such as measuring growth rates and adjusting rotation lengths. Gaining this knowledge led me to start a new job on a 550-cow dairy farm where I am presently managing all feed requirements.
I have been able to maximise pasture consumption to 10 t/ha; and, with 1 to 2 tonnes of grain fed, this will achieve a production of 1,000 kg of solids /ha on 850 mm of rainfall per year. I walk the farm weekly to measure growth rates, allowing me to adjust the rotation to suit the growth rate. I fill the gap with grain so as not to waste pasture. When harvest comes, I can use this method to cut maximum fodder without sacrificing the cows’ pasture.
Why do you think this is a significant change? These changes, which I have been able to apply to the day-to-day running of the farm, have made the farm more efficient. I am also a more efficient employee now, and I have seen what I can achieve by furthering my studies. Last year I completed a Diploma of Agriculture (dairy farm management), and this year I’m studying an Advanced Diploma (dairy farm management). My future plan is to manage the whole day-to-day running of a larger scale dairy farm in every aspect.
Feedback from the Central Executive Committee: • This story is a good example of one person going on to do more learning and expand his horizons. • It is good, as it is written by a farmer. • It is about building confidence, and the story even got down to changes in productivity. • It really shows the full picture.
Feedback from the Round-table Meeting: • Good positive story about practical learning. • Real change directly related to having done Target 10 programs (they look good in the resume)
with measurable results as well. • Substantial behaviour change as a result of the Target 10 program. But the profit/productivity
impact is not so clear. • Great story. Self-improvement outcomes, the subject has strong commitment to the industry and
the desire to achieve. These are the people that will make the industry move forward. • To me it is raw efficiency; that’s what really gets me. It is almost the most you would expect to get
from a program. The guy is really ready to roll and he has got the right attitude.
Text box 3 Story example 1
194
Title of story 2 I’ll Not Be Milking Cows When I Am 55 Name of person recording story: MS, dairy farmer Region Gippsland Date of narration: Round 2 – 21 August 1998 Who was involved: Farmer and family
What happened? We did the pilot Dairy Business Focus Program in March; and for the first time, my wife came along to something. We were able to look at our farm as a business, not just as a farm. As a consequence of doing the program, we did a few sums and made a few decisions. We worked out that we can afford to have her on the farm, and she has left her job at the bank. We will generate enough income on the farm to make it more profitable for her to be here. The kids will benefit from seeing her a lot more, and they won’t be in day care. So far this year, this has made the calving so much easier, we have a joint input, and it has been such a turn around in my lifestyle. It has been so good.
We actually went to the accountant yesterday to get some financial advice on how we should be investing off-farm. He was amazed that what we are doing is treating the farm as a business. I said: ‘Now everything that we earn on this farm is going to be put away so that I am not milking cows when I am 55 years old!’
We have got a debt-reduction program running for the next 12 months, but after that the money will be channelled to off-farm investment. I want to retire young enough to enjoy what we have been working towards for the last 20 or 30 years. My boss is 77 and is still working on the farm. If I am that fit when I am his age, I want to be touring around the world.
It has opened up our lives. We are now looking at off-farm investment, as capital investment on-farm is not that great. We are not going to invest in new machinery but are going to invest in contractors to do any work we can’t do. There is no point buying new machinery, as it depreciates. Instead, we will buy shares and invest off the farm. This proves that you can farm on 120 cows, you don’t have to get big, and you don’t have to milk a lot of cows. It just depends what you do with your money. If only we could educate the younger farmers to think ahead instead buying the largest SS Commodore or the latest dual cab. I followed the same track for a few years until we sat down and worked out where we were going and where we could be. We made a few mistakes in the past, but the past is the past.
Feedback from the Central Executive Committee: • This story generated lots of discussion. But is it really about profitability or quality of life or changes in farm practice? • The general consensus was that there needed to be more detail in the story for it to be about profitability. • It is a really powerful story that shows considerable change.
Feedback from the Round-table Meeting: • The story showed strong evidence of attitudinal change, leading to self-improvement and goal setting. These people will be high achievers and reap the rewards. They will be good role models for others who desire similar rewards. • This approach is okay, but it isn’t necessarily a prescription for others. • It has some good messages, but it hasn’t got all the answers. • This is a very good example of achieving the goal of the DBF Program: i.e., getting strategic thinking/planning followed by farmer action. • I liked this story as it highlights the diversity in personal goals and ways to get there.
Text box 4 Story example 2
195
Title of story 3: Empowered Name of person recording story: Julie Williams, Target 10 extension officer Region: Gippsland Date of narration: Round 5 – 1 April 1999 When did it happen: Between September and March 1999
What happened? At a review day of the Dairy Business Focus Program to share how people had gone with their planning, one of the participants, Barb, was very pleased with herself and feels that she is now more ‘empowered’. She has employed a consultant to help with decisions and bring new information into the family. Her husband grew up on a dairy farm and feels he knows best. The third person (the consultant) has helped her introduce new efficiencies and ideas.
She has recently organised her own holiday, something she’s never done before, and she’s making decisions for herself. Her husband has not supported her before, but now she believes she can do it. There is information available that she now has access to and is now confident to get on with it. She is also keen to do the DFPA Program, with a bit of help, to help with analysis of the farm costs and to help with decision- making. Why do you think this is a significant change? The interaction with the DBF group and the awareness the program created has empowered this woman to seek out assistance to be more proactive in her decision-making.
What difference will it make in the future? It will help her make more effective decisions in her farming business and her life generally.
Additional note from Gippsland Regional Committee: • I should add that Barb actually went on holiday on her own, but this is not a story about the family
breaking up. She got a lot out of the program and is now actively getting out and following some of the plans that she made during the DBF Program.
Feedback from Central Executive Committee: • This story shows a considerable degree of change has occurred due to the program. She was left
empowered to do things that she could not do previously. • I chose this story because of the magnitude of change. • Before it seems that the husband made most of the decisions; now there are more people making
decisions in the family. That’s a huge change.
Text box 5 Story example 3
196
Title of story 4: Saved Me $2,000! Name of person recording story: Frank Mickan, Target 10 extension officer Region: South West Date of narration: Round 1 – 10 June 1998 Who was involved: A farmer in the Soils and Fertilisers Program Where did this happen: Heywood When did it happen: April 1998
What happened? A pilot Soils and Fertilisers Program was presented at Heywood in South-West Victoria. Some calculations were done on day two; and by day three, a very ‘switched-on’ farmer had gone home and calculated the costs of nutrients that he had calculated that he needed for his farm. He had quotes from at least two fertiliser companies, and he calculated that a blend (from his own figures) would save him $2,000.
Why do you think this is a significant change? This man had obviously grasped the concepts of the calculations, shown by his ability to calculate the costing and levels of nutrients in the fertiliser blend.
What difference has it made already or will it make in the future? After two days of the program, this farmer had already saved himself $2,000 and will continue to do so in the future (although the amounts may vary). He can now check on the consultant’s recommendations.
Feedback from the Central Executive Committee: • I like it as it shows how a farmer applied what he had learned in a program to save him money. • It shows that the information that is given in the programs does work. • There is also an issue here about how the programs can improve the confidence of farmers and so
help them to help themselves (empowerment). • This story also illustrated how the farmer changed his behavior.
Text box 6 Story example 4
What the stories revealed in terms of evaluative information: my observations and analysis
En masse the stories present a picture of many farmers implementing part or all of the Target 10
message, and of farmers gaining from the programs in unexpected ways. The most frequent theme
for a story (and also the most frequently theme of selected stories) concern farmers who have
changed to Target 10 recommended practices and gained an increase in production. The second most
common theme for a story concerned farmers who had adopted of Target 10 practices and as a result
experienced short-term financial gains. These findings are consistent with the aims of the project,
and support the other evaluation findings (Target 10 Evaluation report 1999), that the Target 10
activities do have an impact on increased production and short term profitability.
197
However, in addition to stories concerning production and profit outcomes, many stories concerning
other types of change were collected and subsequently selected by the Target 10 committees. The
major themes running through these stories were that farmers, after involvement Target 10 programs,
felt:
• more in control of their business
• empowered to challenge the consultant
• that they had gained increased family communication after the identification of mutual goals
• empowered to make strategic long-term decisions
• able to make systematic decisions about operational issues rather than using rule of thumb
• able to achieve goals in career steps.
As the process progressed, we noted that the stories had improved and that the authors of the stories
were getting more skilled at knowing how to present the stories and what sorts of themes should be
presented in the stories. After six months, the stories seemed to be more ‘finished’ and to be more
closely related to change in the specified domains. The implication from this is that not only was the
project learning to run the process, but also the committee members (farmers, educators, industry
representatives, and NRE staff) were getting better at conceptualising impact.
The process of collecting and analysing stories has seen farmers, collaborators, and NRE staff sitting
together at committee meetings discussing and interpreting qualitative data, casting evaluative
judgements and negotiating about what constitutes a significant change. Feedback from the central
executive committee suggests that learning has also occurred in terms of increased skill in
conceptualising and capturing impact; over the year the storytellers became better at capturing
impact and responding to the suggestions that were provided in the feedback from the story review
process.
In total, 26% of the total stories collected deal with outcomes, both intended and unintended, that can
be classified as ‘the consequences of change’ (Bennett’s Level 7), which are typically found to be
hard to measure and difficult to attribute to project intervention though conventional evaluation
methods.
198
7.5 Problems During the Implementation
Competition
Early in the trial, one of the regional champions brought our attention to the fact that competition
amongst storytellers was becoming an issue in her region. At this stage staff were voting for stories
at the staff meeting. A situation occurred whereby three stories were selected by the staff team to go
to the regional committee, but by accident four stories were sent. The regional committee gave the
story that had slipped in by mistake the highest vote. This story also went on to be voted by the CEC.
Many people commented that it was a very good story and could not understand why it had not been
selected at the staff meeting. It turned out that certain people started voting for their friends’ stories,
at the expense of others. Things had become quite competitive, and people began to feel pressured.
Because of this, voting was subsequently confined to regional committees, and the CEC. At the staff
meetings, the stories were shared and commented upon, but no voting occurred.
Some regional committee members became very enthused when they were getting lots of ‘winning’
stories and disillusioned when their stories didn’t get accepted. The competitive factor was a problem
in a lot of ways. As an aside – my observations as a non-Australian, and having lived in both Hindu
and Muslim cultures are that under an Australian culture, people often dislike processes involving
comparison and promotion of individuals above others – and the MSC process is all about
comparison! This concept is colloquially named ‘the tall poppy syndrome’. It could well be that this
was not an issue in the Bangladesh where the approach was developed due to cultural differences.
Time
Participants frequently lamented that the biggest problem with the MSC process was the amount of
time that it consumed before and during meetings. Reading out the stories, discussing and voting for
them frequently took over an hour, and occurred at the majority of the meetings of the project. Some
felt that this time was justifiable in term of the benefits derived and others did not. This issue is taken
up further in the next chapter, in which an evaluation of the MSC model is presented.
199
Issues of culture and ethics
Early on in the process, it became clear that the approach advanced by Davies had not focused on
issues of ethics and confidentiality of the storytellers. There is a huge omission in the MSC process
described by Davies, in that it does not protect the identity of the informant. Perhaps it is telling of
the situation of farm families in Bangladesh, that confidentiality of the actors mentioned in the
stories was not mentioned as an issue. In fact, in the Bangladesh case the stories were seen to gain
credibility because they contained enough information for the storyteller to be located and the story
to be verified. When the approach was transposed to Australia, the confidentiality of the stories
became paramount. In the application of the MSC process to the Australian context a process was
needed:
• whereby people recording stories of their own experience gave and recorded their consent for
these stories to be used in evaluation
• whereby people recording stories that involved others, either render the stories anonymous, or
seek consent (and record this) from the subjects of the story
• for tracking whether the subjects have given their full consent
• to ensure that the stories are not used as evaluation material in a way which divorces them of
context
• for ensuring that stories are not published without going through the project committees and
gaining approval from the storyteller
• in which committees have some agreed rules and adhere to them as far as ethics are concerned.
In December 1998, protocols were set in place to deal with issues of confidentiality and ethics.
Firstly, a question regarding whether the subjects had granted permission for the story to be used for
evaluation was added to the proforma. Those stories that were placed in the booklet Target 10
Evaluation Stories were carefully dealt with, and all the subjects who featured in the stories were
contacted and their consent sought before distribution. However, there was still much debate about
how to protect the identity of subjects in the stories, especially stories that contained some element of
negative news.
During the 12-month trial, several people asked whether the stories could be used to place in
newsletters or to use as ‘good news stories to give to the minister’. From the beginning a rule was
200
made that no story could be published without the approval of the CEC. However, we continued to
debate whether it was ethical to use the stories for purposes other than evaluation. In the end, the
ethics code from Market Research Association was followed. This required that no stories collected
on the grounds of evaluation could then be used for other purposes. I suggest that if the subject
thinks they are giving information for one purpose it is unethical to use it for another.
7.6 Conclusions Concerning the Implementation of the
MSC Process
It was possible to implement the MSC process across a large Victorian extension project, albeit in a
slightly modified form than that advanced by Davies. A reference group appears to be an important
addition to the model, in order to guide the process, and allow it be adapted to the local context. To
successfully transpose the approach to a different culture (from a Bangladeshi savings and credit
project to Australian extension project) new processes were needed:
1. To get the ball rolling, some pre-implementation activities were needed. These ensured that
participants fully understood the process and were willing to become involved.
2. To manage the complex process of collection and selection of stories, people are needed to
facilitate and organise the process. One way of doing this was to appoint regional champions,
who together form a learning group for the process.
3. A process was needed to gain the consent of the storyteller, and to be able to track this consent.
4. In this Australian context, competitiveness and pressure became an issue that needed careful
management. Where possible an environment should be created to minimise unhelpful
competition by careful choice of the fora where stories are selected.
201
CHAPTER 8
THE IMPACT OF THE MSC PROCESS ON THE TARGET 10
DAIRY EXTENSION PROJECT
In this chapter the findings of a meta-evaluation of the MSC process that was implemented
across the Target 10 Dairy Extension Project are presented. Firstly, a model was constructed
of how the process was thought to bring about improvements to the project. From this model
nine ‘conjectured outcomes’ were identified including: increased participation, increased
knowledge, increased staff morale, and operational and strategic changes to extension
practice. Secondly, empirical evidence was used to test whether the conjectured had been
achieved, and if so, in what project contexts. Significant differences were found between
regions in the extent to which staff perceived some of the outcomes had been achieved. I
suggest that these differences were associated, amongst other things, with the structure and
nature of the meetings at which the stories were reviewed. Despite the differences in level of
success, the MSC process was able to meet many of the current evaluation needs of the
Target 10 stakeholders. The MSC process also appeared to complement the project’s existing
evaluation practice.
8.1 Introduction
In Chapter 7 the process of implementing the one-year trial of the Most Significant Change (MSC)
model was described and some of the outputs of this process were presented. To determine the
effectiveness of the MSC model it is important to look further than the outputs of the process, and
also to determine the value of the MSC model in general. To this end, a meta-evaluation study was
carried out to determine, firstly, the extent to which the MSC process was able to meet the case
project’s evaluation needs, and secondly, the potential of the MSC process to effectively meet
evaluation needs of extension projects more generally. The aim of this chapter is to present the first
part of the findings of this meta-evaluation. So here the emphasis is on investigating the extent to
202
which the process was valuable to the Target 10 Project and able to bring about project
improvements. In Chapter 9 an emphasis is placed on determining the value of the MSC model for
Australian agricultural extension projects in general.
In the first part of this chapter I focus on testing a series of conjectured CMO (context, mechanism,
and outcome) configurations (see Section 6.5) that together represent a theory for how the MSC
process brings about project improvements. In the second part of this chapter I focus on how well
the MSC process was able to meet the articulated evaluation needs and provide other valued
outcomes for the Target 10 Dairy Extension Project.
8.2 Testing the CMO Configurations for the MSC Process
In Section 6.5 an inductive process of developing key conjectured CMO (context, mechanism,
outcome) configurations for the MSC process was described. Nine CMO configurations were
identified, which represent testable hypotheses of how the MSC model brings about evaluative
activity and organisational learning and hence, project improvements (Table 14). In Section 8.4,
each of these nine conjectured CMO configurations are presented, and tested against empirical
evidence.
To test the conjectured CMO configurations (shown in Table 14), Pawson and Tilley’s (1997)
Realistic Evaluation model was followed in part, but not in entirety. A focus was placed on testing
the extent to which conjectured outcomes were achieved in different contexts. Less emphasis was
placed on testing whether the conjectured ‘mechanisms’ caused outcomes in specific contexts –
although mechanisms are referred to where the data existed. In each of the following sections that
refers to an individual CMO configuration, a figure is presented in which the mechanism is
presented in grey rather than black text – to signify the lower emphasis placed on mechanisms than
in the process advocated by Pawson and Tilley (1997). The mechanisms are also shown in grey text
in Table 14.
My rationale for placing less emphasis on mechanisms was related to the nature of the meta-
evaluation purpose, and the availability of data. The purpose of the meta-evaluation was not to test a
tight causal chain of events, but more to gain an idea of the sort of project contexts in which the
MSC process could contribute to project improvements. Pawson and Tilley stress the importance of
investigating how an intervention ‘works, and in what situations, and why’. In this meta-evaluation,
203
I was concerned with ‘how the MSC process worked, and in what situations’.
As explained in Section 6.5, a meta-evaluation questionnaire was developed for the purpose of
testing the relationship between the conjectured outcomes and different project contexts. The meta-
evaluation questionnaire (Appendix 2) was sent to all project staff and was returned by email. A
total of 28 people responded to the survey – a response rate of 80%. The survey was returned
anonymously, although most staff chose to state their names. However, it was possible to associate
names with most of the non-respondents in order to check the representativeness of the sample.
Five of the seven people who chose not to respond were very new staff who had not been involved
in the MSC process. Another person was away at the time of the survey. The non-respondents were
evenly spread across the four regions. Those who had been verbally antagonistic or pessimistic
towards the approach in the past did appear to be represented amongst the respondents. Therefore,
the people who responded to the survey appear to represent those most involved in the 12-month
implementation trial of the MSC model.
204
Table 14 Conjectured context, mechanism, outcome configurations for the MSC process
Context Mechanisms Outcomes
Those who participate are actively involved in the story review sessions.
People find the stories & interpretations interesting, engaging, and illuminating.
a) Increased attention paid to evaluative information
High attention to evaluative information: individuals are more likely to read/ pay attention, and remember the information.
Story review occurs in meetings that have a structure and atmosphere conducive to open discussion and exchange of views between staff and farmers.
Farmers and other committee members become actively involved in the evaluation process
b) Helps committee members to have a voice in the evaluation and to steer the project
As committees represent the wider farmer ‘voice’, this will help the project to continue to respond to farmer needs.
Story review occurs in meetings that have a structure and atmosphere conducive to open discussion and exchange of views between staff and farmers.
• The MSC process results in regular feedback with regard to the impact of extension activities.
• The competitive processes
whereby some stories (to which individuals feel ownership) are selected over others.
c) Increases morale
Reading / hearing success stories increases morale of staff and committee members. Having a story of significant change selected is also morale boosting for those who feel ownership over the events described in the story.
Story review occurs in meetings that have a structure and atmosphere conducive to open discussion and exchange of views between staff and farmers.
Regularly sharing and discussing the stories about outcomes that have occurred in farmers lives in relation to project activities.
d) Knowledge of impact
Participant’s gain new knowledge with regard to what has been achieved, what impacts are desirable.
Story review occurs in meetings that have a structure and atmosphere conducive to open discussion and exchange of views between staff and farmers.
Project stakeholders debate and reflect on project activities on a regular basis throughout the project.
e) Knowledge of each others’
values
Participants gain more knowledge of what the different stakeholders value.
205
Story review occurs in meetings that have a structure and atmosphere conducive to open discussion and exchange of views between staff and farmers.
Project stakeholders debate and reflect on project activities on a regular basis throughout the project. Inter-organisational reflection, communication and dialogue is enhanced.
f) A more fully shared vision
between all project stakeholders
Over time, the increased communication and dialogue will bring about a more shared construction of what has been achieved and what is desirable.
An organisational structure can accommodate change. That stakeholders are willing to enter into dialogue with regard to information gained from the process.
Feedback from the MSC process provides new knowledge with regards to the information requirements of different stakeholders.
g) Beneficial changes in the
way that the MSC process is run
Evolution and changes in the way the MSC process is run so that it can better meet the project’s needs.
Story review occurs in meetings that have a structure and atmosphere conducive to open discussion and exchange of views between staff and farmers.
• In their extension practice
staff try to reproduce outcomes deemed successful and avoid traps described in the stories of ‘lessons learned.’
• Decision-makers gain new
knowledge from the interpretation of stories and from the feedback from the purchasers. They may even examine specific stories to help in planning.
h) Beneficial operational
and strategic changes in practice
Extension staff make operational changes. Those involved in planning make strategic changes to the project through what they have learned in terms of evidence of impact, and stakeholder values of this impact. Purchasers make more informed funding decisions.
Story review occurs in meetings that have a structure and atmosphere conducive to open discussion and exchange of views between staff and farmers.
• Project members learn how to
better detect, reflect, and collectively respond to outcomes at the field level.
• Project members learn how to
better detect, reflect, and collectively respond to changes in stakeholders concerns and wishes with regard to the direction of the project.
i) Organisation develops a
better and more sustainable capacity to learn and change
By having a process in place to examine and challenge the value of what is occurring at the field level, the project team may gain a sustainable capacity to react to the changing external environment i.e., address farmer needs & purchaser requirements.
206
8.3 Different Contexts
The results of the meta-evaluation survey indicated strong patterns relating responses to regions.
Before examining the extent to which the conjectured outcomes were realised, contextual
differences between the four regions of the project will be outlined. The main quantifiable
differences concerned the number of staff attending the respective regional committees (Table 15),
the number of farmers and other people who attended the committees (Table 16) and the number of
stories contributed by people other than the project staff (Table 17).
Table 15 Staff attending their regional committee meetings during the trial
Regional location of respondent Northern Irrigation
Region South West
Region Gippsland
Region North East
Region 2/8
(25%) 7/7
(100%) 7/11
(65%) 2/2
(100%)
Table 16 Attendance at regional committee meetings during trial
Northern Irrigation Region
South West
Region
Gippsland Region
North East Region
Number of staff 2-3 6-7 4-6 2 Number of farmers 4-6 2-6 4-6 6-8 Number of other nominated committee members
1-3 7-14 1-2 1
Total 7-12 15-27 9-14 9-11
Table 17 Number of stories contributed by farmers and other non-staff committee members per
region
Region Number of stories by farmers
Number of stories from
other non-staff committee members
Total number of stories
Northern Irrigation Region
1 4 38
South West Region 4 4 39
Gippsland Region 13 3 46
North East Region 0 2 9
207
The Northern Irrigation Region
During the 12-month trial of the MSC model, staff from the Northern Irrigation Region attended
fewer regional committee meetings than staff from other regions. The practice was for between one
and two members of staff to attend the regional committee meetings which farmers attend. The
regional structure resulted in fewer staff attending regional committee meetings than in other
regions (see Table 15). This was the only region where stories were selected at the staff meetings.
Therefore, extension staff experience of the process was largely associated with discussing the
stories among staff members rather than committee members. In this region staff collected all but
one story, which was collected by a committee member (see Table 17). During the 12-month trial
there was also considerable staff turnover in this region.
The South West Region
The South West regional committee meeting is large, and all staff are expected to attend (see Table
15). In addition to extension staff, other staff from NRE are also members of the committee (largely
those working in dairy research). At the time that the evaluation was conducted, the committee
membership was 41 and was made up of:
• 10 farmer members with between two to six regular attendees
• 12 industry representative members with around eight regular attendees
• five educator members with two regular attendees
• 14 NRE staff members with seven regular attendees (who are the extension officers).
Up to half of the committee members attended the meetings, with some people attending
sporadically. These regional meetings were larger and more formal than regional committee
meetings in the other regions. The meetings also tended to be lengthy and have a full agenda.
The Gippsland Region
Gippsland has a very different committee structure to other regions, with three sub-regional
committees in addition to the regional committee. There was a high representation of farmers at the
regional committee, and lower representation of staff (see Table 16). The other staff attend the sub-
regional meetings, where farmer participation is also high. The meetings tend to be informal in
structure and appear to have relatively regular attendance. The Gippsland Region contributed
considerably more stories authored directly by farmers than the other regions (see Table 17).
208
Purchasers and the Central Executive Committee (CEC) members explained that stories authored
directly by farmers were particularly powerful. In confirmation of this claim, four of the five stories
selected by the purchasers originated from the Gippsland Region.
The North East Region
The North East Region is small, with only two extension staff. Regional committees are held less
frequently than in the other regions (every six months, rather than every 2-3 months). These
committees are attended by several farmers and are relatively informal.
A scale for different meeting contexts
In was hypothesised that the achievement of many of the positive outcomes from the MSC process
was dependent on the atmosphere and structure of the meetings where the stories were reviewed.
For this reason, the regions were placed into a continuum regarding the extent to which the
meetings (at which the stories were reviewed) provided an environment conducive to discussion,
reflection and exchange of views between different stakeholders (see Table 18). Each region was
allocated a score, which was used in quantitative analysis to test whether the structure and
atmosphere of meetings related to staff perceptions of extent to which beneficial outcomes were
achieved.
Table 18 Extent to which meetings in the different regions provided a context conducive to
discussion and exchange of views
South West Region Northern Irrigation Region
Gippsland Region
North East Region Size of meeting Large
(15-27) Small (7-12)
Small (8-13)
Ratio of farmers to other attendees
Low (between 1:4 and 1:7)
High (approx. 1:1)
High (approx. 1:1)
Level of exchange between farmers and other staff
Low (as the meetings are so
large)
Low (as few of the staff
attend)
High
Level of informality of meetings
Low Medium High
Score for environment conducive to open discussion and exchange of views
Low
1Medium
2High
3
209
8.4 Findings
The extent to which the conjectured outcomes were achieved in the different contexts is examined
in the following sections. The score allocated in Table 18 was used as a ‘proxy indicator’ for
context, and the findings were therefore examined for differences between the three categories of
context (concerning the structure and atmosphere of meetings where the stories were reviewed). In
some cases other variables were also used to investigate the different responses, including the
number of story review sessions the individual had attended, the number of stories read by
individuals and the extent to which the stories were remembered. The quantitative data was drawn
largely from the staff questionnaire, with supplementary data being used to help explain the
findings. The supplementary data came from a wide range of sources including face-to-face
interviews and focus group discussion, open questions in the questionnaire, and my observations of
the implementation of the MSC process.
a) Increased attention paid to evaluative information
________________________________________________________________________________
The first conjectured CMO configuration is shown in the flow chart above. It is hypothesised that
when people actively participate in the story review sessions then they will find the stories and
interpretations interesting and so will pay considerable attention to the evaluative information. This
hypothesis was tested by examining the relationship between the conjectured outcome and context.
The context was determined by the extent to which individuals had attended the story review
sessions. The attainment of the outcome of high attention paid to evaluative information was
judged by examining the extent to which individuals had read the stories, and to which they
remembered them. For this CMO configuration, the conjectured mechanism (that people find the
stories interesting) was also examined, but it was not tested against the other variables.
ContextThose who participate are actively involved in the story review sessions.
MechanismPeople find the stories and interpretations interesting.
Outcome High attention paid to evaluative information: • individuals are more
likely to read the stories • individuals are more
likely to remember the information.
210
Conjectured mechanism – interest in the stories
In the questionnaire, respondents were asked to openly comment on their impression of the stories
in general, and the responses were coded into a number of categories and presented in Table 19.
Responses were all generally positive; with many respondents making some reference to the fact
that the stories were interesting or enjoyable (58%); one respondent stated that they were ‘fantastic,
and inspiring to read’. In addition to the comments in the Table 19, two respondents stated that they
found the interpretation of the stories to be more valuable than the stories alone.
Table 19 Respondents’ descriptions of the stories
Main words to used to describe the stories % of responses
n=28* Enjoyable or interesting 54% Capture the impact of the project well 32% Inspiring or encouraging to the reader 18% ‘Real world’ feel or a ‘human’ approach 18% Powerful 11% Good approach 7% Good in that they illustrated different peoples views’ on the project 7% Easy to read 4% Important for advocacy 4% *One respondent chose not to answer this question and several respondents gave more than one response
However, one respondent stated that at the beginning of the 12-month trial the stories were not
meeting his expectations:
…they were a bit corny – a bit ‘Micky Mouse’ – I was not quite sure how these were going to go. So I
suppose I was more skeptical at the start’ (quote: project staff).
Feedback from the story review sessions at the CEC meeting stated that committee members felt
that the stories became better and more credible as the process ran, and as people learned what sort
of stories were valued (see Section 7.4). From the data it appears that that majority of staff did find
the stories interesting, and perceived them to improve over time. The demand for additional copies
of the ‘story booklet’ bears witness to this, with over 300 copies now having been circulated. The
interest in the stories (and interpretations) clearly extended beyond those immediately involved in
their collection and selection.
211
Conjectured outcome – attention to evaluative information
All respondents reported that they had received the booklet entitled Target 10 Evaluation Stories
(Dart, 1999c). This contained the 24 stories selected by the CEC over the year, and feedback from
the purchasers concerning these stories.
The extent to which individual staff had read the stories was examined against the number of story
sessions they had attended, but no significant correlation was found between these variables (Table
20). The two respondents who reported that they had not read any of the stories in the booklet, both
stated that they had listened to some of the stories reviewed at meetings. It can therefore be
surmised that all respondents were exposed to at least some of the stories.
Table 20 Number of stories read in booklet against number of story sessions attended
Number of stories read in the booklet Number of sessions attended
none a few most all Totals
0 0 0 0 0 0
1 0 1 2 0 3
2-3 1 2 3 1 7
4 or more 1 6 7 3 17
Totals 2 9 13
4 n=27
Gamma value =0.063, approximate significance =0.814 No statistically significant differences at p<0.01.
Respondents were also asked to comment on their ability to remember the stories. Their responses
were coded into eight categories and are displayed in Table 21. The majority of respondents
remembered at least some of the stories. There was no statistically significant relationship between
individuals’ memory of stories and the number of story review sessions attended, or with meetings
to which they attended.
212
Table 21 Extent to which respondents remembered the stories
Memory of the stories* Number of sessions attended
1 2 3 4 5 6 7 8 Totals
0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 1 1 0 1 3
2-3 0 0 0 1 4 0 0 2 7
4 or more 1 1 1 3 3 6 1 0 17
Totals 1 1 1 4 8 7 1 3 n=26**
Gamma value = -0.12, approximate significance =0.62. There was no statistically significant difference at p<0.01. * Memory of stories was scored from 1, (remember none) to 8 (remember all the stories). **There were 2 non-
respondents
While only 15% of the respondents stated that they remembered many of the stories well, it seems
that many of the respondents suggested their own ‘mechanisms’ for why they recalled particular
stories or parts of stories quite vividly. The reasons given for a particular story ‘standing out’ were
variable and included:
• that the respondent had written the story
• that the respondent knew the farmer referred to in the story
• that the respondent had experienced a similar event as described in the story
• that the particular stories were discussed at length at a meeting that the respondent had attended
• that the story had been selected at the CEC
• that the story was particularly ‘powerful’ in content.
In other cases, respondents remembered parts of stories; for example, one person remembered the
titles or the punch lines of the stories. Several people commented that the catchy ‘titles’ had also
helped them to remember the stories.
One respondent commented that an important beneficial outcome of the MSC process was that
many staff remembered ‘common stories’ and that this may have augmented the collective memory
of the organisation:
The fact that the stories contribute to the collective memory is a very important thing – considering
the really high turn over of staff in the project (quote: project staff).
213
There are two important points behind the concept of a ‘collective organisational memory’. Firstly,
having the stories documented and available means that new staff can tap into the past experience
and achievements of the project and what the stakeholders value. Thus, as old staff leave and new
staff enter the project, some of the collective experience is retained. Secondly, as many staff and
project stakeholders hold common stories and interpretations in their memories, this may contribute
to the creation of a shared vision as to what the project has achieved and how these achievements
are valued. This concept is further elaborated in Section 8.4f.
Relationship between the conjectured context and outcome
There was no obvious pattern relating the conjectured context (degree of participation in the
process) and outcome (attention paid to the evaluative information). In this case the conjectured
mechanism (interest in the stories) and the conjectured outcome (high attention to evaluative
information) seem to have been realised regardless of the context.
In all regions, and at all levels of involvement, people found the stories interesting, and read a
considerable number of them, and therefore paid considerable attention to the evaluative
information created through the MSC process. While most people seemed to remember parts or all
of some stories, the extent to which respondents stated that they could remember the stories varied
widely from person to person. It could be that people in general (regardless of context) find stories
interesting. This alternative hypothesis would concur with the sensemaking literature (e.g., Weick,
1995) that suggests that stories are the preferred sensemaking media in organisations.
b) Helps steering committees to ‘steer’ the project
________________________________________________________________________________
The second conjectured CMO configuration is shown in the flow chart above. One of the ideas from
the evaluation theory model was that the MSC process would help the committees have a ‘voice’ in
the evaluation, and help the committees to function well and to steer the project. This question was
posed to staff in the meta-evaluation questionnaire. This conjectured outcome is congruent with the
ContextStory review occurs in meetings that have a structure and atmosphere conducive to open discussion and exchange of views between staff and farmers.
MechanismFarmers and other stakeholder become actively involved in the evaluation process.
OutcomeThis helps the farmers to have a voice in the evaluation and help steer the project.
214
underlying conceptual framework of the project (Chapter 6, Figure 13). One of the facets of the
Target 10 conceptual framework is that ‘at the community level, the project provides opportunities
for stakeholders to participate in the development, organisation, and delivery of project elements, as
well as in communication strategies and collaborative alliances at state and regional levels’
(Boomsma et al., 1996).
Conjectured outcome – MSC process helps the committees to steer the project
Table 22 shows that over half of the respondents felt that the process had helped the committees
steer the project ‘a lot’ or ‘to some extent’. Nevertheless, some respondents felt that it had no effect
or negative effect.
Table 22 Staff perception of the extent to which the MSC process helped farmer committees to
‘steer’ the project against meeting context
Meeting context *
Response 1
South West2
Northern Irrigation
3Gippsland and
North East
Totals
Hindered a lot 0 0 0 0
Hindered to some extent 1 0 0 1
No effect 2 2 2 6
Helped to some extent 4 3 4 11
Helped a lot 0 2 6 8
Totals 7 7 12 **n=26 Gamma value =0.556, approximate significance =0.008, Statistically significant difference at p<0.01. * The meeting context was scored on a scale of 1-3, where 1 is low and 3 is a high. The scale refers to the degree to which the meetings (to which respondents attended) in different regions were conducive to open discussion and exchange of views between staff and farmers (see Table 18). ** 2 people chose not to respond to this question
However, there were statistically significant differences between the three meeting contexts. This
can be explained by examining qualitative data. In the South West participants found the process
less enjoyable and excessively time consuming and indicated that MSC had not helped their
regional committee to steer the project. Some participants explained that this could be due to
problems with the meetings themselves. Informants attributed difficulties with the MSC process to
the fact that the stories were seen as an extra task in an already busy timetable at the regional
committee meeting. A distinct concern was raised that the meetings were too large for everyone to
215
have their voice heard:
I think the pressure comes from the size of the group, because if you have a big group there is always
pressure for everyone to have their say. The meetings just drag on and on for that very reason, that
everyone wants to have their say and everyone wants to know what got discussed at the last meeting –
it could be a function of that. We are pressured for time in the Target 10 meeting. There’s another
agenda too here – and that’s quite a few of us in the SW feel that you go to meeting after meeting, and
we’d like to cut the Target 10 regional committee out. You have to bring a note from home if you
don’t go to the meeting! (Quote: extension staff from the South West).
In contrast, in Gippsland the MSC process was perceived as being highly beneficial to the work of
the regional committees. Here meetings are smaller, with between 8-15 people attending. They are
also more informal in structure and appear to have a more regular attendance than in the South
West. One respondent explained that the MSC process helped the regional committees to feel they
were contributing to the project:
One the things that we have had increasing difficulty with has been the roles of the regional
committees. Back at the start there was a whole heap of work for them to do in terms of developing
the content of the programs, organising launches and many more things. Over time the role of the
committee has changed – and there are not the same opportunities for the committees any more. In
the Gippsland Region the committee got a lot of enjoyment and satisfaction out of the story process –
and felt they were actually making a contribution to the project. They saw this more as the evaluation
than the other processes – such as taking attendance records at courses and identifying whether
learning objectives had been achieved and whether competencies had been achieved. Because the
committees were directly involved this – they don’t get involved in the rest of the evaluation (quote:
extension staff).
A recent report (Dart, 1999b) suggests that when program development is complete (as was the case
at the time of the 12-month trail of the MSC process), the role of the regional committee members
change. The role of farmers and industry representatives becomes one of overseeing project
activities, and they assume a more passive role in the committee structure. In this role, farmers have
less opportunity to influence the project and experience less satisfaction and sense of ownership.
Thus, the MSC process was implemented at a particularly opportune time, when there was less
opportunity for farmers to be involved, yet still a strong commitment to have a strong farmer voice
in the running of the project.
At the Central Executive Committee (CEC) meeting, the story review process was well received.
216
Participants who attended the CEC meetings suggested that the MSC process had been particularly
effective in encouraging communication in the meetings, especially in regard to farmers’
involvement in this dialogue:
I think there has been an increased communication between and within the committees as people have
talked much more openly. The stories have become a vehicle and an excuse for people to talk more
openly. It has been a way that people have increased their confidence to be heard at meetings –
because the stories are a subject matter which farmers are very familiar with. Management meetings
can often become abstract and people; especially farmers sometimes feel "do I have anything to
contribute to this?" But when you talk to farmers about something in the stories then you are talking
about something that is very familiar and dear to them. So they are much more relaxed and confident
when they are talking (quote: one of the project co-ordinators who attended all CEC meetings).
Relationship between the conjectured context and outcome
While no firm cause and effect claims can be made, the outcome (the MSC process helps committee
members ‘steer’ the project) does seem to be more likely to occur in the conjectured context (where
there stories are reviewed in meetings more conducive to open discussion and exchange of views
between staff and farmers). The conjectured context-outcome configuration does appear to offer a
plausible explanation for the data. The MSC process seems to have helped committees to ‘steer’
the project in contexts where there are small meetings with high levels of farmer participation.
Farmer involvement in Target 10 committees is seen as a crucial part of the conceptual framework
of the project; involving farmers in steering committees helps the project to be focused on farmer
needs, and helps build a community supportive of change. An unanticipated outcome of the MSC
process was that in some contexts it provided a role for these committees, at a time when their role
was uncertain.
217
c) Increased morale
________________________________________________________________________________
A conjectured outcome of the process was that hearing or reading ‘success stories’ would be
motivating for those who feel some ownership of the stories. A second associated outcome is that
having stories selected would also be motivating for those who feel some ownership over the
‘winning’ story. To test this conjectured outcome, staff were asked (through the use of the
questionnaire) whether the stories had helped staff morale. Following this, the qualitative and
observational data were examined for evidence about whether the competitive selection (at both
individual and region levels) had affected staff morale.
Conjectured outcome (1) – increased morale through hearing stories of
change
For some staff, the MSC process boosted their morale, especially through hearing how their work
had contributed to positive outcomes in farmers’ lives. Table 23 shows how 22/26 extension staff
stated that the approach had helped staff motivation ‘a lot’ or to ‘some extent’.
The data in Table 23 suggest that there were regional differences (although not statistically different
at p< 0.001). Staff from the South West Region were less likely to allocate a high scores for
motivation than the respondents from the other regions. One respondent from this region stated that
the process had actually hindered staff morale. This can be partly explained by examining the
qualitative data. Staff from the South West Region stated that they felt pressured into generating
stories and found this to be uncomfortable. These sentiments are illustrated through the extract
presented in Text box 7.
ContextStory review occurs in meetings that have a structure and atmosphere conducive to open discussion and exchange of views between staff and farmers.
Mechanism Regular feedback occurs with regard to the impact of extension activities.
The competitive process is motivating for those concerned with the ‘winning’ story.
OutcomeIncreased morale by: 1) reading / hearing success stories 2) having a story of significant change selected.
218
Table 23 Staff perception of morale boosting effect of the MSC process against meeting context
Meeting context *
Response 1
South West2
Northern Irrigation
3Gippsland and
North East
Totals
Hindered a lot 0 0 0 0
Hindered to some extent 1 0 0 1
No effect 2 1 0 3
Helped to some extent 2 4 7 13
Helped a lot 1 3 5 9Totals 6 8 12 **n=26
Gamma value =0.500, approximate significance =0.057, Not statistically significant difference at p<0.01. * The meeting context was scored on a scale of 1-3, where 1 is low and 3 is a high. The scale refers to the degree to which the meetings (to which respondents attended) in different regions were conducive to open discussion and exchange of views between staff and farmers (see Table 18). ** There were 2 non respondents
C.I: I thought we had organised it that everyone should write a story and bring it in? M.N: Yes that’s what happened, they did do this but they felt very pressured, and they were very
upset about being pressured. One meeting that Jessica wasn’t there, when Ian [regional champion] wasn’t there, Jessica was sick.
DJ: There was quite a lot of confusion at that meeting wasn’t there? MN: What it came down to was that a lot of people had been doing course work and this was… C.I: An extra thing on the top? M.N: It was… but when it really came down to it they were not getting out to the farmers to
actually pick up the stories. C.I: So it was the extension staff that got upset? M.N: Well it was, but they felt like they were under pressure, but when it got down to the bare
bones of it they weren’t having enough contact with farmers. C.I: The contact was there to be able to generate the stories. M.N: They were under pressure and not able to do their 1:1 farm visits and consequently the
stories weren’t coming through. And the rest of the committee weren’t pulling their weight and there were other people that perhaps had more farmer interaction than they did and they were just sitting back expecting them [extension officers] to come up with the stories. So we had that out, and then we agreed once again, and that’s when the farmers decided to come into the process.
Text box 7 Extract of discussion between two extension staff from the South West Region, taken
from a focus group discussion between regional champions
219
In the particular year of the trial, the staff (and committee members) from the South West Region
were busy with organising a large conference in their region, and perhaps had more scheduled work
than extension officers in other regions. One informant explained that it was hard to collect stories,
as she was not having any contact with farmers during that particular period. So a combination of a
busy schedule, low contact hours with farmers, and pressure from ‘above’ to produce stories,
appears to have minimised the impact of the morale boosting effect of the MSC process in this
region. This case also illustrates the point that the story collection process should evolve naturally
and people should not be forced into participating against their will.
Conjectured outcome (2) – competitive aspects of the MSC process
A second conjectured outcome thought to affect staff morale is the competitive aspect of the story
selection process. Observation and qualitative data revealed that some staff became particularly
enthused when the stories, for which that they felt some ownership, were selected. In these cases,
the language was frequently of ‘winning’ or ‘losing’. This mechanism can have two possible
outcomes; those who ‘win’ feel increasingly enthused, and those who ‘lose’ become increasingly
demoralised, or negative toward the approach. This mechanism appeared to occur at two levels:
between the regions, and between individual staff within one region.
With regard to the regional level, it seemed that there was some competition between the different
regions to ‘win’ stories at the CEC meeting. Records show that ‘wins’ were approximately evenly
distributed across the three major regions. Nevertheless, voting for the most significant stories at the
state and purchaser level did appear to create competitive feelings between the three large regions.
The competitive aspect between different regions is illustrated by the comments presented in Text
box 8.
I have not taken the feed back session very seriously, as often it seemed to be a point scoring exercise against the other regions, to see how many stories ‘got up’ from our region (quote: extension staff).
Another important thing said by SP was that she was getting really pissed off with being compared to the other regions (quote: extension staff).
The feelings I have in the state exec meetings? Its them and us – its very difficult to take that hat off and see which is the best story. Because I want the stories from my region to win! Luckily I don’t think the comments that I make have as much impact as when farmers make comments about the stories (quote: extension staff).
Text box 8 Comments concerning the competitive aspect of the MSC process
220
At the individual level, some respondents also explained that they experienced competitive
pressure. In the Northern Irrigation Region, individual competition was particularly problematic.
Initially in this region stories were reviewed at monthly staff meetings. This was the only forum at
which staff effectively voted for stories written by themselves. This process was not seen as
successful and was eventually changed in line with the other regions. The prime reason for
changing this structure was that the voting process between staff had become excessively
competitive, with people voting stories submitted by their ‘friends’ and voting against stories
written by people from other factions of the group (see Section 7.3). Nevertheless, the competitive
aspect of the process did not appear to effect staff perception that the process had been beneficial in
terms of staff morale. The main issue that people were concerned about was that competitiveness
would lead to bias in the story selection process.
Relationship between the conjectured context and outcome
For this CMO, the conjectured outcome (increased motivation) did seem to be realised in the
conjectured context (where stories were reviewed in meetings that were conducive to open
discussion and exchange of views between staff and farmers). The CMO configuration did appear
to be a plausible explanation for the data.
However, there appeared to be several additional contextual factors important to bring about
increases in morale that were not anticipated. It seems important that participants enter the process
on a voluntary basis. The story selection process also involves a considerable amount of
competition, and this can be either motivating or de-motivating for those involved – this being
partly determined by whether the stories, for which the individual feels ownership, are selected by
review panels. It also appears that unhelpful competition can be reduced by ensuring that groups of
people are not casting judgement on stories authored by themselves.
221
d) Increased understanding of the project’s impact
________________________________________________________________________________
This conjectured CMO is represented in the flow chart above. The context-outcome relationship
was tested by examining informant’s perceptions of increased staff understanding of project impact
across the different regional contexts.
Conjectured outcome – increased understanding of project’s impact
My observations of the implementation of the story review sessions were that committee members
were ‘surprised’ at some of the stories – they were hearing stories that they did not already know.
The staff questionnaire also supported this conjectured outcome; with all the respondents stating
that the approach had helped staff understanding of project impact ‘to some extent’ or a ‘lot ‘(see
Table 24). This was the highest scoring of any of the variables considered, with no respondents
indicating the MSC had no effect on increasing their understanding of the project’s impact.
Nevertheless, there were statistically different responses across the three regions. Unlike
respondents from other regions, non of the respondent’s from the South West Region stated that the
MSC process had helped staff understanding ‘a lot’. This supports the conjectured context-outcome
configuration that learning from the stories is more likely to occur in small informal meetings where
there is opportunity for staff and committee members to share their views.
My observation of the process in all regions was that less time was spent debating the stories in the
South West than in the other regions. It could be said that the large meetings and formal atmosphere
of the South West meetings was less conducive to learning. Other plausible explanations for the
regional differences are concerned with the number of stories generated, and the degree of farmer
involvement in the story collection and selection process. Perhaps hearing stories directly from
farmer-clients is more likely to bring about new knowledge than hearing stories told by extension
staff – there was a lower ratio of farmers to other committee members at the South West regional
ContextStory review occurs in meetings that have a structure and atmosphere conducive to open discussion and exchange of views between staff and farmers.
Mechanism Regularly sharing and discussing the stories about outcomes that have occurred in farmers lives in relation to project activities.
Outcome Participants gain new knowledge about achievements, and impacts considered desirable.
222
committee meeting than in other regions (see Table 16).
Table 24 Staff perceptions of the extent to which the MSC process affected staff understanding of
project impact against meeting context
Meeting context *
Response 1
South West2
Northern Irrigation
3Gippsland and
North East
Totals
Hindered a lot 0 0 0 0
Hindered to some extent 0 0 0 0
No effect 0 0 0 0
Helped to some extent 7 4 3 14
Helped a lot 0 4 8 12
Totals 7 8 11 **n=26 Gamma value =0.813, approximate significance =0.0007, Statistically significant difference at p<0.001. * The meeting context was scored on a scale of 1-3, where 1 is low and 3 is a high. The scale refers to the degree to which the meetings (to which respondents attended) in different regions were conducive to open discussion and exchange of views between staff and farmers (see Table 18). ** There were 2 non-respondents
At the round table meeting several of the purchasers also expressed the view that the approach had
helped them to further understand the impact of the approach:
I found the stories really illuminating, I enjoyed reading them. I felt a number of them really nicely
demonstrated the technical competency, or skills learning outcomes of Target 10. And I thought there
was some other stories in there, which to me showed how participating in Target 10 had actually
changed people’s attitude to learning, and fact in some cases it had actually affected the farm business
and family situation! (Quote: Purchaser at the round table meeting).
Relationship between the conjectured context and outcome
All of the respondents perceived that the conjectured outcome of increased understanding of project
impact had been achieved to some extent, regardless of regional context. However, there did appear
to be some relationship between the strength of the perceived achievement of the outcome and the
conjectured context. Respondents who attended meetings that were more conducive to open
discussion and exchange of views between farmers and staff were more likely than other staff to
perceive that the MSC process had helped staff understanding ‘a lot’.
223
e) Increased knowledge of each others’ values
________________________________________________________________________________
Another conjectured outcome was that MSC process would bring about increased knowledge of
other stakeholder values through the regular feedback from the interpretation and selection of
stories. The idea was that this outcome was more likely to be realised in meeting contexts more
conducive to open discussion. To test this conjecture, staff were asked whether they felt that the
feedback from the story review sessions had (1) helped them to understand what outcomes were
valued by the CEC and (2) helped them to understand what outcomes were valued by the purchasers
of the project. These outcomes were examined in relation to the different regional contexts.
Conjectured outcome (1) – Increased knowledge of the CEC’s values
Table 25 shows that the majority (20/25) of informants stated that the approach had helped staff to
understand what outcomes the CEC valued. Yet five were of the opinion that the feedback had had
no effect.
There were no statistically significant differences between respondents from the different regions.
However, unlike respondents from other regions, none of the respondents from the Northern
Irrigation Region stated that the MSC process helped them ‘a lot’ to understand what outcomes the
Central Executive Committee (CEC) valued. The comments relating to this question indicated that
staff from this region felt that they did not get sufficient positive feedback or involvement with their
regional committee, or from the CEC. Staff in the Northern Irrigation Region attended significantly
less regional committee meetings than those from other regions (see Table 15). These staff had
much less first hand experience of sessions where the regional committee discussed and selected
stories.
ContextStory review occurs in meetings that have a structure and atmosphere conducive to open discussion and exchange of views between staff and farmers.
Mechanism Project stakeholders debate and reflect on project activities on a regular basis throughout the project.
OutcomeParticipants gain more knowledge of what the different stakeholders value.
224
Table 25 Extent to which the MSC process affected staff understanding of outcomes valued by
the CEC against meeting context
Meeting context *
Response 1
South West2
Northern Irrigation
3Gippsland and
North East
Totals
Hindered a lot 0 0 0 0
Hindered to some extent 0 0 0 0
No effect 1 3 1 5
Helped to some extent 3 5 3 11
Helped a lot 5 0 6 9
Totals 7 8 10 **n=25 Gamma value =0.25, approximate significance =0.35. Not statistically significant difference at p<0.01. * The meeting context was scored on a scale of 1-3, where 1 is low and 3 is a high. The scale refers to the degree to which the meetings (to which respondents attended) in different regions were conducive to open discussion and exchange of views between staff and farmers (see Table 18). ** There were 3 non-respondents
In addition, there was a tendency for the Northern Irrigation regional committee to be quite critical
of the stories that were submitted. On one occasion they sent the majority of stories back to the staff
to be re-written:
At the regional meeting they read the four stories through and for the first time they were introduced to
the feedback from the state executive meeting. And they took this very seriously and actually rejected
three of the four stories saying that they didn’t have enough detail and that they were too subjective!
So they had to be re-written, and these stories have already been re-written at the dairy extension
meeting. They seemed to be going for perfection and I was told that it was a fairly tense meeting
anyway. But it seems like they did discuss the stories anyway (quote: staff member from the Northern
Irrigation Region).
Some respondents from this region also explained that they felt the feedback process from both the
regional committee and the CEC had not been fully effective.
It appears that for most staff, the story process helped them gain a better understanding of the
outcomes valued by the Central Executive Committee. However, this outcome can only occur
where there is regular (and constructive) feedback from the story selection process at both the
regional and state levels.
225
Conjectured outcome – Increased knowledge of the purchasers’ values
Table 26 shows that the majority respondents (19/24) stated that the approach had helped them (‘to
some extent’ or ‘a lot’) to understand what outcomes the purchasers of the project valued.
Table 26 Extent to which staff felt that the MSC process had affected their understanding of the
outcomes valued by the purchasers against meeting context
Meeting context *
Response 1
South West2
Northern Irrigation
3Gippsland and
North East
Totals
Hindered a lot 0 0 0 0
Hindered to some extent 0 0 0 0
No effect 2 2 1 5
Helped to some extent 4 4 4 12
Helped a lot 1 1 5 7
Totals 7 7 10 **n=24 Gamma value =0.48, approximate significance =0.06, No statistically significant difference at p<0.01. * The meeting context was scored on a scale of 1-3, where 1 is low and 3 is a high. The scale refers to the degree to which the meetings (to which respondents attended) in different regions were conducive to open discussion and exchange of views between staff and farmers (see Table 18). ** There were 4 non-respondents
Nevertheless, five informants (17%) felt that the approach had had no effect on developing a more
fully shared vision. There was no regional difference in response, and in this case, all staff from all
regions received the same quantity and quality of feedback. They all received the booklet Target 10
evaluation stories which contained the purchasers comments, in addition to receiving verbal
feedback presented at a meeting which all project staff attended.
The comments associated with lower scores tended to be concerned with the nature or extent of
feedback that was given from the round table meeting of purchasers. For example, one respondent
explained that as the purchasers were quite divided in their views of the stories, the feedback had
not been particularly useful:
The process still left us a little unsure of what areas the funders were keen to see stories on. As the
feedback from them was more confused (quote: extension staff).
Another respondent suggested that more follow-up was needed after the round table meeting:
226
Project funders response was a ‘one-off’ event and needed to be reinforced by some action to have
any significant impact (quote: extensions staff).
During discussions, several participants explained how they felt that the feedback from the
purchasers should have been more frequent; as it only occurred at the end of the year, there was no
possibility of adapting the approach to meet their needs in the short term.
Relationship between the conjectured context and outcome
This conjectured outcome (increased staff understanding of other people’s values) did not appear to
be strongly related to the conjectured context (that this outcome would occur where meetings were
conducive to open discussion and open exchange between staff and farmers).
Yet staff members felt that the MSC process had contributed to some extent to an increased
understanding of the outcomes valued by both the state executive committee and the purchasers. In
both cases, it appears that this outcome is more likely to be achieved where the feedback from the
story review process is circulated at regular intervals, where that feedback is constructive and where
it is clear in its message.
f) A more fully shared vision
________________________________________________________________________________
A further conjectured outcome was that the MSC process would enhance inter-organisational
reflection, communication and dialogue, and so result in a more fully shared vision. The
conjectured context-outcome relationship was tested by examining informant perceptions of
whether a more fully shared vision had been created in different meeting contexts. In this case the
conjectured mechanism was also examined. The idea was that the MSC process would lead to new
forms of inter-organisational communication and that by regularly challenging assumptions
(concerning the stories of change) this would ultimately lead to a more shared construction of what
is happening and what is desirable.
ContextStory review occurs in meetings that have a structure and atmosphere conducive to open discussion and exchange of views between staff and farmers.
Mechanism Project stakeholders debate and reflect on project activities on a regular basis throughout the project. Inter-organisational reflection, communication and dialogue are enhanced.
OutcomeOver time, the increased communication and dialogue will bring about a more shared construction of achievements and what is desirable.
227
Conjectured mechanism – enhanced inter-organisational dialogue
During implementation of the MSC process, I observed that there was a marked increase in inter-
organisational communication; for example, feedback documents from the CEC meeting were being
regularly circulated amongst the committee members and extension staff. Increased dialogue was
also apparent at meetings at which stories were selected. In some regions, participants valued this
dialogue highly (illustrated by the quotations in Text box 9).
The opportunity to meet and discuss the stories helped to promote an understanding of each other’s viewpoint, if not necessarily a shared vision. Often in our work there is not enough time taken to reflect on what we have done and to learn from what we have done and I think the story approach is really good in that encourages us to do this (quote: extension staff).
The way people have been encouraged to talk and communicate during this process has been very beneficial for the project (quote: extension staff).
But I certainly think it is a good team building thing between extension officers and farmers or between the state exec and extension officers. But to have all the different heads all sitting around a story, deciding whether a story is good or not is very useful. But also it shows the different perceptions between all the different individuals and that is interesting and illuminating (quote: extension staff).
Text box 9 Comments associated with the benefits of discussing the stories
For some participants however, the most valuable aspect of the MSC process was the way that it
opened the door for dialogue with the purchasers (see Text box 10).
The approach was also really good in building up relationship with the purchasers - helped us to understand more about how they think – and that they do think differently and we need to report to them in different ways. That’s been a real eye opener. The stories really brought this point to a head (quote: project staff).
I think the stories process has opened up so many doors for us. It is the best thing we could have possibly done – just for the doors – the doors to the purchasers – and been a lead in for the rest of the evaluation. People have had a good experience with it and are more likely to be keen to be involved in evaluation in the future. Plus all the positive promotion it has done (quote: project staff).
Text box 10 Comments concerning the value of the MSC process for creating dialogue with
purchasers
228
Conjectured outcome – a more fully shared vision
To test the conjectured outcome-context relationship, staff were asked whether they felt that the
MSC process had helped create a more fully shared vision between all the project stakeholders.
Most respondents (20/25) stated that the MSC process had helped (to ‘some extent’ or ‘a lot’) to
bring about a more shared vision between all the project stakeholders (Table 27).
Table 27 Staff perception of the extent to which the MSC process affected the creation of a more
shared vision against meeting context
Meeting context *
Response 1
South West2
Northern Irrigation
3Gippsland and
North East
Totals
Hindered a lot 0 0 0 0
Hindered to some extent 0 0 0 0
No effect 2 2 1 5
Helped to some extent 3 4 6 13
Helped a lot 1 2 4 7
Totals 6 8 11 **n=25 Gamma value =0.37, approximate significance =0.17, No statistically significant difference at p<0.01. * The meeting context was scored on a scale of 1-3, where 1 is low and 3 is a high. The scale refers to the degree to which the meetings (to which respondents attended) in different regions were conducive to open discussion and exchange of views between staff and farmers (see Table 18). ** There were 3 non-respondents
There were no significant differences in responses between the different meeting contexts. From the
comments associated with this questionnaire item, it is apparent that respondents selecting the ‘no
effect’ option felt that while a shared vision has not yet been created, this was important and
possible in the long term. One senior member of staff explained how he saw the MSC process as a
dynamic vision capable of reacting to external change:
The story process is like a dynamic vision – the vision statement is static – this doesn’t help you to
respond to a new things – this mechanism is more ‘adaptive reactive’ – that’s what is different about
it. At the moment we are going through a period of price drop, deregulation and GST and we have all
of them coming within a year or so. And this is not taking into account seasonal things, or global
fluctuations in the market – so we do need a reactive mechanism to deal with this change together.
Even just giving people permission to come forward with negative stories – sometimes as an
229
organisation we are criticised for always looking at the positive outcomes – the good news. Perhaps
we should use the story approach to work more in the direction of learning (quote: project co-
ordinator).
Another informant stated that:
It is certainly important that people have a common understanding of what the project is about. It
[the MSC process] is like a living philosophy of what the project is about (quote: project co-
ordinator).
Relationship between the conjectured context and outcome
For this conjectured CMO, the data suggests that the mechanism is being fired in some contexts, but
that the time frame to achieve the outcome might be longer than the 12-month study. There
appeared to be no strong associations between the regional context and staff perceptions of
achievement of this outcome. Many participants felt that there were important increases in inter-
organisational communication and dialogue and that the MSC process had helped them to
understand the views of different stakeholders. Some respondents did feel that the approach
hadhelped to bring these views more in alignment and hence to create a more shared vision. Others
felt that this was likely to be a more of a long term outcome, but nevertheless an important outcome
to aim for.
230
g) Increased knowledge of how to run the process to meet the project stakeholders’ needs
________________________________________________________________________________
Another conjectured outcome was that the MSC process would encourage evolution and changes in
the MSC process itself – so that it could be adapted to better meet the project’s needs. The
mechanism behind this is that the regular feedback would provide new knowledge with regard to
the information requirements of different stakeholders. The iterative nature of the process means
that these requirements could then be built into the process with each successive round of feedback.
In this case the context-outcome relationship was not tested, but observational data were used to test
the extent to which the conjectured outcome was realised.
Conjectured outcome – evolution and change in the MSC process
Chapter 7 described ways in which the approach was modified to meet the information needs of the
project during the 12-month trial. These modifications included the provision of a system of
ensuring that data were collected in an ethical manner, developing a fifth domain to encourage
stories about ‘lessons learned’ and the nominating of ‘regional champions’ to co-ordinate the
process in each region. After the trial ended, extension staff and committees decided to lengthen the
time span between the review of stories at the Central Executive Committee meeting, to reduce the
time consumed in review of stories. All these changes provide evidence that the extension staff
have been constantly modifying the approach to meet their needs, and are likely to continue to do so
in the future.
I was initially surprised to realise that very few people I spoke to understood the aims of the process
in the way that I did, even though I had explained it them many times in the past. One of the classic
critiques of participatory action research is that often staff do not understand the process fully, or
that they construct the process differently. Nevertheless, multiple constructions of an evaluation
ContextThat the organisational structure can accommodate change. That stakeholders are willing to enter into dialogue with regard to information gained from the process.
Mechanism Feedback provides new knowledge with regards to the information requirements of different stakeholders. Communication, learning and dialogue are enhanced.
OutcomeEvolution and changes in the way the MSC process is run so that it can better meet the project’s needs.
231
process can also be seen as beneficial, especially if people construct different, yet beneficial
outcomes from the process. One of the project managers explained to me that the reason he valued
the MSC process most, was the very fact that it did offer different positive benefits to different
stakeholders:
In my mind it certainly has been a very successful process. Just the support of the Target 10 team
illustrates this – as we are all very busy – given half an excuse to offload something and we will. Yet
the commitment to continue is still there. I still don’t think I grasp the entirety of what the approach
can do. What I find interesting is that different people seem to get different things out of it. What I
find interesting and important is not necessarily interesting and important for other people – but the
important thing is that everyone that I have spoken to has got something out of it. Which goes back
to the street performance indicator of participation rates – normally there is only a very small number
of people interested in the numbers - but in this technique everyone gets something out of it. So there
is more commitment to it (quote: project staff).
The MSC process is complex and operates across a whole organisational hierarchy. One thing that
shone clearly through the data was that people see different aspects of the process, depending on
where they are located in the project hierarchy. This is not necessarily a negative point, if people are
getting something they need form the process at the hierarchical level from which they view the
process. Indeed, for any one evaluation approach there may exist several different, yet functional
constructions of how the approach works. The conjectured CMO configurations present just one
plausible construction of how the MSC process worked across the Target 10 Project. This highlights
the importance of sharing and modifying the model with several key informants.
However, the fact that there are multiple constructions of the MSC process reinforces the need to
have someone (or a small team) driving the whole process, and that these people should have an
understanding how the process works at a range of levels. During interviews, several people pointed
out the importance of having someone with good organisational and facilitation skills to drive the
whole process. They expressed concern about what would happen after I stopped fulfilling this role.
It appears that the MSC process did allow for modifications to the process itself, to better meet the
project’s needs. However, participants held different constructions of what this process was; this
appeared to be related to the hierarchical position of the participant in the project. I suggest that the
process therefore needs someone or a small team to drive it, and to understand the process at
multiple levels. For this conjectured CMO configuration the relationship between context and
outcome were not explored.
232
h) Beneficial operational and strategic changes in practice
________________________________________________________________________________
In the above conjectured context–outcome relationship, in conducive contexts beneficial operational
and strategic changes would occur. As in the other CMO configurations it was hypothesised that
this outcome would occur in meeting contexts conducive to open discussion and exchange of views
between staff and farmers. The postulated mechanism behind this outcome was that, at an
operational level, extension staff would try to reproduce the outcomes deemed successful in the
stories and avoid the events described in the stories of ‘lessons learned.’ At a more strategic level,
decision-makers would make changes to the project through what they have learned in terms of
evidence of impact, and stakeholder values of this impact.
Conjectured outcome – beneficial and strategic changes in practice
During interviews, key informants explained how they used the stories for various purposes in their
extension practice including to explain a point to farmers or other extension staff and to help plan
future activities. To quantify the extent to which stories were used, and to understand whether they
were more likely to be used in certain project contexts, the questionnaire included some questions
about use of stories. Table 28 shows that around 50% of the respondents stated that they had used
the stories for each of the named purposes.
Table 29 shows that the extent to which respondents used the stories was significantly related to the
regional context. Respondents from Gippsland and the North East were more likely to use the
stories for all three of the named uses than respondents from other regions. The comments
accompanying this question suggest that the lower use by respondents from the Northern Irrigation
Region reflects the fact that there are several new staff who have had little chance to use the stories.
There is no clear explanation of why more respondents from the South West reported using the
stories less than in other regions, but it may be associated with the generally less positive reaction of
ContextStory review occurs in meetings that have a structure and atmosphere conducive to open discussion and exchange of views between staff and farmers.
Mechanism Staff try to reproduce outcomes deemed successful and avoid traps described in the stories of ‘lessons learned.’ Decision-makers change attitudes in light of knowledge gained. They may also examine specific stories to help in planning.
Outcome Extension staff make operational changes. Those involved in planning make strategic changes to the project through what they have learned in terms of evidence of impact, and stakeholder values of this impact. Purchasers make more informed funding decisions.
233
participants from this region. This may be associated with the low participation of farmers in the
South West, and large formal meetings that appear to be less conducive to discussion and debate.
Table 28 Extent to which staff used the stories
Category % of responses (n=29)
Used stories to help explain a point to farmers
54%
Used stories to help explain a point to other staff members
50%
Used stories to help plan a new activity 54%
Table 29 Extent to which staff used the stories against meeting context
Meeting context *
Response*** 1
South West2
Northern Irrigation
3Gippsland and
North East
Totals
Have not used the stories 2 5 1 8
Used the stories for one of the named uses
2 1 0 3
Used the stories for two of the named uses
3 1 4 8
Used the stories for all three of the named uses
0 1 5 6
Totals 7 8 10 **n=25 Gamma value =0.51, approximate significance =0.003. Statistically significant difference at p<0.01. * The meeting context was scored on a scale of 1-3, where 1 is low and 3 is a high. The scale refers to the degree to which the meetings (to which respondents attended) in different regions were conducive to open discussion and exchange of views between staff and farmers (see Table 18). ** There were 3 non-respondents *** The three ‘named’ uses for stories were (1) to help explain a point to farmers, (2) to help explain a point to other staff members (3) to help plan a new activity.
Stories as a tool to help extension officers to explain things to farmers
An outcome of the MSC process was that several extension officers (46%) recalled stories as part of
their normal extension practice: e.g., to explain certain points to a farmer. This is illustrated by the
following quote:
I frequently relate stories to help explain a point to farmers. Yes, I have used some of the stories that
have come from the approach – especially for a group of farmers who are wondering what they are
going to get out of the process – like they can learn to do the calculations themselves, and talk to
people sales reps more confidently - and so on (quote: extension staff).
234
One informant explained how extension officers utilise this approach naturally, by recalling to
farmers, stories derived from their own experience, but were now co-opting the stories from the
MSC process into this technique. In the long term, augmenting the folk wisdom of extension
officers may lead to a more unified understanding of possible project outcomes. This use of stories
may be particularly beneficial to extension officers with little field experience. However, the use of
stories to explain project outcomes and practices is not necessarily related to operational changes in
extension practice.
Stories as a communication tool with other staff and stakeholders
Several respondents (46%) explained how they used certain stories to communicate with other staff,
or with people from outside the project, especially in relation to the nature of their work:
Actually I had to give a seminar to the research staff, and I read one of the stories out. It illustrated
how complex extension is – that their research is just one part of the complexity of practice change.
The story helped them to understand the whole process of extension. They seemed quite surprised at
how complex it all was (quote: extension staff).
Stories as a tool to change extension practice
There were few examples of extension staff significantly changing their extension practice though
the learning gained from the stories. However, one respondent was adamant that this outcome was
occurring and relayed an example from his own practice:
I have an example! The story ‘turned a loss into a profit’, was seen very positively. This was about
a real ‘aha’ moment when a farmer came along and picked up the concept that the extension officer
had been explaining and then proceeded to explain it to all the other farmers. The farmer in the story
had taken the whole group with her. I followed the example in the story, encouraging the whole
group to work through the issues as a group, and they were beginning to identify the same issues as
the farmers in the story! (Quote: extension officer).
Stories as a tool for guiding planning
Fifty-three percent of staff reported that they had used the stories as part of a planning process.
Respondents explained that they had examined the stories and the interpretations as a part of
planning process of new activities. It appears that stories of lessons learned were seen as
particularly useful in this regard.
235
During interviews and discussions, it emerged that the stories were also being used in a number of
other ways, including providing information to include in the evaluation report.
Stories used to provide information for the evaluation report
Towards the end of the 12-month trial, extension staff were becoming increasingly involved in
writing a final evaluation report for the end of the first phase (five years) of funding for their
extension project. This report was to be sent to the project funders, and was a large effort, involving
many staff. This document was expected to be important in terms of influencing purchasers’
decisions on allocation of funds for a second phase of the project. The quotations in Text box 11
illustrate how the stories were especially valued when it came to writing the evaluation report.
Many of the stories or parts of the stories were included in the evaluation report, and placed next to
quantitative data that had been collected on the achievement of valued outcomes of the individual
programs. The stories were seen to add another dimension to the information.
I had to write the evaluation report for the DBF program – and a lot of evidence - especially at the higher levels of Bennett’s Hierarchy came from the stories. There would have been a real gap otherwise. So with the stories it is an easy way of documenting that higher order stuff. When I was writing the impact section of the evaluation report, it was great to have the stories. It would have been drier and less personal without the stories (quote: extension staff).
Stories are a valuable evaluation tool and help complete the picture. We can have used the stories with funding bodies, farmers and steering committees to highlight the outcomes and benefits of the T10 programs (quote: project co-ordinator).
I think it has helped us capture a whole lot of things that we would never have captured because it has been an on-going thing. So the value has been that it has put another dimension on to the evaluation. But I know there are other things that we have to do, like collecting names and stuff. But this collects the personal experience which adds another dimension – that I thought was really good (quote: extension staff).
Text box 11 Comments associated with the value of stories for evaluation reporting
236
Stories as a tool for guiding strategic changes and organisational restructure
There is no evidence that the stories strongly influenced the project redevelopment in terms of
large-scale strategic changes, which occurred shortly after the 12-month trial ended. Some
respondents felt that a system of responding to the lessons gained from the stories was lacking:
I don’t think that we have set up a system to respond to the lessons learned from the stories – so this
has not happened…The limitations are that it can take a fair amount of the time during meetings, we
perhaps did not as a group sit down and say, well here is all this great information we are generating,
how are we going to use it? (Quote: committee member).
Informants also inferred that the way in which the stories affect operational and strategic change is
less than direct, as strategic changes are obviously influenced by other factors in addition to
information gained from the MSC process. Indeed there is no evidence of any major strategic
changes being made that were a direct result of something learned from a story (see Text box 12).
Yet, there was a feeling that the process had influenced strategic planning at the project level.
There was no evidence to suggest whether or not the purchasers had used any of the information
gained from MSC process to make decisions about funding allocations. All that can be said is that
they were supportive of the process, and keen to be involved in the process in the coming years.
I would find it difficult to think of an individual story that has radically altered anything in Target 10 – possibly stories like ‘outdoing the consultant’ might trigger actions in people or encourage people. But I think that the stories themselves carry messages that need to be provided with a supportive environment – you can’t just say to someone ‘if you do this you will get this result’. You have to provide a whole environment – the stories are one of the things that help to encourage them. The stories provide peer group support for other farmers to try new things (quote: project co-ordinator).
It’s like in extension – when you make a decision, lots of things come together to influence you. The stories play a role in confirming what we already believed and this supports us in making the decision, along with other things. The stories play a role in helping you make the decisions. In some ways they also challenge things that you thought you believed. But I don’t see too many stories that really blew me out of the water. They either confirm what we already know and others challenge us a bit (quote: project co-ordinator).
Text box 12 Comments referring to how stories have affected change at a strategic level
Patterns between the conjectured context and outcome
There appeared to be some relationship between the different regional contexts and the achievement
of the conjectured outcome. The region in which staff and stakeholders were most actively involved
in the MSC process was also the region from which respondents most frequently reported using the
237
stories in planning and extension.
There was some evidence that the stories were influencing operational and strategic changes in
practice. Nevertheless, some participants expressed the view that the outcome of operational and
strategic changes in practice had not been fully achieved. The stories were used in a number of
ways that were not anticipated, especially the use of stories in the practice of extension. One further
use of the stories and feedback was to provide evidence for inclusion in the final evaluation report
submitted to the purchasers.
i) Organisation develops a sustainable capacity to learn and change
________________________________________________________________________________
A final conjectured outcome of the MSC process is that by having a process in place to examine and
challenge the value of what is occurring at the field level, the project team would gain an improved
and sustainable capacity to react to the changing external environment. This external environment
represents farmers’ as well purchasers’ requirements. It was postulated that this outcome would be
more likely to occur in contexts where story review occurs in meetings that are conducive to open
discussion and exchange of views between staff and farmers.
In the model building process, we envisaged that this conjectured outcome would take considerable
time to fully realise, as it requires long-term changes in organisational culture. It is perhaps the
ultimate goal of the MSC process. So the conjecture was not tested – as there was no expectation
that the outcome would have been realised in the period of 12 months. Yet the other outcomes
mentioned in the evaluation theory model could, in effect, be ‘stepping-stones’ to the achievement
of this final outcome. There is no evidence to date that this final outcome has been achieved but
several of the key informants believed that the MSC process does have the potential to realise this
outcome in the long term. Other outcomes that we hypothesised were stepping-stones to this final
outcome have been achieved. This is a positive indication that institutional learning and
ContextStory review occurs in meetings that have a structure and atmosphere conducive to open discussion and exchange of views between staff and farmers.
Mechanism Project members learn how to better detect, reflect, and collectively respond to outcomes at the field level and to respond to changes in stakeholder concerns and wishes.
Outcome By having a process in place to examine and challenge the value of what is occurring in the field, the project team may gain a sustainable capacity to react to the changing external environment.
238
sustainability could be realised if the project were to continue with the MSC process.
8.5 Summary of Testing Model
The conjectured CMO configurations illustrate the theory that the MSC process brings about project
improvements through involvement and illumination of program stakeholders, dialogue, beneficial
changes in extension practices, and organisational learning. The data suggest that most of these
outcomes are occurring for specific groups of people. This indicates that the MSC model is a
plausible, ‘implementable’ model that can lead to most of the postulated outcomes in at least some
contexts.
However, there appear to be certain contexts more conducive to the achievement of these outcomes.
Positive outcomes appears to be associated with:
• high participation of farmers in the story selection and collection process
• small informal meetings where selection occurs
• a skilled facilitator available to run the review sessions
• regional champions with good organisational skills, and perhaps ability to motivate people to
narrate/write the stories
• voluntary rather than forced participation of people in collecting the stories
• sufficient contact between extension workers and clients for the stories to be generated in the
first place
• a space for story review, where the emphasis is placed on participation rather than competition
• extension staff attendance at some of the review sessions at the regional or state level to
understand how farmers react to the process
• regular and constructive feedback from the regional committees to the extension staff with
regard to which stories were selected and why
• situations where the story review process is not conducted by the same group of people who
have written the stories, especially when there are social factions present in the group.
239
8.6 Impact of the MSC Process on the Project
Extent to which process met the evaluation needs of the project managers and co-ordinators
At the start of the 12-month trial, the project managers articulated that the evaluation needs of the
project were to:
• gain an overarching understanding of the impact, (including unexpected outcomes) of the
project on farmers’ lives
• employ an evaluation process that was capable of fostering dialogue and negotiation between
the different project stakeholders (as discussed in Section 5.2)
• employ an evaluation process that would combine well with the existing portfolio of evaluation
activities.
With regard to the first aim, evidence suggests that the MSC process helped many stakeholders and
extension staff to gain a better understanding of project impact. A wide range of outcomes, many
beyond those expected, were captured by the stories (see Chapter 7, Table 10). With regard to the
second aim, the data have demonstrated that the MSC process was able to adequately foster
dialogue and negotiation between many different stakeholders. The third aim also appears to have
been met, in that the stories were used to complement and add a ‘human touch’ to the quantitative
findings in the final evaluation report that was submitted to the purchasers.
While project staff and managers stated that the MSC process was of as high importance to their
evaluation needs as the other evaluation initiatives, it was always regarded as an addition to the
important ‘goal-based evaluation’ of individual programs. It was a telling sign that in the final
evaluation report (sent to the project purchasers), the findings of the cost benefit analysis were
placed at the front, followed by the evaluation of the extent of achievement of valued outcomes for
individual programs. While all of these sections included some isolated stories to illustrate points,
the findings of the MSC process were placed in the appendix of that report.
240
The MSC model offered other positive outcomes for the management that they had not anticipated
before the trial began:
It has given us more information than we had before – it is have given us a better evaluation
outcome. It has bolstered our evaluation capacity, improving our credibility, improves our position
in securing resources. And helped in training staff in terms of working out what evaluation is all
about. I think we are more aware of what we are trying to do – it has really contributed to that
process (quote: project co-ordinator).
At the onset of the trail, a focus group was held between the four regional-coordinators of the
Target 10 Project, to capture their expectations of the process. The expectations tended to be
concerned with the formative aspects of the MSC process, for example:
It think it will bring to an action stage a lot of things that, I believe, would have been there in the
past, but we never actually captured them to the same extent, or acted upon them to the same extent,
as we have potential do here…I think that there will be a lot of things that get fixed up as a result of
the process… (quote: project co-ordinator).
Other expectations included that the approach would have beneficial spin-offs for marketing and for
getting farmers involved in evaluation. From the evidence presented in this chapter, it seems that
the approach has met many of these expectations. However, it may not have brought about as many
tangible changes to extension practice as anticipated by the regional co-ordinators.
Extent to which process met the evaluation needs of the extension staff and committee members
Prior to the 12-month trial of the MSC model, the evaluation requirements of 150 project
stakeholders were surveyed using a Delphi approach (see Section 6.3). This included all project
staff, all committee members and a number of key stakeholders of the project. A consensus was
reached with regard to the domains of change that were to be used to guide the story selection
process. Thus, from the onset, the collection of evaluative information was guided by the needs
articulated by the project stakeholders.
Qualitative data from this study suggests that most stakeholders felt that the outputs of the approach
had met their information needs. However, there was suggestion that information about
productivity and profitability had not been adequately captured. At times the purchasers suggested
that the stories would be better if they contained more background information.
241
Where the context was suitable, staff perceived positive outcomes that they had not anticipated
from the MSC process, i.e., that:
• the documentation of stories and memory of the stories augmented the collective memory of the
project
• the process was particularly enjoyed by the farmer committee members, and some perceived
that this helped the farmer committees to have a voice in the evaluation and to contribute to the
project
• the sharing of stories had a positive impact on staff morale
• the MSC process helped staff and stakeholders to understand project impact
• the MSC process helped staff to understand what outcomes were valued by different
stakeholders
• the process of story selection encouraged positive dialogue concerning the ‘living philosophy’
of the project
• involving the purchasers in the process was seen as an important step in opening doors to
negotiation
• the stories were used in a variety of ways in extension practice and in planning
• participants felt that, in the long term, the MSC process could bring about a more shared vision
between all the project stakeholders.
Extent to which the MSC process met the evaluation needs of the purchasers
The purchasers were not asked to specify their expectations of the process before the trial. In
hindsight this would have been a beneficial addition to the research. Nevertheless, as mentioned in
Section 3.5 several reports have explicitly stated the evaluation needs of the purchasers from the
Agriculture Division of NRE. The extent to which these needs were met will be discussed in
Chapter 9, as well as an examination of the broader values of the MSC model.
However, at the round-table meeting of purchasers, and during the focus group that followed it, the
purchaser group appeared to find the approach highly valuable. In general, they suggested that the
approach was valuable in that:
• the stories demonstrate more than just technical outcomes
• the stories demonstrate many unexpected outcomes
242
• the review process of the stories demonstrated the organisational capacity of the project
• the stories captured the complex nature of farming system. However, one purchaser suggested that the stories needed more background information to fulfil his
requirement for evaluation information:
I think it’s pretty unclear to the uninitiated what the changes actually mean in tangible terms. It’s
hard to know what 1,000 kg of solids per ha means if you’re not a dairy farmer. A lot of people who
are interested in these stories are not dairy farmers. I think the story is good, but a bit is missing. A
lot of the stories don’t go far enough, they don’t take you right through to what it really means
(quote: purchaser).
The purchasers suggested that the quantifiable figures in these stories should be converted, where
possible, into dollar values. This would help people from other industries to understand the
significance of the changes described. They also suggested that specific types of stories could be
targeted towards different evaluation audiences. For example, those stories regarding profitability
should spell out the tangible outcomes in dollar figures and be accompanied by background data,
such as regional production benchmarks.
Nevertheless, the overall sentiment was that the MSC process was valuable, and all eight
participants (the purchasers) elected to participate in a similar process in the following year. The
participants seemed particularly pleased with the holistic nature of the information that the stories
presented: What struck me, was just how valuable it is, to be actually able to capture the much wider
ramifications of what we are doing rather than the very narrow one that we tend to walk down
(quote: purchaser).
Unexpected negative outcomes of the process
There were instances of non-constructive, competitive pressure between staff with regard to having
stories selected, over which that they felt some ownership. In some instances (and regions) staff
also felt that the MSC process was just another thing to do on top of an already busy schedule. Of
all the limitations and issues mentioned in the interviews and discussions, the ‘time factor’ was
mentioned most frequently. It could be that the amount of time consumed by the approach had
negative effects upon other work.
243
8.7 Conclusions
The outcomes stipulated in the conjectured CMO configurations (constructed on the basis of the
collective experience of eight key informants and myself) did appear to be realised in at least some
of the project contexts. The model appears to represent a valid construction of how the MSC
process can bring about organisational learning and provide evaluative information. Using Pawson
and Tilley’s (1997) Realistic Evaluation model to guide the meta-evaluation, the concept of
mechanisms being ‘fired’ in certain contexts to bring about outcomes was found to be valuable. It
helped draw out how outcomes were achieved, and the particular contexts in which this occurred.
The intention is that information gained in this case study will provide some general guidelines for
the sort of extension projects that might and might not gain value from the MSC process. Chapter 9
will examine the extent to which the MSC model represents a good, valid approach to evaluation,
and the extent to which it could usefully add to the ‘basket of choices’ for evaluation of extension in
Australia.
244
CHAPTER 9
THE MOST SIGNIFICANT CHANGE MODEL FOR
EVALUATION IN AUSTRALIAN AGRICULTURAL
EXTENSION
In this chapter, the second part of the meta-evaluation of the MSC model is presented. Here,
the focus is on the merit of the approach in general, and the extent to which the MSC model
is appropriate to agricultural extension projects in Australia. To do this, the MSC model was
examined using the key meta-evaluation questions that were developed in Chapter 2. I
concluded that the MSC process could contribute usefully towards meeting evaluation needs
at the national level. In terms of the MSC process contributing to project improvements, the
capacity of the MSC model will depend on the individual context of the project in which it is
applied. I suggest that this approach is particularly appropriate for large extension projects
with diversity in outcomes, with a high degree of farmer participation in project planning and
decision-making. While the model satisfies some of the premises for good evaluation, it also
has inherent biases and weaknesses. This emphasised the need to combine the MSC model
with complementary evaluation approaches, to help meet all evaluation needs and also to
offset the bias inherent in the approach.
9.1 Introduction
In Chapter 8, the first part of the findings of a meta-evaluation of the Most Significant Change
(MSC) model were presented. The meta-evaluation focused on the extent to which the MSC model
was able to meet the needs of the case project stakeholders and to contribute to project
improvements. As the aim of the thesis is also to develop a model of evaluation for agricultural
extension projects more broadly, it is important to evaluate the MSC model at a more macro level
245
too. The aim of this chapter, therefore, is to determine the extent to which the MSC model meets the
premises of ‘good’ evaluation. This includes the extent to which the model is appropriate for the
project context – in this chapter that context is extended to extension projects across Australia.
Lessons learned from the case study evaluation are examined to provide clues as to the applicability
of MSC model to extension projects more broadly. The Target 10 Project cannot be considered
representative of all large extension projects in Australia, and it is unlikely that other projects would
have the same outcomes and reactions to the MSC process, if implemented. Nevertheless, the
understanding gained in terms of the CMO configurations (presented in Chapter 8), will provide
some indication about the sort of project contexts in which the MSC model is likely to bring about
desirable outcomes.
The chapter begins by recalling the meta-evaluation questions that were developed in Chapter 2,
and applying these to the MSC model. Following this, I discuss the extent to which the MSC
process is able to offer a useful addition to the ‘basket of choices’ for extension evaluation in
Australia. Recommendations are made about the sort of extension projects that might gain more
fully from the MSC model to evaluation, and what evaluation approaches are likely to be
complementary.
9.2 Applying the Meta-Evaluation Questions
In Chapter 2, eight meta-evaluation questions were developed, in order to help determine the
‘goodness’ of an evaluation approach. The questions were:
• Is the evaluation relevant to the context and purpose in hand, and does it fulfil the evaluation
needs of the stakeholders?
• To what extent can the evaluation contribute to the project being better able to meet needs?
• Is the evaluation guided by program theory?
• Does the evaluation make an attempt to account for different program outcomes?
• Are the evaluation processes socially and politically just?
• Is there an appropriate ratio of costs to benefits for the project from the process (and outcomes)
of the evaluation?
• To what extent are the design and data collection/ analysis of the evaluation valid?
• Does the evaluation attempt to judge the merit and worth of projects?
246
In the following sections (9.2 to 9.9) the MSC model is considered in terms of its ability to meet the
needs of evaluation in Australian extension programs, in terms of each of the above questions.
Lessons from the case evaluation (with the Target 10 Project) are also drawn out to provide
indications of the applicability of the MSC model to extension projects more broadly.
9.3 Is the Evaluation Relevant to the Context and
Purpose in Hand?
In Section 8.7, I claimed that the MSC model was able to go a considerable way in meeting the
articulated needs of the Target 10 Project management, the staff and the purchasers of the project.
This was ultimately verified by their commitment to continue with the MSC process after the trial
was complete. Nevertheless, while project staff and managers stated that the MSC process met their
expectations, the model was always seen as an addition to the important ‘goal-based evaluation’ of
individual programs.
Extent to which the MSC model can address the challenges posed by changes in extension
In Chapter 3, recent changes in the philosophy that underpins agricultural extension and the
organisations that deliver extension were examined. I argued that these changes have influenced
the way publicly-funded extension projects are planned, managed and valued. I concluded that the
changes represent unresolved dilemmas for evaluating the new genre of projects, therefore there is a
need for evaluation that can:
• provide evidence on the extent to which predetermined objectives have been achieved
• provide accountability for group-based models of extension that produce diverse, context-
specific outcomes
• provide a process whereby farmers can participate in evaluation
• be conducted as part of an internal evaluation capacity and hence foster organisational learning
• provide performance information for communication between multiple stakeholders
• involve processes that foster dialogue between multiple stakeholders.
247
Provide evidence of the extent to which predetermined objectives have been
achieved
Recent changes in public sector management have seen an increased demand for the summative
evaluation of projects, to provide accountability to both the ‘purchasers’ and the farmer-clients. In
the case evaluation, the MSC process was seen to help stakeholders understand impact, yet the
emphasis was squarely on ‘understanding’ rather than ‘proving’. The MSC process was not
designed to provide the sort of data that is generally accepted as ‘highly objective’ (e.g., for
summative evaluation purposes). Nor did it provide quantitative evidence with regard to the extent
of achievement of predetermined objectives. In the case evaluation this need was fulfilled through
other evaluation techniques.
It could be argued that in using the MSC model for accountability it would be all too easy to hide
negative outcomes. The MSC model privileges the voice of those inside (or close to) the project –
those most likely to be advocates of the project. It does not seek the views of antagonists of project
practices. In addition, the MSC process does not give an indication of the spread of adoption of
technologies across the dairy farming population. Instead, the MSC model is designed to capture
‘remarkable events’ rather than the average experience of farmers. The MSC model is an example
of formative evaluation, aimed at encouraging learning between project stakeholders and
‘improving’ the project, rather than ‘proving’ what the project has achieved. Consequently, if used
in isolation, the MSC is unlikely to satisfy stakeholders’ demands for summative evaluation
concerning the achievement of predetermined objectives.
Provide accountability for group-based models of extension that produce
diverse, context-specific outcomes
In instances where projects have few or no pre-determined outcomes (but instead are more process-
orientated) and produce diverse and context-specific outcomes, the MSC model may be able to play
an important stepping-stone to summative evaluation. Davies (1996) suggests that the MSC process
may be best viewed as a ‘search mechanism’ for unexpected or unanticipated outcomes. Where
outcomes are not predetermined the function of ‘searching’ for outcomes could be very important.
However, to provide summative information, the MSC model would still need to be coupled with
some more empirical research and analysis. In the case evaluation, the stories were analysed and
grouped into themes with the help of Bennett’s Hierarchy. A further step could have been to
conduct a survey across a representative sample of participating farmers to quantify the spread of
248
these ‘emergent’, valued outcomes. The idea is that the MSC model would be used to search for
unanticipated, valued outcomes, then coupled with a system of measuring these ‘dynamic
indicators’ (replacing the need to measure predetermined, ‘static’ indicators over the entire life of
the project – as is practised in conventional monitoring). Davies (1996) suggested that the MSC
model could accommodate a stage of quantification in this manner, but this was not implemented in
either the Bangladesh case, or the Target 10 case. In the Target 10 case, this was not seen as
necessary, as there were other evaluation process being conducted to fulfil this requirement.
Provide a process whereby farmers can participate in evaluation
The MSC model could be particularly relevant for evaluating the new genre of participatory, group-
based projects. It can accommodate a wide range of participants in the evaluation process itself, and
there is no reason why this could not accommodate more farmers than were involved in the case
evaluation. As the structure of the story review process ‘rides on the back’ of a pre-existing project
structure, if farmer-clients are more involved in that structure, they would also be more involved in
the story review process. To promote the participation of farmers in the process, the first stage of
story collection and analysis could occur during farmer meetings. In Section 3.3, I advocated that,
in highly participatory projects, the process of evaluation itself should reinforce, rather than inhibit,
the effort towards participation and empowerment of participants. I argued that highly participatory
projects require participatory evaluation; approaches that allow the stakeholders and beneficiaries to
have a say in which changes are important, and which changes should be measured. The MSC
process has potential to meet these requirements. In the case evaluation the MSC process was
effective in helping staff and farmers to understand critical incidents surrounding success and
failure.
Evaluation that can be conducted as part of an internal evaluation capacity
and hence foster organisational learning
Evidence from the case evaluation suggests that the MSC model is ideally suited to provide internal
evaluation to foster organisational learning. Firstly, the MSC model is internally driven at the
project level, which is in line with the trend to promote internal evaluation with facilitation by
evaluation experts. Secondly, the MSC model seeks to enhance the organisational learning capacity
of the project, which is one of the demands being made by the ‘purchasers’ in the Victorian
Agriculture Division. In Chapter 8, evidence was presented to suggest that the MSC process
contributed to the achievement of several outcomes in the case project that could support
249
organisational learning, i.e., the outcomes were:
• enhanced organisation memory – an aspect of organisation learning that is seen to be especially
valid under times of rapid change
• encouraged staff to reflect on the impact of their extension practice
• helped staff and stakeholders to understand project impact
• helped staff to understand what outcomes that were valued by different stakeholders
• encouraged positive dialogue concerning the ‘living philosophy’ of the project
• resulted in stories being used in a variety of ways in extension practice and in planning
• had potential in the long term to bring about a more shared vision between all the project
stakeholders.
Provide performance information for communication between multiple
stakeholders
The stories and interpretations together represent the performance information produced by the
MSC model. The stories contain a wealth of information, both factual and tacit. The Target 10
implementation of the MSC model revealed that the stories were perceived to be ‘powerful and
engaging’ for those who read them. The interpretations themselves represent another domain of
information, and reveal what stakeholders do and do not value in terms of impacts of project
activities. I suggest that the MSC model can produce highly appropriate performance information
for the purposes of communication of project impacts between multiple stakeholders.
Involve processes that foster dialogue between multiple stakeholders
Evidence from the case evaluation suggests that the MSC process is well designed to foster
dialogue between multiple stakeholders. The MSC model is particularly suited to deal with
numerous organisations working in partnership to bring about extension outcomes as was seen in
the case study where a whole range of stakeholders were drawn into dialogue with one another. The
on-going nature of the MSC model supports an iterative process that can accommodate emerging
information needs of a range of stakeholders.
Like many large extension projects, the organisational structure under which the Target 10 Project
operates is complex and several purchasers and co-providers interact in a dynamic project
environment (McDonald and Kefford, 1998). Taking this organisational complexity into account,
250
it is vitally important that time is allocated for ensuring that the various stakeholders enter into a
meaningful dialogue about happenings in the field, and whether these experiences represent
desirable outcomes. At the round table meeting (Section 7.3), it became apparent that different
purchasers interpreted the stories in differing ways and held different things to be of value. This
finding supports the concept that evaluation is conducted in a value-pluralistic context (Toulemonde
et al., 1998). Thus the MSC process encourages negotiation and dialogue between the various
evaluation stakeholders (including the purchaser and the provider) which addresses at least some of
the challenges posed by changes in public sector management.
Summary
In general, the MSC process appears to go some way in addressing the needs of greater
communication of evaluation findings, managing organisational complexity and value pluralism,
but it does not provide the sort of data that is being demanded under the call for greater
accountability to clients and purchasers.
9.4 To What Extent Can the MSC Evaluation Contribute to
Projects Better Able to Meet Needs?
In the case study evaluation the MSC process was seen to lead to several outcomes that had
potential to contribute to project improvements. Using Pawson and Tilley’s (1997) Realist
Evaluation model to guide the meta-evaluation, an attempt was made to determine the project
contexts in which conjectured ‘mechanisms’ (inherent in the MSC process) were triggered to bring
about beneficial outcomes. I found that there were marked differences between the four project
regions across which the MSC process was implemented, and these were examined to try to explain
the different outcomes produced. Even within one project environment, certain contexts were more
conducive than others to the realisation of useful outcomes.
Following this logic, it can be assumed that some projects, are likely to benefit more than others
from the MSC process in terms of project improvement. The study revealed that beneficial
outcomes in terms of project improvements from using the MSC model are more likely to be gained
in projects with the following characteristics:
• high participation of farmers in the project development and in the committee structure
251
• small informal meetings where the story selection processes can occur, which emphasise
participation rather than competition
• skilled facilitators available to run to review sessions and to motivate people to narrate/write the
stories
• having the story review process conducted by a different group of people to those who have
written the stories, especially when there are social factions present in the group
• voluntary, rather than forced participation of people in the collection of stories
• close contact between extension workers and farmers for stories to be generated (if extension
officers are to narrate stories).
In addition to the above contexts, I suggest that the project structure itself is important. The MSC
model was designed to evaluate large projects, with several hierarchical ‘layers’ and a number of
project stakeholders. In the case of the Target 10 Project, the pre-existing structure appeared to be
well suited to MSC model. Nonetheless, there is no evidence to suggest that smaller projects would
not benefit from collecting, sharing and evaluating stories of significant change. However,
implementation of the whole model might not be possible or desirable in all projects.
9.5 Is the Evaluation Model Guided by Program Theory?
In Chapter 2, I suggested that good evaluation should consider the theories that underpin the
project. Considering the arguments of Pawson and Tilley (1997) and Chen (1990), good evaluation
should be guided by program theory, and should not be simply a ‘black box’ venture. Even Scriven
(1994), who levelled early criticisms at a theory-driven evaluation, recently conceded that program
theory is usually helpful, even if not always essential.
In the case evaluation, the MSC process was not guided by questions developed from a carefully
constructed model of the theory that underpins the Target 10 Project. Yet it did incorporate
questions guided by stakeholder concerns within defined domains of change. The MSC model
involves collecting impact information in an almost ‘goal-free’ manner and then coming an
understanding about key areas of findings through a process of induction. In the case evaluation, the
domains of change for the story collection (approximate to key evaluation questions) were
developed with considerable consultation with the management and stakeholders using the Delphi
process (see Section 7.3). In a sense, a trade-off was made between participation and ownership of
the evaluation process at the expense of a strong theory-driven development of key evaluation
252
questions. Perhaps this is quite a common dilemma in the evaluation of participatory extension
programs.
Program theory that is largely deductive in orientation (as described by Pawson and Tilley, 1997
and Chen, 1990), may be best suited to expert-driven evaluation rather than participatory
evaluation. Leviton (1994) argues that evaluation theory is most seriously deficient in assignment of
value to community-based programs. She suggests that when ‘experts’ frame questions for the
community they encounter understandable fury. The Target 10 Project is not exactly ‘community
based’ but it was influenced by theories of participation and community ownership. To some extent
the same issues as Leviton describes in community-based programs come into play. The MSC
model endorses the position that the clients’ and key stakeholders’ values are the ones that should
be used to assign value. Thus, use of a deductive-normative theory for developing the key
evaluation questions would be rejected in favour of developing key questions through the
achievement of some sort of consensus from the project stakeholders with regard to what they
believe should be measured in the evaluation. As a result of the lack of emphasis on drawing the
key questions from an explicit program theory, there is little analysis of the underlying mechanisms
that are triggered by the project.
However, there are other ways of incorporating program theory into evaluation that are more
inductive and allow for the inclusion of different stakeholder values. Patton (1997:224) describes
three varieties of program theory:
• User focused or ‘theories of action’ – were facilitators work with beneficiaries of a program to
articulate and specify their implicit theory of action. This model is largely derived from the
work of Argyris and Schon (1996) who contend that people in programs operate of the basis of
theories of action of which they may not be aware. Argyris and Schön (1996) distinguish and
try to draw out the contrast between two kinds of theories that people hold: ‘espoused’ theories
and ‘theories-in-use’.
• Inductive approach – involves doing fieldwork on a program to generate grounded theory. The
product of this is an empirically derived theoretical model of the relationship between program
activities and outcomes, framed in terms of important contextual factors. In inductive
approaches, the evaluator builds the theory from observations and fieldwork rather than from
discussion with those involved.
• Deductive approach – involves drawing on scholarly theories from academic texts and current
253
knowledge and testing them against empirical evidence. For the large part Pawson and Tilley
(1997) advocate a deductive approach; while they consider ‘folk-knowledge’ in their
conjectured program theories, these are not grounded in field work.
In the case evaluation, the MSC process did not incorporate any of these approaches to program
theory, but it could be modified to incorporate Patton’s ‘inductive’ or ‘user-focused approach’. I
suggest that there are two ways in which the MSC model could be usefully modified to incorporate
some program theory.
Firstly, the stories could be analysed en masse by a researcher to determine critical success and
failure incidents (as perceived by project stakeholders) and from this draw out a conjectured (yet
partially grounded) program theory model. This theory could then be tested against empirical data
to develop further knowledge of how the project works, and in what situations, and why. This
proposal, is similar to Patton’s inductive approach to program theory.
Secondly, the stories could be analysed en masse by a group of project stakeholders – and this
analysis form the basis of a conjectured (yet partially grounded) theory model of how the program
works. This process may need to be facilitated by someone versed in constructing program theory
models, but conducted with participation of stakeholders. The model itself could then be tested, as
is conducted in Realistic Evaluation by the same group. This approach lies somewhere between
Pattons’ ‘inductive-approach’ and his ‘user-focused approach’: it is inductive, as the models would
be grounded by data from the stories, yet unlike Patton’s inductive approach, it would involve
considerable participation of stakeholders. In contrast to Patton’s ‘user-focused’ approach, the aim
would not be to compare what extension staff say they do (espoused theory) with what they actually
practice (theory-in-use). Instead the intention would be for a group of project stakeholders to work
with the observations and constructions of program impacts (in the form of the stories and
interpretations) to develop a more sophisticated (and shared) construction of how the project works,
in what contexts, and why.
254
9.6 Does the Evaluation Model Make an Attempt to
Account for Different Program Outcomes?
The MSC model emphasises collecting as wide a range of outcomes as possible (in the forms of
stories), and in this sense is almost ‘goal-free’ in approach. Nevertheless, the use of ‘domains of
change’ does ensure that certain types of information are included in the evaluation. In the Target
10 case the domains of change were:
1. Changes in farmer decision-making skills
2. Changes in on-farm practice
3. Changes in profitability or productivity
4. Stories about ‘lessons learned’
5. Any other changes
These domains are left deliberately ‘loose’ so that those changes that are perceived to be important
by the storytellers can be captured through the process.
While the process may capture a range of unexpected outcomes perceived to be important by the
project stakeholders, it does not capture unexpected or unintended outcomes unless they are deemed
‘significant’ by those involved in the process (staff and committee members and purchasers). For
example, in Section 7.1, I mentioned that there has been some criticism of the Target 10 Project for
being focused too strongly on production, at the expense of social and environmental impact.
While there were multiple stories concerning negative and positive social impact of the program,
there was only one story concerning environmental impact of project activities. If stories had been
collected from known antagonists of project practices, then these sorts of views might have been
drawn into the process. It seems that, in the Target 10 case, the MSC process only picked up
unanticipated or unintended outcomes that were located within the discourse of those involved in
the project management (the staff and committee members).
This appears to be a serious omission to the evaluation process. However, I am of the opinion that
the Target 10 staff would have been willing to engage in discussion of stories from project
‘antagonists’ and that that this could easily be accommodated into the MSC process. Indeed, on
255
several occasions, staff mentioned the need for more stories about ‘lessons learned’ – because of the
high level of discussion and learning that stories about negative impact tended to generate.
9.7 Are the Evaluation Processes Socially and Politically
Just?
In terms of who was represented in the evaluation, it can be argued that the evaluation process was
not socially / politically just, in that it favoured the inclusion of some stakeholders over others.
Indeed, the story selection process is extremely biased in favour of those people who attend the
story review sessions. In the case of the Target 10 Project, the committees were usually attended by
an even spread of farmer members, project collaborators and staff. However, this was not the case
in one region, where farmers were greatly outweighed by the other members.
In addition, it should be pointed out that the farmers attending the project committees cannot be
considered fully representative of the wider farmer voice. At an early stage of the trial evaluation, a
wide range of people were asked about the extent to which the farmers on the committees were
considered ‘representative’ of the wider farmer voice (Dart, 1999b). There was general consensus
from informants (including farmers, industry representatives, staff from the University and staff
from the project) that the farmer members of the committees were not representative of the general
farming community (See Text box 13). However, this was seen as an unavoidable situation, and
informants felt that these farmers were able to articulate the wider farming community ‘felt needs’
to a reasonable extent. The wider farming community needs are also brought to the committee
forum by the industry representatives, who have a wider client base than the project staff (and
perhaps even some farmers).
256
They are as representative as can be expected. The other types of farmers wouldn’t attend the meetings. Hard-nosed, business-orientated, or less-articulate farmers would never go, but the ones that do go, do a reasonable job of representing the farming community. It is not perfect, but that is the best that you can do (quote: Industry representative).
They’re not representative. I don’t think they are at all. They’re the ones with the organisational skills, the interest and the curiosity and the time. Aren’t they? And I think, in a lot of ways, NRE falls into the trap of going back to the same people time and time again. They’re getting a select view. But then how do you get “Farmer Jake”, who doesn’t move from his farm, and, you know, knows nothing, goes out and plays sport? He would hate to sit in a room for a day. You don’t get them, do you? (Quote: Farmer representative).
As far as I am aware, there is no formal mechanism for the farmers [committee members] to gather the opinions of the farming community. As far I am aware, this is an informal gathering. I mean, they are out there farming with other farmers, and I presume that they get an idea about what other farmers want just from casual conversation and contacts. But I think most of the people on the committee are aware of what’s happening, and they are involved with discussion groups and those sorts of things, so they do provide feedback in terms of what those other farmers are thinking (quote: extension staff).
Text box 13 Informant views of the extent to which the farmer committee members ‘represent’
the wider dairy farming community
The MSC process is clearly embedded within the discourse of the project staff and committee
members. The farmer-clients who were involved in the selection process cannot be considered to
fully represent the farmer voice on matters of project practices and outcomes. Farmers on the
committees are generally advocates of the project, and thus there is a lack of the ‘critical voice’ in
the story review process. The MSC model makes no attempt to capture the opinion of those farmers
who choose not to participate in the Target 10 Project. So antagonists of the project may not have
been given a say in this evaluation. In some other evaluation approaches, processes are employed to
ensure that ‘antagonists’ are represented in the evaluation. For example, in Fourth Generation
Evaluation it is proposed that the ‘victims’ of a program are identified and their view sought (Guba
and Lincoln 1989). Inclusion of a similar process could add to the validity of the MSC model.
If the MSC model was combined with an approach to evaluation that did seek out non-participants
of the Target 10 Project and antagonists of the project, this could offset the bias and provide a more
comprehensive evaluation outcome. Another possibility is that a researcher could seek out stories
from antagonists and include them in the review forum, with stories containing elements of negative
outcomes (referred to as stories about ‘lessons learned’).
257
9.8 Is there an appropriate ratio of Costs and Benefits for
the Project from the Process (and Outcomes) of the
Evaluation?
Respondents reported that the biggest cost associated with the MSC process was the amount of time
that it consumed. While they felt that it was valuable to continue the process, they suggested that
the story review process should be less frequent. Since the trial, this change has been made; the
stories are being selected at every other committee meeting (every four months instead of two
months).
In an attempt to address this question, informants were asked whether they thought that the benefits
of the approach outweighed the costs in terms of time and resources, ie ‘was it worth doing at all?’
All informants felt that it was a worthwhile venture. They tended to see the main benefits as being
in the form of staff motivation, reflective practice in general and a new and very important
negotiation process with the purchasers. However, many respondents did not conceptualise the
process as a form of evaluation at all. Evidence of support for the approach is provided in the fact
that staff and committees voted to continue the approach, albeit in a modified form, after the 12-
month trial was complete.
It is also worth considering alternative approaches to the MSC model, and what the cost would have
been to the project if it had not been implemented. The question of whether another approach would
have produced the same ends more efficiently remains largely unanswered. Other qualitative
techniques, such as focus groups and in-depth interviews, while good at eliciting a wide range of
unexpected outcomes, are not continuous and would not capture the iterative feedback and learning
that is seen with the MSC model.
9.9 To what Extent are the Design and Data Collection/
Analysis of the Evaluation Valid?
At several occasions in the implementation of the MSC model across the Target 10 Project, people
expressed doubts about the ‘scientific validity’ of the output of the process. Some people were
258
especially concerned with the sampling technique and bias of the approach toward positive
outcomes.
Sampling technique
The sample technique of the MSC model is selective rather than inclusive, and will not provide
information on the ‘average condition’ of a farmer who attends the programs, but rather will
provide information about the exceptional circumstances, particularly successful circumstances.
This is referred to as purposive sampling. Some would argue that this is not an appropriate sample
technique to produce reliable information to enable judgements to be made about the performance
of the program.
Nevertheless, in qualitative research, purposive sampling (or purposeful sampling) is seen as a
legitimate form of data inquiry, forming a dominant part of the logic of qualitative research. Patton
states that:
The logic and power of purposeful sampling lies in selecting information-rich cases for study in depth.
Information-rich cases are those from which one can learn a great deal about issues of central
importance to the purpose of the research, thus the term purposeful sampling (Patton, 1990:169).
Patton goes on to describe several different strategies of purposive sampling, the logic of each
strategy serving a particular evaluation purpose. One category is ‘extreme or deviant case sampling’
– this approach focuses on cases that are rich in information because they are unusual or special in
some way. In the MSC model, the sampling system corresponds to this category where the logic is
to capture significant instances of success or failure. The purpose of this strategy is to learn from
these extreme stories, and ultimately to change extension practice to move towards more success
and away from failure. Therefore strategy is to select stories from which most can be learned.
If the purpose of the evaluation is to precisely document the natural variation among outcomes for
farmers who attend programs, then a random sample of sufficient size to be representative would be
needed to permit generalisations about the experience of all farmers attending programs. Whoever,
Patton (1990:170) suggests that ‘in many instances more can be learned from intensively studying
extreme or unusual cases than can be learned from statistical depictions of what the average case in
like’. Another option frequently practiced is to combine approaches, so that an understanding can be
gained of the normal distribution of participants, and the extreme cases. In the Target 10 Project, the
259
MSC process was combined with other approaches that did capture the normal distribution of
farmers attending programs.
Positive bias (and possible modifications)
In the case evaluation, the MSC model was biased in favour of success rather than ‘bad news’
stories. About 90% of the stories collected concerned positive outcomes. However, toward the end
of the trial, the staff and committees elected to include a further ‘domain’ named ‘lessons learned’,
to ensure that each region will present at least one ‘bad news’ story in each selection period. As was
previously described, the MSC model adopts a purposive sampling system, and captures instances
of success rather than the average farmer experience. However, in the case evaluation, several staff
pointed out that there seemed to be more potential for learning from the ‘bad news’ stories, and for
this reason, there was a move to encourage the collection of these stories.
The selection process is extremely biased towards the views of the committee members, and those
involved in the story selection process. However, unlike other research approaches, the bias itself is
offered up as another source of data about the organisational capacity of the project. The reasoning
behind the selection of certain stories was recorded and documented along with the stories
themselves. The inclusion of these interpretations as another form of evaluative data affords a high
level of transparency.
Reliability of factual content of stories
Early in the trial participants were concerned with the authenticity of the stories; ie., ‘how do we
know that they are not fictitious?’ All stories were accompanied by the names of those involved in
the event, and the location of the event. This meant that it would have been possible to follow up
the stories. Although such checking never occurred, there was considerable pressure by peers to
record information accurately; stories that seemed implausible, or incorrect in factual content, were
not selected.
Nevertheless, as was discussed in Chapter 5, the validity of the stories and interpretations is
somewhat dependent of the epistemological stance adopted. Under a constructivist epistemology the
stories represent reality in their own right, and it is in the telling of the stories that the storytellers
and reviewers construct reality. In addition to this, the interpretations that the tellers and the
reviewers provide give strong clues as to how these actors construct reality. The sharing of the
260
stories and of the interpretations made of these stories allows a new, shared and more sophisticated
reality, to be jointly constructed by the storytellers, reviewers and the story readers. In this sense
then, the process helps stakeholders apprehend reality (in its variously constructed forms) to make
decisions. Yet under an objectivist epistemology, the stories themselves require cross-checking and
may not represent the best way of accurately determining the events that have occurred.
9.10 Does the Evaluation Attempt to Judge the Merit and
Worth of Projects?
Scriven (1991) contends that evaluation should involve the determination of merit and worth of the
object being evaluated. In the case evaluation, the thing being evaluated was the Target 10 Project.
While stories of individual farmer outcomes were judged for their merit and worth by stakeholders,
there was no point at which the evaluators (stakeholders in this case) made an overall determination
of the merit and worth of the Target 10 Project. However, the stories and interpretations did
contribute to the evidence (in addition to the other evaluation activities) that was intended to help
the purchasers determine whether the project was of sufficient merit and worth to receive continued
funding and support. But this is not the same thing as the evaluators making a determination of
merit and worth of the project – as purchasers represent only one stakeholder group.
A similar criticism can be leveled at Guba and Lincoln’s (1991) ‘Fourth Generation Evaluation’
(FGE) model. Yet the MSC model goes one step further than the FGE model, which feeds interim
results back to the participants, but does not have a system to enable the participants to interpret and
judge the findings. In the MSC process during the story selection, participants discuss the relative
merit and worth of the individual outcomes (stories) and make judgements as to which stories best
reflect a significant account of change. Perhaps a further stage in the MSC process would be to
(periodically) analyse the stories en masse, rather than confining judgement to instances of localised
outcomes.
Level of extension outcomes addressed
Bennett’s Hierarchy can be used to help tease out the level of extension outcome that an evaluation
addresses, and whether the evidence is sufficient to make a legitimate judgement of worth and merit
about the program. Bennett (1977) lists seven levels of goals in extension (Section 4.5, Table 2),
and claims that it becomes more difficult to evaluate at higher levels of the hierarchy, as it becomes
261
more difficult to show that changes at these levels are the result of extension activity and not of
other factors. Conversely, changes at the lower level of the hierarchy are more easily attributed to
the intervention, but provide lower quality of evidence in terms of making valid determinations of
merit and worth. Using Bennett’s Hierarchy, good extension evaluation should include some
reference to changes at:
• ‘Level five’; changes in farmer knowledge, attitudes and skills, in addition to lower level
outcomes such as farmers participation in activities. As evaluations which only address issues
such as participation (level 3) would not provide any evidence of worth or merit of a program.
• ‘Level six’; changes in behaviour. Despite the fact that the higher level outcome of behavior
change is harder to attribute to the program activities alone, evaluation of these programs
should make an attempt to provide evidence concerning the extent to which these programs
contribute toward behavioural change (which is after all the ultimate aim of extension
programs).
The review of evaluation in extension (Chapter 4) revealed that of the 50 evaluation case studies
analysed, only a handful addressed the outcomes above Bennett’s level 4 (see Section 4.7). By
contrast, it appears that the MSC model can provide evidence that would sit in Bennett’s Hierarchy
levels 5 and 6. Changes in skills, and attitudes are very much the subject matter of the stories
produced. There is also some attempt to monitor changes in behaviour and outcomes in terms of
changes in on-farm practice and profitability. As the stories of change are embedded in the context
around the event in question, along with reasons why the change occurred, the issue of attribution is
brought to the surface rather than ignored. Because of this, it is also possible to provide more
credible evidence for change occurring at level 6 (behaviour change). It would appear that one of
the strengths of this approach is that it can monitor change at a higher level of the outcome
hierarchy than would a traditional approach based on simple performance indicators, and provide
some evidence for casual attribution.
Nevertheless, without some further meta-analysis of the stories, the MSC process provides only
limited information that would help stakeholders make a full determination of the merit and worth
of the project being evaluated.
262
9.11 Summary: Extent to which the MSC Model adds to
the ‘Basket of Choices’ for Extension in Australia
In terms of adding to the existing ‘basket of choices’ for extension evaluation, the MSC process is
novel in being able to cope with the new genre of extension projects with diverse, context-specific
outcomes. In Chapter 3 I suggested that agricultural extension projects pose their own particular
challenges to the practice of program evaluation. An analysis of the changing nature of extension
and its organisations led to the conclusion that there is increasing demand for monitoring
approaches capable of fostering dialogue and learning. A new genre of process-oriented extension
projects also present challenges for evaluation that objectives-based evaluation approaches are
unable to meet. The review of evaluation practice in Australia (Chapter 4) revealed a range of
approaches to evaluation employed in extension evaluation, but that most of the studies were
limited to a summative assessment of project impact. The review showed a predominance of
objectives-testing approaches. There also appeared to be very little project monitoring and few
approaches able to foster long-term dialogue between multiple stakeholders. For these reasons, it
would appear that the MSC model has much to offer the ‘basket of choices’ for extension
evaluation.
However, it should be cautioned that it is unlikely that the MSC model will suffice as stand-alone
evaluation approach. It is best used in careful combination with other evaluation approaches. I also
suggested that some project contexts are more conducive to gaining positive benefit from this
evaluation model than others. In particular, it would seem that large, process-orientated projects,
with high levels of farmer involvement, could gain most from this process.
9.12 Improving the Contribution of the MSC Model
It has been shown how the MSC model meets some of the premises for good evaluation, but does
not comply with them all. Like other evaluation approaches, the MSC model has inherent bias and
weaknesses. I would suggest two strategies to those considering adopting the MSC model. Firstly,
improvements that could be made to the process itself, to reduce some of the bias and
methodological weakness. Secondly the approach could be combined with other complementary
approaches to evaluation.
263
The MSC model could be improved by:
• Ensuring that there is a process to formally include the lessons learned from the stories into the
program planning, both long-term and short-term.
• Revising the sampling system of collecting stories to ensure that they are collected from
antagonists (including those who chose not to participate in the project) as well as advocates.
• Periodic review of the stories en masse. Unanticipated, valued outcomes should be identified
from the process, and a further step should be added to quantifying the spread of some these
outcomes. This could help to develop a system of quantifying dynamic indicators.
• Ensuring that more regular feedback is provided to the project from the purchasers.
The MSC model has quite a different set of biases to conventional methods of monitoring and
evaluation. For this reason, it is a particularly good tool to use in combination with other methods
of evaluation. The sorts of evaluation demands being made on extension projects under the current
reforms in extension organisations mean that it is highly unlikely that any one method of evaluation
would be sufficient on its own. Thus, the MSC model should be seen as one of a number of
methods, carefully chosen to offset different biases and meet the numerous evaluation demands. I
suggest that evaluation approaches that would work well with the MSC model, would be those that
can provide:
• evidence of spread of the emergent outcomes
• evidence of the achievement of predetermined outcomes, (if these have been articulated)
• evidence of the ‘average’ experience of participants (or of subgroups of participants) as well as
exceptional outcomes
• information on the views of non-participating dairy farmers, and other ‘victims’ of the project
• information to help improve internal organisational and management processes
• improved knowledge with regard to the logic of the project intervention
• evidence of whether conjectured outcomes have been achieved, and in what situations, and
why.
One model that would appear to combine well with the MSC model is Pawson and Tilley’s (1997)
‘Realistic Evaluation’ model. This would depend, of course, on the evaluation needs of the specific
project to be evaluated. In the case of the Target 10 Project, for example, the current evaluation of
264
achievement of valued outcomes could be replaced by a realistic approach to evaluation, helping to
meet the need for more evaluative information for accountability purposes. The following
discussion presents one way that this marriage could occur.
The MSC process could provide a ‘search mechanism’ for identifying conjectured outcomes and to
tap into the ‘folk wisdom’ of the project stakeholders. This information could then built into an
initial program theory model, in conjunction with the stakeholders. The construction of a program
theory model based on critical incidents could further help those constructing the model to
understand the underlying logic of the program. If, during the MSC process, some stories were
collected from antagonists of project practices, then it might also be possible to construct some
conjectured CMO configurations of unexpected negative outcomes.
The conjectured CMO configurations (developed from the MSC process) could then be used as the
starting point for a ‘Realistic Evaluation’. In this process, the conjectured CMOs would be refined,
and the achievement of these CMOs tested against empirical, and possibly quantitative data. It
might be beneficial at this point to employ external evaluators to conduct this part of the evaluation.
This would help provide evaluative information with high validity for project purchasers.
The MSC process could counterbalance some of the criticisms discussed in Section 5.5; i.e., that
‘Realistic Evaluation’ does not provide processes to:
• capture monitoring information
• address values (Julnes and Mark, 1998)
• involve stakeholders adequately in the evaluation (Leviton, 1994)
• incorporate more utilisation (Patton 1997) – the MSC model does this through the inclusion of
purchasers and other stakeholders actively in the evaluation process
• address unanticipated outcomes.
A realistic approach on the other hand, can counter criticism that the MSC process does not provide
generalisable findings, is not a theory-guided approach, and does not produce ‘objective’ evidence
of achievement of predetermined outcomes. In addition, potential outcomes of the MSC process,
such as enhanced communication and dialogue, would complement the sort of evaluative
information produced theory-driven evaluation processes and perhaps make it more palatable to
participatory or community based projects.
265
CHAPTER 10: CONCLUSIONS
The aim of this thesis was to develop a model of evaluation that had the potential to meet the needs
of agricultural extension projects in Australia. To develop this novel approach to evaluation, five
contemporary models of program evaluation were examined against a set of criteria. These criteria
were associated with the extent to which the model was (a) a novel approach, (b) could help
address the needs of evaluation at a national level, and (c) could meet the needs of one particular
case study project. From this analysis, the Most Significant Change model was selected (Davies
1996). It was a highly novel approach in that it had never been implemented in Australia before,
and was based on the unusual methodology of the collection and participatory selection of stories of
significant change.
Through the implementation of a modified version of the MSC model, I found that the process was
able meet the articulated needs of the stakeholders of one large extension project. This was
ultimately verified by their commitment to continue with the MSC process after a 12-month trial
was complete. The MSC model combined well with this extension project’s existing evaluation
practices that were objectives-based. The MSC model offered a process to encourage reflective
practice of the staff and collaborators, and helped them to make sense of unanticipated impact and
of each others’ values in relation to these impacts. The MSC model also helped draw staff, farmers
and other collaborators more centrally into the evaluation process.
The lessons learned from the implementation of the MSC model across the case evaluation were
considered in relation to the evaluation of agricultural extension projects more broadly. While the
case project cannot be considered representative of all large extension projects, the methods applied
in the meta-evaluation provided information concerning the sort of project contexts in which the
MSC model is likely to bring about desirable outcomes. In general, the MSC model showed
potential to address some of the evaluation needs of extension projects at a national level. It is
ideally suited to provide performance information that can foster organisational learning and
enhance communication in a context of organisational complexity and value pluralism. The stories
contain a wealth of information, both factual and tacit and the interpretations of the stories reveal
what stakeholders value and do not value in terms of impacts of project activities.
266
However, the MSC model does not offer processes to meet all the evaluation needs that were
identified at a national level. It did not provide the sort of data that is being demanded under the call
for greater accountability to clients and purchasers: it was not designed to provide the sort of data
generally accepted as ‘highly objective’. The MSC model as applied privileges the voice of people
internal (or close) to the project – those most likely to be advocates of the project. The MSC process
did not provide indication of the spread of adoption of technologies across the farming population:
the model is designed to capture ‘remarkable events’ rather than the average experience of farmers.
The MSC model aims to encourage learning between project stakeholders and to ‘improve’ the
project, rather than ‘prove’ what the project has achieved. Consequently, if used in isolation, the
MSC is unlikely to satisfy all stakeholders’ demands for evaluation.
In terms of adding to the existing ‘basket of choices’ for extension evaluation, the MSC process is
novel in that it is able to cope with the new genre of participatory extension projects. Where
projects have few or no pre-determined outcomes and produce diverse and context-specific
outcomes, the MSC model may be able to play an important stepping-stone to summative
evaluation. The MSC process can be viewed as a ‘search mechanism’ for unexpected or
unanticipated outcomes. Where outcomes are not predetermined the function of ‘searching’ for
outcomes could be very important. The MSC model could also be particularly relevant for
evaluating highly participatory projects. It can accommodate a wide range of participants in the
evaluation process itself. Highly participatory projects require participatory evaluation; approaches
that allow the stakeholders and beneficiaries to have a say in which changes are important, and
which changes should be measured. The MSC process has potential to meet these requirements.
However, I found that the MSC model does not meet with all the premises for good evaluation that
were developed in Chapter 2. Firstly, it is not guided by questions developed from a carefully
constructed model of the theory that underpins the project being evaluated. The MSC model
involves collecting impact information in an almost ‘goal-free manner’ and then coming an
understanding about key areas of findings through a process of induction. As a result of the lack of
emphasis on drawing the key questions from an explicit program theory, there is little analysis of
the underlying mechanisms that are triggered by the project.
Secondly, while the process may capture a range of unexpected outcomes perceived to be important
by the project stakeholders, it does not capture unexpected or unintended outcomes unless they are
deemed ‘significant’ by those involved in the process. This appears to be a serious omission to the
267
evaluation process. Thirdly, it can be argued that the MSC process was not socially /politically just,
in that it favoured the inclusion of some stakeholders over others. The MSC process is clearly
embedded within the discourse of the project staff and committee members. However, unlike other
research approaches, the bias itself if offered as another source of information about the
organisational capacity of the project.
Fourthly, the sampling technique of the MSC model is selective rather than inclusive. It will not
provide information on the ‘average condition’ of a farmer who attends the programs, but rather
will provide information about the exceptional circumstances, particularly successful
circumstances. Some would argue that such purposive sampling is not an appropriate technique to
produce reliable information to enable judgements to be made about the performance of the
program. Nevertheless, in qualitative research, purposive sampling is seen as a legitimate form of
data inquiry.
Finally, while stories of individual farmer outcomes were judged for their merit and worth by
stakeholders, there was no point at which the evaluators (stakeholders in this case) made an overall
determination of the merit and worth of the Target 10 Project. One of the strengths of this approach
is that it can monitor change at a higher level of the outcome hierarchy than would a traditional
approach based on simple performance indicators, and provide some evidence for casual attribution.
Nevertheless, without some further meta-analysis of the stories, the MSC process provides only
limited information that would help stakeholders make a full determination of the merit and worth
of the project being evaluated.
It has been shown how the MSC model meets some of the premises for good evaluation, but does
not comply with all. Like other evaluation approaches, the MSC model has inherent bias and
weaknesses. Two strategies were proposed to those considering adopting the MSC model. Firstly,
improvements could be made to the process itself, to reduce some of the bias and methodological
weakness. Secondly, the approach could be combined with other complementary approaches to
evaluation. The MSC model could be improved by including processes to formally include the
lessons learned into the program planning, ensure that the views of antagonists are captured,
periodically review the stories en masse, and ensure that regular feedback is provided to the project
from the funders.
268
As the MSC model presents quite a different set of biases to conventional methods of monitoring
and evaluation, it represents a particularly good tool to use in combination with other methods of
evaluation. The sorts of evaluation demands being made on extension projects under ‘new public
management’ mean that it is highly unlikely that any one approach to evaluation would be sufficient
on its own. I suggest that the MSC model should be strategically combined with other evaluation
approaches selected to offset the bias and to meet the other demands of evaluation. A model such as
Pawson and Tilley’s realistic evaluation might be particularly well suited for use alongside the MSC
process. Nevertheless, the MSC model would be more useful to extension if it was modified to
overcome at least some of its inherent biases.
The MSC model is an important contribution to the ‘basket of choices’ for extension evaluation, but
it does not provide an overarching solution to the myriad of evaluation demands being felt by
extension agents in Australia. I suggest that those evaluating extension projects should firstly come
to understand the evaluation needs of the project stakeholders, and then develop a carefully crafted
collection of evaluation approaches and methods to meet these requirements. For some projects, the
MSC model may provide an important new component of their evaluation bricolage. In particular,
it would seem that large, process-orientated projects, with high levels of farmer involvement, could
gain much from this process.
In this thesis, a novel evaluation approach has been empirically tested against standards for good
evaluation, and against evaluation needs. This type of research is extremely rare in the literature of
program evaluation, which is dominated by texts prescribing one evaluation model over others. I
suggest that meta-evaluation research conducted on other evaluation models could greatly assist
evaluators in the strategic selection and combination of evaluation approaches to meet the needs of
particular extension projects. In conducting meta-evaluation it seems valuable to adopt an empirical
and theory guided approach. Experience in this study suggests that Pawson and Tilley’s (1997)
Realistic Evaluation could provide a good model to use for this purpose, although more research is
needed to fine-tune such meta-evaluation techniques.
269
REFERENCES
Alasuutari, P. (1995), Researching Culture: Qualitative Method and Cultural Studies, Sage, London.
Alford, J. and O’Neill, D. (1994), The Contract State: Public Management and the Kennett Government,
Centre of Applied Social Research, Deakin University, Melbourne.
Alkin, M. C. (1972), ‘Evaluation comment: Wider context goals and goal-based evaluators’, Journal of
Educational Evaluation, 3, no.4, 10-11.
Alkin, M. C., Daillak, R. and White, P. (1979), Using Evaluations: Does Evaluation Make a Difference? Sage
Publications, Beverly Hills, CA.
Allen, W. J. (1998), ‘Towards improving the role of evaluation within natural resource management R&D
programs: The case for learning by doing’, Canadian Journal of Development Studies, 18 Special
Issue on Results-Based Evaluation, 629-643.
Anderson, F. M. and Dillon, J. L. (1985), ‘Farming systems research in the IARCs and other international
groups’, in Agricultural Systems Research for Developing Countries,vol. 11 (ed Remenyi, J. V.)
ACIAR Proceedings.
Appleyard, L. (1996a), ‘Target 10 Project evaluation’, in Agriculture and Food Initiative: Dairying.
Evaluation Report: Completed Projects, (ed Lew, M.) Department of Agriculture, Energy and
Minerals, Melbourne, 31-40.
Appleyard, L. (1996b), ‘APPRAISAL, A spreadsheet for evaluating returns to agricultural research’, Working
Paper, Performance Evaluation Division, Economics Branch, Department of Natural Resources and
Environment, Melbourne.
Argyris, C. and Schon, D. A. (1996), Organizational Learning II, Addison-Wesley Publishing Company,
Reading, Massachusetts.
Australasian Evaluation Society (1997), Membership Directory, Australasian Evaluation Society, Curtin,
Western Australia.
Bardsley, B. (1997), 3 Par Court, Bacchus Marsh, Victoria 3340.
270
Bardsley, J. B. (1981), Farmers’ assessment of information and ITS sources; An investigation using
interactive computer techniques, PhD thesis, Department of Agriculture, Horticulture and Forestry,
University of Melbourne.
Baume, P. S. (1977), ‘The politician’s role in the evaluation process’, Proceedings of the Conference of the
Australian and New Zealand Society for Epidemiology and Research in Community Health,
University of Adelaide.
Bawden, R. J. (1991), ‘Systems thinking and practice in agriculture’, Journal of Dairy Science, 74, 2362-
2373.
Bebbington, A. J., Merrill-Sands, D. and Farrington, J. (1994), ‘Farmer and community organisations in
agricultural research and extension: functions, impacts and questions’, ODI Agricultural
Administration (Research and Extension) Network paper, no. 47.
Beckingsale, D. (1997), Natural Resources and Environment, 8 Nicholson Street, East Melbourne, Victoria,
3002.
Beilin, R. (1998), Looking for Landcare: The landscape and the family farm, PhD thesis, RMIT University,
Melbourne.
Bell, J. (1983), Contemporary Social Welfare, Macmillan, New York.
Bennett, C. F. (1975), ‘Up the hierarchy’, Journal of Extension, March / April, 6-12.
Bennett, C. F. (1977), ‘Analysing impacts of extension programs’, Extension service, US Department of
Agriculture, Washington D.C.
Bickman, L. (1987), Advances in Program Theory, Jossey-Bass, San Francisco, CA.
Blank, S. C. (1999), The End of Agriculture in the American Portfolio, Quorum Books, Westport, USA.
Boje, D.-M. (1991), ‘The storytelling organization: A study of story performance in an office-supply firm’,
Administrative Science Quarterly, 36, no.1, 106-126.
Boomsma, J. (2000), Target 10 Dairy Extension Project, Natural Resources and Environment, RMB 2460,
Ellinbank, Victoria, 3821.
Boomsma, J., McDonald, B. and Walton, M. (1996), ‘Target 10: A report on the first three years’, Internal
report, Agriculture Victoria, NRE, Warragul.
271
Campbell, A. and Junor, R. (1992), ‘Land management and extension in the ‘90s: Evolution or
emasculation?’ Australian Journal of Soil and Water Conservation, 5, no. 2, 16-23.
Campbell, D.T. (1969), ‘Variation and selective retention in socio-cultural evolution’, General Systems, 16,
69-85.
Campbell, D. T. (1969b), ‘Reforms as experiments’, American Psychologist, 24, 409-429.
Campbell, D. T. (1991), ‘Methods for the experimenting society’, Evaluation Practice, 12, 223-260.
Campbell, K. O. (1980), Australian Agriculture: Reconciling Change and Tradition, Longman Cheshire,
Melbourne.
Caracelli, V. J. and Greene, J. C. (1997), ‘Crafting mixed-method evaluation designs’, in New Directions for
Program Evaluation, vol. 74: The Challenges and Benefits of Integrating Diverse Paradigms,
Advances in Mixed-Method Evaluation, (eds, Caracelli, V. J. and Greene, J. C.) Jossey-Bass, San
Francisco, CA.
Caron, D. J. (1993), ‘Knowledge required to perform the duties of an evaluator’, Canadian Journal of
Program Evaluation, 8, 59-78.
Cary, J. W. and Salmon, P. W. (1976), ‘Delphi and participatory planning: Focussing the planning process in
an agricultural service organisation’, Agricultural Extension Research Unit, School of Agriculture
and Forestry, University of Melbourne,
Chambers, R. (1983), Rural Development: Putting the last first, Longmann, Harlow, UK.
Chambers, R., Pacey, A. and Thrupp, L. A. (1991), Farmer First, Intermediate Technology Publications,
London.
Chen, H. T. (1990), Theory-Driven Evaluations, Sage Publications, Newbury Park, US.
Clark, R. A., Bourne, G. F., Cheffins, R. C., Esdale, C. E., Filet, P. G., Gillespie, R. C. and Graham, T. W. G.
(1996), The sustainable beef production systems project: Beyond awareness to continuous
improvement, Project Report Series, Queensland Department of Primary Industries, Brisbane.
Cook, D. L. (1966), Program Evaluation and Review Technique Applications in Education, Government
Printing Office, Washington D.C.
Cook, T. D. and Campbell, D. (1979), Quasi-Experimentation: Design Analysis Issues for Field Settings,
Rand McNally, Chicago.
272
Coutts, J. A. (1997), ‘Changes in extension: An Australian perspective. The coming of age of extension’,
Proceedings of 2nd Australasia Pacific Extension Network (APEN) Conference, Albury, New South
Wales, APEN, 22-32.
Cronbach, L. J. (1963), Evaluation for Course Improvement, Harper & Row, New York.
Cronbach, L. J., Ambron, S. R., Dornbusch, S. M., Hess, R. D., Hornik, R. C., Phillips, D. C., Walker, D. F.
and Weiner, S. S. (1980), Toward Reform of Program Evaluation: Aims, Methods, and Institutional
Arrangements, Josey-Bass, San Francisco.
Cunningham, J. B. (1993), Action Research and Organisational Development, Praeger, Westport,
Connecticut.
Curtis, A. and De Lacy, T. (1994), ‘Landcare: Does it make a difference?’ Charles Sturt University, Albury,
New South Wales.
Dart, J. J. (1997), ‘Facilitating tribal farmers to identify change to their farming systems using participatory
farming systems models’, Evaluation Journal of Australasia, 9, no.1&2, 31-37.
Dart, J. J. (1999a), ‘A story approach for monitoring change in an agricultural extension project’, Proceedings
of Conference of the Association for Qualitative Research, Melbourne, published on the world
wideweb: http://www.latrobe.edu.au/www/aqr/offer/papers/JDart.htm.
Dart, J. J. (1999b), Evaluating the ‘kinder surprise model’: The evaluation of the organisational service
delivery model of the Target 10 Dairy Extension Project, Evaluation report, Target 10 Project,
Natural Resources and Environment, Warragul, Victoria.
Dart, J. J., (ed) (1999c), ‘Target 10 evaluation stories, May 1998-May 1999’, Evaluation report, Target 10
Project, Department of Natural Resources and Environment, Warragul, Victoria
Dart, J. J., Petheram, R. J. and Straw, W. (1998), Review of Program Evaluation in Agricultural Extension,
Rural Industries and Research Development Corporation, no. 98/136, Canberra.
Datta, L. (1994), ‘Paradigm wars: A basis for peaceful coexistence and beyond’, The qualitative-quantitative
debate: New perspectives, (eds Reichardt, C. S. and Rallis, S. F.) 53-70.
Datta, L., E. (1997), ‘A pragmatic basis for mixed-method designs’, New Directions for Program Evaluation,
vol. 74: Advances in Mixed Method Evaluation: The Challenges and Benefits of Integrating Diverse
Paradigms (eds Green, J. C. and Caracelli, V. J.) Jossey-Bass, San Francisco, 33-46.
273
Davies, R., J. (1999), Centre for Development Studies, University College of Swansea, Singleton Park,
Swansea, SA2 8PP, Wales, UK.
Davies, R. J. (1996), An evolutionary approach to facilitating organisational learning: An experiment by the
Christian Commission for Development in Bangladesh, Centre for Development Studies, world wide
web publication: http://www.swan.ac.uk/cds/rd/ccdb.htm, Swansea, UK.
Davies, R. J. (1998), Order and Diversity: Representing and Assisting Organisational Learning in Non-
Government Aid Organisations, PhD thesis, The Centre for Development Studies (CDS), University
of Wales, Swansea, UK.
De Vaus, D. A. (1995), Surveys in Social Research, Allen and Unwin, St Leonards, New South Wales.
Denzin, N. K. and Lincoln, Y. S. (eds) (1998), Collecting and Interpreting Qualitative Materials, Sage,
Thousand Oaks, CA.
Department of Finance (1991), Handbook of Cost-benefit Analysis, AGPS, Canberra.
Department of Health (1993), Code of practice for the commissioning and management of research and
development, HMSO, London.
Dillon, J. L. (1965), ‘Farm management in Australia as an academic discipline’, Review of Marketing and
Agricultural Economics, 33,175-189.
DRDC (1994), DRDC- Funded Extension: Programs in Victoria, New South Wales and Tasmania, by Falvey
Consulting, Dairy Research and Development Corporation (DRDC), Melbourne.
Dunn, T. (1996), ‘Changing paradigms for farmers-researcher-extensionist relationships: Exploring methods
and theories of farmer participation in research’, European Journal of Agricultural Education and
Extension, 3, no.3, 167-181.
Eisner, E. W. (1985), The Art of Educational Evaluation: A Personal View, Falmer Press, London.
Engel, P. G. H. and Van den Bor, W. (1995), ‘Agricultural education from a knowledge systems perspective:
from teaching to facilitating joint inquiry and learning’, European Journal of Agricultural Education
and Extension, 1, no.4, 1-23.
Estrella, M. and Gaventa, J. (1998), Who Counts Reality? Participatory Monitoring and Evaluation: A
Literature Review, Institute of Development Studies, London.
274
Farquhar, R. N. (1961), Comparative extension education in anglo-saxon countries with particular references
to Australia, PhD thesis, Cornell University, US.
Farrington, J. and Nelson, J. (1997), ‘Using logframes to monitor and review farmer participatory research’,
Agricultural Research and Extension Network, no.73.
Feinstein, O. N. (1998), ‘Book review: Realistic Evaluation’, Evaluation, 4, no.2, 243-246.
Fisher, R. A. (1956), Statistical Methods and Scientific Inference, Oliver and Boyde, Edinburgh.
Fishman, D. B. (1992), ‘Postmodernism comes to program evaluation: A critical review of Guba and
Lincoln’s Fourth Generation Evaluation’, Evaluation & Program Planning, 15, no.3, 263-270.
Foale, M. A. (1997), ‘Management of dryland farming systems in North Eastern Australia. How can scientists
impact on farmer’s decision making?’, Agricultural Science, 10, no., 34-37.
Freebairn, J. W. (1989), ‘How to cut public spending’, Policy, Autumn.
Friere, P. (1972), Pedagogy of the Oppressed, Penguin, Harmondsworth, UK.
Geertz, C. (1973), The Interpretation of Cultures, Basic Books, USA.
Giddens, A. (1998), The Third Way: The Renewal of Social Democracy, Polity Press, Cambridge, UK.
Guba, E. G. and Lincoln, Y. S. (1981), Effective Evaluation, Jossey-Bass, San Francisco.
Guba, E. G. and Lincoln, Y. S. (1989), Fourth Generation Evaluation, Sage, Newbury Park, US.
Guerin, L. J. and Guerin, T. F. (1994), ‘Constraints to the adoption of innovation in agricultural research and
environmental management: a review’, Australian Journal of Experimental Agriculture, 34, 549-
571.
Guijt, I. and Gaventa, J. (1998), Participatory monitoring & evaluation: Learning from change, Policy
Briefing, Institute of Development Studies, Brighton,UK.
Hamilton-Smith, N. and Hopkins, M. (1998), ‘Theory-driven interviewing: from theory into practice’,
Bulletin de Methodologie Sociologique, 60, 80-102.
Harre, R. (1986), Varieties of realism: a rationale for the natural sciences, Blackwell, Oxford, UK.
275
Hart, E. and Bond, M. (1995), Action Research for Health and Social Care: A Guide to Practice, Open
University Press, Buckingham, UK.
Hood, C. (1991), ‘A public management for all seasons?’ Public Administration, 69, no.1, 3-19.
House, E. R. (1978), ‘Assumptions underlying evaluation models’, Educational Researcher, 7, no.3, 4-12.
House, E. R. (1991), ‘Realism in research’, Educational Researcher, 20, no., 2-9.
Hynes, H. J. (1963), ‘Opening address’, Proceedings of the Australian Agricultural Extension Conference,
Melbourne, 4-6.
Jiggins, J. (1995), ‘Development impact assessment: Impact assessment of aid projects in non-western
countries’, in Environmental and Social Impact Assessment, (Eds, Vanclay, F. and Bronstein, D. A.)
Wiley and Sons, Chichester, New York.
Julnes, G. and Mark, M. M. (1998), ‘Evaluation as sensemaking: knowledge construction in a realist world’,
in New Directions for Evaluation, vol. 78: Realist Evaluation: An Emerging Theory in Support of
Practice (Eds, Henry, G. T., Julnes, G. and Mark, M. M.) Jossey-Bass Publishers, San Francisco, 33-
53.
Julnes, G., Mark, M. M. and Henry, G. T. (1998), ‘Promoting realism in evaluation: Realistic evaluation and
the broader context’, Evaluation, 4, no.4, 483-504.
Kaplan, A. (1964), The Conduct of Inquiry, Chandler, San Francisco.
Kazi, M. A. F. (forthcoming), ‘Realist evaluation’, in The Blackwell Encyclopaedia of Social Work, (ed
Davies, M.) Blackwell, Oxford.
Kefford, B. (1999), Department of Natural Resources and Environment, 8 Nicholson Street, East Melbourne,
Victoria 3002.
Kolb, D. A. (1984), Experiential Learning: Experience as a Source of Learning and Development,
Englewood Cliffs, New Jersey.
Lane, J-E. (1997), ‘Public sector reform in the nordic countries’, in Public Sector Reform: Rationale, Trends
and Problems, (ed Lane, J-E.) Sage, London, 188-208.
Lasswell, W. (1936), Politics: Who Gets What, When, How, McGraw Hill, New York.
Lazerfield, P. F. and Rosenburg, M. (eds) (1955), The Language of Social Research, Free Press, Glencoe, IL.
276
Lees, J. W. (1990), ‘More than accountability’, Rural Development Centre, University of New England,
Armidale, New South Wales.
Leviton, L. C. (1994), ‘Program theory and evaluation theory in community-based programs’, Evaluation
Practice, 15, no.1, 89-92.
Lewin, K. (ed.) (1948), Resolving Social conflicts: Selected papers on group dynamics, Harper and Brothers,
New York.
Louis, M. R. (1980), ‘Surprise and sensemaking: What newcomers experience in entering unfamiliar
organizational settings’, Administrative Science Quarterly, 25, 226-251.
Macadam, R. D. (1996), ‘Developments in agricultural industries research and development in Australia with
particular reference on extension and the relevance of systematic developments as a response to the
current situation’, Proceedings of the Asia Pacific Seminar on Education for Rural Development,
Obihiro, Japan.
Madaus, G. F., Scriven, M. S. and Stufflebeam, D. L. (1983), Evaluation Models, Kluwer-Nijhoff Publishing,
Boston.
Manicas, P. T. and Secord, P. F. (1983), ‘Implications for psychology of the new philosophy of science’,
American Psychologist, 3, no., 399-413.
Marsh, S. P. and Pannell, D. J. (1997), ‘The changing relationship between private and public sector
agricultural extension in Australia’, Proceedings of the 2nd Australasia Pacific Extension Network
(APEN) Conference, Albury, New South Wales, APEN, 350-357.
Martin, S. and Sanderson, I. (1999), ‘Evaluating public policy experiments: Measuring outcomes, monitoring
processes or managing pilots?’ Evaluation, 5, no.3, 245-258.
McCown, R. L. (1991), ‘Research in a farming systems framework’, in Dryland Farming: A Systems
Approach, an Analysis of Dryland Agriculture in Australia (eds Squires, V. R. and Tow, P.G.),
Oxford University Press, Melbourne, 242-249.
McDonald, B. (1999), Natural Resources and Environment, RMB 2460, Ellinbank, Victoria, 3821.
McDonald, B. and Kefford, B. (1998), ‘Exploring a new approach to evaluation’, Proceedings of
Performance Measures for the Public Sector, Sydney, Institute for International Research, Sydney.
277
Morrison, D. (1993), ‘Square pegs in round holes: perspectives of cost-benefit gained from evaluation of
Western Australia Department of Agriculture activities’, Gains for Grains, 4: Benefit-cost Methods
in Research Evaluation, Canberra.
Nelson, C., Treichler, P. A. and Grossberg, L. (1992), ‘Cultural studies’, in Cultural Studies, (eds Grossberg,
L., Nelson, C. and Treichler, P. A.) Routledge, New York.
Nelson, R. and Winter, S. G. (1982), An Evolutionary Theory of Economic Change, The Belknap Press of
Harvard University Press, Cambridge, Mass.
NRE (1997), ‘Standard for project development and delivery’, by the Agricultural Industries Team, Natural
Resources and Environment (NRE), Victoria, Melbourne.
Oakley, P., Pratt, B. and Clayton, A. (1998), Outcomes and Impact: Evaluating Change in Social
Development, INTRAC, Oxford.
O’Brien, G. and Hepworth, G. (1993), ‘Evaluation of a pasture management program for dairy farmers in
West and South Gippsland’, Department of Agriculture, Melbourne.
O’Faircheallaigh, C., Wanna, J. and Weller, P. (1999), Public Sector Management in Australia: new
challenges new directions, MacMillan Education Australia, Melbourne.
Owen, J. M. (1993), Program Evaluation, Forms and Approaches, Allen and Unwin, New South Wales,
Australia.
Patton, M. Q. (1990), Qualitative Evaluation and Research Methods, Sage, Newbury Park,US.
Patton, M. Q. (1997), Utilization Focused Evaluation, Sage, Thousand Oaks, US.
Patton, M. Q. (1999), ‘Book review of realistic evaluation’, American Journal of Evaluation, 20, no.2, 385-
388.
Patton, M. Q. (2000), Re: Ohio’s Evaluation Conference, description of workshop run by Micheal Patton,
GOVETAL (an international E-mail discussion list on public sector program evaluation, Thursday
April 2000).)
Pawson, R. and Tilley, N. (1997), Realistic Evaluation, Sage, London.
Petheram, R. J. (2000), School of Forestry, University of Melbourne, Water Street, Creswick, Victoria, 3363.
Petheram, R. J. and Clark, R. A. (1998), ‘Farming systems research: relevance to Australia’, Australian
Journal of Experimental Agriculture, 38, 101-15.
278
Polanyi, L. (1985), Telling the American story: A structural and cultural analysis of conversational
storytelling, Ablex, Norwood, NJ.
Pollitt, C. (1997), ‘Evaluation and the New Public Management: An international perspective’, Proceedings
of the Australasian Evaluation Society Conference (AES), Adelaide, AES, 17-23.
Popper, K. (1959), The Logic of Scientific Discovery, Hutchinson, London.
Pretty, J. N. (1995), ‘Participatory learning for sustainable agriculture’, World Development, 23, no.8, 1247-
1263.
Pretty, N. P., Guijit , I. and Scoones, I. (1995), Participatory Learning and Action: a Trainer’s Guide,
International Institute for Environment and Development, London.
Prince, G. (1973), A Grammar of Stories, Mouton, The Hague.
Pusey, M. (1996), Economic Rationalism in Canberra, Cambridge University Press, Cambridge, UK.
Reason, P. (ed) (1988), Human Inquiry in Action: Developments in new paradigm research, Sage, London.
Rhoades, R. and Boothe, R. (1982), ‘Farmer-back to farmers: A model for generating agricultural
technology’, Agricultural Administration, 11,127-137.
Riches, J. R. H. (1973), Evaluation of a farm business management extension program, Working Papers,
Glenormiston Agricultural College, Noorat, Victoria.
Riessman, C. K. (1993), Narrative Analysis, Sage, Newbury Park.
Rist, R. C. (1997), ‘Evaluation and organisational learning: Some international perspectives’, Proceedings of
the Australasian Evaluation Society (AES), Adelaide, AES, 24-38.
Roberts, G. M. O. (1997), Action researching my practice as a facilitator of experiential learning with
pastoralist farmers in central West Queensland, PhD thesis, University of Western Sydney,
Richmond, New South Wales.
Roberts, K. C. (1998), A coexistive model of evaluation using Landcare as a case study, PhD thesis, School of
Natural and Rural Systems Management, University of Queensland, Gatton.
Rogers, E. M. (1962), Diffusion of Innovations, Free Press, New York.
279
Rogers, P. R. (1996), Evaluating approaches to program evaluation: A framework and its application to a
meta-evaluation of Patton’s Utilisation-Focused Approach, PhD thesis, Faculty of Education,
University of Melbourne.
Röling, N. (1988), Extension Science, Cambridge University Press, Cambridge.
Röling, N. and Jiggins, J. (1987), ‘Developments in European agricultural extension practices and the
implications for Australia - present and future’, Proceedings of the QDPI Extension Symposium,
Brisbane, Queensland Department of Primary Industries.
Rossi, P. H., Freeman, H. E. and Write, S. R. (1979), Evaluation: A systematic Approach, Sage publications,
Beverly Hills, CA.
Russell, D. B., Ison, R. L., Gamble, D. R. and Williams, R. K. (1989), A critical review of rural extension
theory and practice, Australian Wool Corporation, Sydney.
Scriven, M. (1967), ‘The methodology of evaluation’, Curriculum Evaluation, American Educational
Research Association Monograph Series on Evaluation, vol. 1 (ed Stake, R. E.), Rand McNally,
Chicago, 39-83.
Scriven, M. (1976), ‘Evaluation bias and its control’, Evaluation Studies Review Annual, vol. 1 (ed Glass
G.V.) Sage, Beverly Hills, 101-118.
Scriven, M. (1980), The Logic of Evaluation, Edgepress, Inverness, CA.
Scriven, M. (1986), ‘New frontiers of evaluation’, Evaluation Practice, 7, 7-44.
Scriven, M. (1991), Evaluation Thesaurus, Sage, Newbury Park,US.
Scriven, M. (1994), ‘The fine line between evaluation and explanation’, Evaluation Practice, 15, no. 1, 75-77.
Sechrest, L., West, S. G., Phillips, M., Redner, R. and Yeaton, W. (1979), ‘Some neglected problems in
evaluation research: Strength and integrity of treatments’, in Evaluation Studies Review Annual, vol.
4 (ed Sechrest & Associates), Sage, Beverly Hills, 15-35.
Shadish, W. R. (1992), ‘Theory-driven meta-evaluation’, in Using Theory to Improve Program and Policy
Evaluations, (eds Chen, H-T. and Rossi, P. H.), Greenwood Press, New York.
Shadish, W. R., Cook, T. D. and Leviton, L. C. (1995), Foundations of Program Evaluation, Sage, CA.
Shadish, W. R. and Epstein, R. E. (1987), ‘Patterns of program evaluation practice among members of
Evaluation Research Society and Evaluation Network’, Evaluation Review, 11, 555-590.
280
Sharp, C. and Lindsay, A. (1992), ‘An interim history of program evaluation in Australia and New Zealand
and the Australasian Evaluation Society’, Special Keynote Paper, Proceedings of Australasian
Evaluation Society (AES) International Evaluation Conference, AES, Melbourne.
Shaw, G., Brown, R. and Bromiley, P. (1998), ‘Strategic stories’, Harvard Business Review, May-June, 41-
50.
Shaw, H. (2000), Natural Resources and Environment, Bendigo, Cnr Midland Highway and Taylor Street,
Epsom, Victoria 3351.
Smith, M. F. (1989), Evaluability Assessment: A Practical Approach, Kluwer Academic, Boston.
Smith, N. L. (1994), ‘Evaluation models and approaches’, in International Encyclopedia of Education, vol. 4
(eds Husen, T. and Postlethwaite) Elsevier Science, Oxford, 2101-2109.
Spencer, A. (n.d.), ‘Improving the usefulness of the pigpulse report, an evaluation’, Rural Extension Centre,
University of Queensland, Gatton.
Spradley, J. P. (1979), The Ethnographic Interview, Holt / Rinehart and Winston, New York.
Sri Pathmanathan, C. (1978), Agricultural research project abstracts, Publications in Agricultural Extension,
University of Melbourne, School of Agriculture and Forestry, Melbourne.
Stake, R. E. (1967), ‘The countenance of educational evaluation’, Teachers College Record, 68, 523-40.
Stake, R. E. (1973), ‘Program evaluation: particularly responsive evaluation’, in Evaluation Models, (Eds,
Madaus, G. F., Scriven, M. S. and Stufflebeam, D. L.) Kluwer-Nijhoff, Boston.
Stanfield, R. B. (1997), The Art of Focused Conversation: 100 ways to Access group wisdom, A TOP
Method, Institute of Cultural Affairs, Melbourne.
Stewart, L. (1994), ‘A review of the monitoring and evaluation of property management planning in South
East Queensland’, Department of Primary Industries, Queensland.
Stufflebeam, D. L. and Webster, W. J. (1981), ‘An analysis of alternative approaches to evaluation’, in
Evaluation Studies Review Annual, vol. 6 (Eds, Freeman, H. E. and Solomon, M. A.) Sage, Beverly
Hills, 70-85.
Tashakkori, A. and Teddlie, C. (1998), Mixed Methodology: Combining Qualitative and Quantitative
Approaches, Sage, Thousand Oaks, Country.
281
The Virtual Consulting Group (1999), ‘Review of extension and human resource development for the
Australian dairy industry’, Evaluation report, Melbourne.
Toulemonde, J., Fontain, C., Laudren, E. and Vincke, P. (1998), ‘Evaluation in partnership: Suggestions for
improving their quality’, Evaluation, 4, no. 2, 171-189.
Tyler, R. W. (1967), ‘Changing the concepts of educational evaluation’, in Perspectives of Curriculum
Evaluation, vol. 1 (ed Stake, R. E.) Rand McNally, New York.
Van den Ban, A. W. and Hawkins, H. S. (1988), Agricultural Extension, Blackwell Science, Oxford.
Van Dissel, H. (1986), ‘Editorial comment’, AREX Newsletter, 3, no.1.
Vanclay, F. (1994), ‘A crisis in agricultural extension’, Rural Society, 4, no.1, 10-14.
Vanclay, F. and Lawrence, G. (1995), The Environmental Imperative: Eco-social Concerns for Australian
Agriculture, Central Queensland University Press, Rockhampton, Queensland.
Wadsworth, Y. (1991), Everyday Evaluation on the Run, Action Research Issues Association (incorporated),
Melbourne.
Watson, A., Hely, R., O’Keefe, M., Cary, J., Clark, N. and Allen, J. (1992), ‘Review of field-based services in
the Victorian Department of Food and Agriculture’, Department of Natural Resources and
Environment,Victoria.
Weick, K. E. (1995), Sensemaking in Organisations, Sage, Thousand Oaks, CA.
Weinstein, D. and Weinstein, M. A. (1991), ‘Georg Simmel: Sociological flaneur bricoleur’, Theory, Culture
and Society, 8, 151-168.
Wholey, J. S. (1983), Evaluation and Effective Public Management, Boston, Little Brown.
Whyte, W. F. (1991), Participatory Action Research, Sage, Newbury Park, USA.
Williams, D. B. (1968), Agricultural Extension, Melbourne University Press, Melbourne.
Winston, J. (1997), Program for Public Sector Evaluation (PPSE), Faculty of Applied Science, Royal
Melbourne Institute of Technology (RMIT), PO Box 71, Bundoora, Victoria 3083.
Winston, J. A. (1991), ‘Linking evaluation and performance management, review paper’, Proceedings of
National Evaluation Conference, Adelaide, Australasian Evaluation Society, 598-607.
282
Woodhill, J., Trevethan, P., Frankenburg, J., Beaulieu, C. and West Hume Landcare Group (1994), A case
study undertaken with the West Hume Landcare Group as part of an international research project
on participatory watershed development, International Institute for Environment and Development,
Land and Water Resources Research and Development Corporation; and Greening Australia.
Woods, E., Moll, G., Coutts, J., Clark, R. and Irvine, C. (1993), Information exchange: a review, Land and
Water Resources Research and Development Corporation, Canberra.
Worthen, B. R., Sanders, J. R. and Fitzpatrick, J. L. (1997), Program Evaluation: Alternative Approaches and
Practical Guidelines, Longman, New York.
Wortman, P. M. (1983), ‘Evaluation research: A methodological perspective’, Annual Review of Psychology,
34, 223-60.
Zohar, D. and Marshall, I. (1994), The Quantum Society: Mind, physics and a new social vision, Flamingo,
London.
Zuurbier, P. J. P. (ed) (1983), De Rekaties Tussen Onderzoek, Voorlichting en de Boer, Ministry of
Agriculture and Fisheries, The Hague.
284
APPENDICES
285
Appendix 1 Interview guides
The following lists of questions were used as guides for the interviewer in the semi-structured
interviews.
1.1 Guide questions for focus group with regional co-ordinators
1. Why do we need evaluation?
2. You know about the approach I am using, what kind of information do you think it will gather
and how are we going be able to use it?
3. Sentence completion: The thing that I find most useful for Target 10 about the approach is:
4. Sentence completion: The thing that I am most worried about with this process is….
5. Imagine that a year has gone past, how will we know that this approach has been useful?
1.2 Guide questions for focus-group of purchasers after the round table
meeting
1. Was there anything about this process – this methodology that makes you feel uncomfortable?
2. Was there anything about the process that you felt was particularly encouraging?
3. What messages should we as a group send back to Target 10?
4. If anther project comes to me i.e., TOPCROP, FARMSMART, WOOLPRO. And they say we are
interested in this story approach – should we do it? How should I respond to them?
1.3 Guide for semi-structured interviews with ten key informants after the trial
1. Do you remember when the story approach was first introduced to you? What were your initial
responses?
2. How do you feel about it now?
3. In what ways, if any, has the story approach affected the project? In other words, what impact
has the story approach had on the project?
Probe: how have the stories been used?
Probe: how could the story approach help improve the project?
4. How has it/ or will it affect your ability to do your work?
5. What changes do you recommend to the approach?
6. What kind of priority should this process have in the project?
286
Appendix 2 Internet meta-evaluation questionnaire for staff
287
288
289
290
291
292
293
294
Appendix 3 Internet ethics declaration
An ethics declaration was placed on the internet site for those responding to the questionnaire to
inspect. A hypertext link was provided from the questionnaire site (appendix 2) to the page shown
below.
295
296
297
Appendix 4 letter detailing rights of informants
The following letter was provided to all those who where to participate in either focus group
discussions or semi-structured interviews.
Dr R J Petheram Ms Jessica Dart Department of Forestry Institute of Land and Food Resources Longerenong College, RMB 3000, Horsham, Vic 3401
Department of Forestry Institute of Land and Food Resources Royal Parade, University of Melbourne, Parkville, Vic 3052
Tel 03 53 622253 Tel : (03) 9344 8356, Fax 03 53 622213 Fax: (03) 9344 4665 Email: [email protected] Email [email protected]
Dear Sir/Madam, This letter is to inform you of your rights of participation and confidentiality during your interview led by Jessica Dart. Jessica Dart is a PhD student from the Institute of Land and Food Resources, University of Melbourne. Her supervisor there is Dr R. Petheram. The research project that Jessica is undertaking, alongside the Target 10 Project staff, is concerned with developing a tool for evaluating agricultural extension projects. This tool involves the collection of stories of change and has been implemented with the Target 10 Dairy Extension Project since May 1997. In May 1997 this research project was endorsed by all four Regional Committees and by the Statewide Executive Committee of the Target 10 Project. These committees agreed to participate in this process for a period of one-year (June 1998 to June 1999) and to any interviews that may be associated with assessing the tool in June 1999. The interview that you have been asked to attend is concerned with your experience of being involved with the story approach. You are under no obligation to participate, and even after agreeing to participate, you may still choose to withdraw from the interview at any stage. During the interview you will be asked to about the extent to which you found the story approach to be useful, and about any issues of concern that you might have. For the purposes of her research Jessica is interested in both negative and positive attitudes towards the approach and we maintain that all views are valid. On your agreement, the interviews will be taped by Jessica and later transcribed. If you agree to this, your words may be quoted in publications, but your identity will be disguised. Any quotations used will be sourced in terms of the position of the informant and the individual identity will be protected at all times. (e.g., Source: NRE employee). Confidentiality of data will be preserved subject to any legal requirements. We have a commitment to the confidentiality of your responses (subject to legal requirements) so that you are not identifiable personally or any comments you make are not traceable to you personally. To ensure this, the following protocols will be followed: there will be no discussions between the researcher and research participants concerning comments made by others at separate interviews transcripts of interviews will be identified or coded in a way that is only recognisable to
298
the researcher (names and farm locations will be stored separately from the interview transcripts). In this way anonymity will be preserved. When results are reported, the identity of participants will be obscured through the use of pseudonyms only identifiable by the researcher. The findings of these interviews will be published in the PhD thesis of Jessica Dart, and in other academic publications. However, you are more than welcome to check any quotations prior to publication, and are free to withdraw your comments at any time.
If you have any inquiries please do not hesitate to ring Jessica Dart, or Dr RJ Petheram. Yours sincerely
Jessica Dart DR R J Petheram
299
Appendix 5 First round of Delphi questionnaire
In the case evaluation while developing the ‘domains of change’ for the MSC model, the Delphi
process was followed to ensure that the views of the numerous stakeholders were considered.
Delphi is a form of interactive (postal) surveying that utilises an iterative questionnaire and
feedback and provides participants with an opportunity to revise earlier views based on the response
of other participants, until some desired level of consensus is reached (Cary and Salmon, 1976).
This appendix shows that first letter and survey that was sent to participants, Appendix 6 shows the
results of this round, and Appendix 7 provides questionnaire for the second round of the process.
________________________________________________________________________________
Question Sheet – to Establish the Main ‘Types’ of Change to Monitor in the Target 10 Project In order to focus the monitoring process, we are seeking your input to help us choose the most important ‘types’ of change on which to collect stories. In this monitoring approach 3-4 broad ‘types’ of change are usually agreed upon before the story collection begins. Please indicate on this form which types of change you think are most important to monitor in order to understand the impact of the Target 10 activities. Please bear in mind that this approach aims to track change across all the programs and to gain information on the total impact of Target 10 activities. Please answer the following questions by placing tick in the box of the option that you most closely agree with: I believe that changes in perceived quality of life of project participants would be: π an essential type of change to monitor π a useful type of change to monitor π OK to monitor π of no value to monitor π wasteful to monitor π Other_____________________________________________________________________ I believe that changes in project participant skills (interpersonal, practical, coping) would be: π an essential type of change to monitor π a useful type of change to monitor π OK to monitor π of no value to monitor π wasteful to monitor π Other_____________________________________________________________________
300
I believe that changes in the goals and aspirations of project participants would be: π an essential type of change to monitor π a useful type of change to monitor π OK to monitor π of no value to monitor π wasteful to monitor π Other____________________________________________________________________ I believe that changes in farm productivity would be: π an essential type of change to monitor π a useful type of change to monitor π OK to monitor π of no value to monitor π wasteful to monitor π Other____________________________________________________________________ I believe that changes in total on-farm management practice would be : π an essential type of change to monitor π a useful type of change to monitor π OK to monitor π of no value to monitor π wasteful to monitor π Other____________________________________________________________________
Please indicate any other types of change that you feel should be monitored in order to understand the overall impact of Target 10 activities: ________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
Please also tick the appropriate boxes below: Which region are you from? Which group(s) do you participate in? a) Northern Irrigation Region ρ a) Target 10 Statewide Executive ρb) North East ρ b) Target 10 regional committee ρc) South West ρ c) Target 10 working groups ρd) Gippsland ρ d) Extension staff ρe) Other / not relevant ρ e) Purchaser group ρ
301
f) Other ρ
Are you? a) A farmer ρb) A University/TAFE provider ρc) A dairy factory representative ρd) An NRE employee ρe) An agribusiness representative ρf) Other___________________________ ρ
Thank you for completing this form. In 2-3 weeks time you will receive the results of this survey and will have a chance to re-cast your vote with consideration of what other stakeholders in the project consider should be monitored. If I have not received your reply within 10 working days I will assume that you chose not to participate at this stage.
302
Appendix 6 Results of the first round
Results Sheet In May a survey was sent to 150 of the Target 10 stakeholders. This sheet explains the results of this survey to which 101 people responded. 1. Results to the tick-box questions In the first mail-out you were asked to indicate which types of change you thought were the most important to monitor in order to understand the impact of the Target 10 activities. You were asked to answer the five questions by placing a tick in the box of the option that you most closely agreed with. For example: I believe that changes in perceived quality of life of project participants would be: π an essential type of change to monitor π a useful type of change to monitor π ok to monitor π of no value to monitor π wasteful to monitor The results of the first mail-out are presented in the graph below. It can be seen that more people thought that changes in on-farm management practice were essential to monitor than any other ‘type’ of change. The second most highly scored ‘type’ of change was changes in farm productivity and thirdly changes in project participants’ skills.
0%
25%
50%
75%
100%
Responses
On-farmmanagement
practice
Farm productivity Project participantskills
Goals andaspirations
Perceived quality oflife
OK
Useful
Essential
303
In addition to answering the tick-box questions, respondents were also encouraged to offer other ‘types’ of change that they felt should be monitored. The graph on this page shows the number of times each alternative ‘type’ of change was mentioned. On this graph only the ‘types’ of change that were mentioned more than twice are listed. The graph below shows that 13 respondents listed changes in profitability to be an important type of change to monitor. Five respondents also thought that changes in the way farmers make decisions were important to monitor.
In the response sheet you will be asked to make a second vote for the different ‘types’ of change that the project could monitor to understand the overall impact of Target 10. This time please consider the responses of the other Target 10 stakeholders while deciding how to vote.
You may have noticed that I have not included changes in ‘who is participating’ in the response sheet. This is because Target 10 already keeps records of how many people attend meetings and so forth.
0
5
10
15
Responses
Env
ironm
enta
lco
nditi
ons
Atti
tude
sto
war
dsto
T10
Dea
ling
with
chan
ge
Farm
erco
nfid
ence
Indu
stry
wid
ech
ange
Who
ispa
rtici
patin
g
Farm
erde
cisi
onm
akin
gsk
ills
Farm
prof
itabi
lity
304
Appendix 7 Second round of Delphi questionnaire
Question Sheet – to Establish the Main ‘Types’ of Change to Monitor in the Target 10 Project In order to focus the monitoring process, we are seeking your input to help us choose the most important ‘types’ of change on which to collect stories. In this monitoring approach 3 broad ‘types’ of change are usually agreed upon before the story collection begins. Once the ‘types’ of change are decided on, stories will be collected each month concerning this type of change. There will be an opportunity to collect stories concerning other types of change as well. This process of defining ‘types’ of change is being carried out in order to provide some focus to the process and to ensure that certain types of change are tracked every month. Please indicate on this form which types of change you think are most important to monitor in order to understand the impact of the Target 10 activities. Please bear in mind that this approach aims to track change across all the programs and to gain information on the total impact of Target 10 activities. Please consider the results of the first survey (yellow sheet) in making your decision. When we start the monitoring process we will track 3 main ‘types of change’. ******************************************************************************** Please choose 3 ‘types’ of change that you think are most important to monitor every month – place a tick in 3 boxes only.
Changes in on-farm management practice ρChanges in farm productivity ρChanges in project participant skills (interpersonal, practical, coping) ρChanges in goals and aspirations of project participants ρChanges in the quality of life of project participants ρChanges in farm profitability ρChanges in participants’ decision-making skills ρChanges in participants’ confidence ρChanges in participants’ ability to deal with changing industry conditions ρChanges in the industry in general ρChanges in attitudes towards Target 10 ρChanges in the environmental conditions on the farm ρOther_________________________________________________ ρOther_________________________________________________ ρ
Any comments that you would like to add about this process? ________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
305
Your name______________________________________________________________
Please also tick the appropriate boxes below:
Which region are you from? Which group(s) do you participate in?
a) Northern Irrigation Region � a) Target 10 Statewide Executive �b) North East � b) Target 10 regional committee �c) South West � c) Target 10 working groups �d) Gippsland � d) Extension staff �e) Other / not relevant � e) Purchaser group �f) Other__________________________________________________________________ Are you? a) A farmer � d) An NRE employee �b) A University/TAFE provider � e) An agribusiness representative �c) A dairy factory representative �f) Other__________________________________________________________________
Thank you for completing this form. If I have not received your reply within 10 working days I will assume that you chose not to participate at this stage. The results of this survey will be made available to all stakeholder groups and committees as soon as possible.
306
Appendix 8 Facilitators guide for story selection process
The following guide details how the story selection process was facilitated during project meetings.
1. Write all the titles of the stories on the whiteboard, grouped by domain. Leave a space next to
each story for additional comments e.g.
Domain Title Comments 4 Using the consultant Strong, written by a farmer, incomplete, story not finished 4 Feeling empowered Moving story, beginning middle and end. Attribution to project is
questionable. Great story, not sure if it is about the project. 4 Better decisions for the
family Good solid story. Heard many times before. Small change yet crucial. Not sure about the dates mentioned
4 Now I understand OK, not enough information to really understand what it going on..
2. Ask the participants to read out all the stories belonging to the first domain of change. After
each story ask,
What is this story really about?
Write any comments next to the title on the white board as above.
3. When all the stories have been read out for the first domain, facilitate a discussion about the pros and cons of the stories and jot comments on the white board. Prompts can include:
• Is there any one story that particularly stands out to you?’ - why is that?
• Does a different story stand out to anyone else’?
• What about this story, no one has mentioned this?
• Are there any stories that stand out to you as being not so good - why is that?"
• What do you think of the stories in general?’
4. Ask the committee to move to a vote by hands, one vote per person. If there is total agreement,
about which story to select, move onto the next domain. If there is a spread of votes, facilitate a
further discussion and try to reach a consensus. In some cases, it may be better to choose two
stories for one domain, or choose non at all. A re-vote is sometimes helpful.
307
Appendix 9 Example vote form for Selection process
The following form was sent out to staff at the Central Executive Committee meetings, along with
the nominated stories from the four regions. The idea was that committee members would read the
stories before the meeting and record their impressions of each individual story in the form below –
to act as a memory jogger. The stories were then discussed during the meeting.
Domain: Title of story: Comments
Changes in farmer decision making skills
A) ‘More accurate decisions made now’ B) ‘Don’t need the consultant now’
C) ‘Becoming independent.
Changes in on-farm practice
G) ‘Production graphs help change practice’
H) ‘Top operator’
Changes in profitability
I) ‘I’ll not be milking cows when I am 55!’
J) ‘Doubled herd’s late lactation production’
Other type of changes:
D) ‘Information from neighbours’
E) ‘Meeting farmers needs’ F) ‘The most useful information ever received’