an#integrated#approach#to#evaluation:## a#participatory# ...eprints.qut.edu.au/97728/3/ellise...
TRANSCRIPT
An#integrated#approach#to#evaluation:##A#participatory#model#for#reflection,#evaluation,#analysis#and#
documentation#(the#‘READ’#model)#in#Community#Arts#
!
!
!
!
!
!
!
!
!
!
!
Ell ise!Jane*Ann!Barkley!
Creative!Industries,!Queensland!University!of!Technology!
!
Submitted!in!fulfilment!of!the!requirements!of!the!degree!of!!
Doctor!of!Creative!Industries!
!
July!2016!
!
!
–!!This!dissertation!is!accompanied!by!the!READ%in%Practice%portfolio!!–%
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! ii!
#
#
#
#
#
#
#
#
#
#
#
©!2016!–!Ellise!Barkley.!
Cover!image:!Designed!by!Phoebe!Barton!
!
#
! #
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! iii!
Abstract#
Evaluation!is!a!routine!requirement!for!external!funding!in!community!arts,!and!practitioners!face!mounting!
pressure!to!demonstrate!the!value!of!their!work!in!competitive!environments.!Evaluating!the!art*making!
processes,!outcomes!and!impact!of!community!arts!requires!practitioners!to!traverse!myriad!philosophical!and!
technical!challenges.!Despite!the!significant!theoretical!developments!over!the!past!30!years,!effective!
evaluation!practice!remains!elusive,!unresolved!and!highly!contested.!At!one!extreme,!opponents!argue!
evaluation!in!its!current!usage,!oriented!towards!the!quantification!of!arts!impact,!is!detrimental!for!the!field.!!
This!research!focuses!on!the!development,!operational!use,!and!critical!review!of!the!READ!model,!a!dynamic!
and!rigorous!model!for!evaluation!of!community!arts.!The!innovation!of!the!model!lies!in!the!integration!of!
four!key!appraisal!and!learning!strategies*!Reflection,!Evaluation,!Analysis!and!Documentation!(READ)!*!to!
cater!for!the!multi*faceted!demands!of!complex!creative!partnership!initiatives.!The!READ!concept!was!
collaboratively!devised!at!the!site!of!my!professional!practice,!and!my!role!as!researcher!has!involved!
transforming!the!READ!concept!into!a!substantiated!model!of!practice!and!evaluation!product,!by!engaging!
with!partners!and!participants,!leading!the!design!and!delivery!process,!and!then!evaluating!the!efficacy!of!the!
developed!model!in!action.!The!body!of!evidence!represents!the!first!iteration!of!critically!analysing!the!READ!
model’s!development!and!testing!during!the!initial!phase!of!the!Nuclear!Futures!Partnership!Initiative!(2013!*!
2015),!an!international!arts!and!cultural!program!funded!by!the!Australia!Council!for!the!Arts!(2014!*!2016).!!
The!research!is!embedded!in,!and!furthers,!my!own!professional!evaluation!practice,!advancing!my!expertise!
in!the!design!and!delivery!of!reflexive!learning!and!appraisal!systems!suited!to!the!creative!arts!context.!The!
findings!establish!an!important!foundation!for!the!further!development!of!innovative!evaluation!practices!
across!the!Nuclear!Futures!program,!including!the!identification!of!the!newly!developed!model’s!strengths!and!
constraints!critical!for!the!program’s!summative!evaluation!in!2017.!At!the!site!of!practice,!the!successful!READ!
approach!created!impact!through!an!enhanced!understanding!of!the!artistic,!theoretical,!academic!and!wider!
social!implications!of!the!community!arts!practice,!and!promoted!engagement,!acceptance,!and!participation!
in!evaluation!by!mixed!stakeholder!groups.!!
In!addressing!identified!knowledge!gaps!on!effective!community!arts!evaluation,!the!research!engages!with!
the!complexities!of!evaluation!of!creative!partnership!initiatives,!delivering!an!original!and!practical!
contribution!that!will!have!significant!impact!within!the!field.!!For!the!field,!the!READ!model!offers:!
•! an!integrated!evaluation!method!that!can!be!conducted!‘in!house’;!
•! a!model!of!evaluation!best!practice!that!embraces!the!ethos!of!community!arts;!
•! a!flexible!framework!for!generating!robust!data,!that!is!linked!with!management!and!creative!arts!
practice!so!that!data!can!be!fed!back,!continuously,!into!the!project!for!real*time!impact;!
•! original!strategies!for!engagement!and!for!developing!stakeholder!ownership!of!project!evaluation,!
particularly!where!trepidation!or!resistance!to!traditional!evaluation!means!exists;!
•! possibilities!for!more!general!application!in!(research*led)!creative!management.!
! !
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! iv!
! !
Blank!page!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! v!
Table#of#contents#
Abstract!...................................................................................................................................!iii!
Table!of!contents!......................................................................................................................!v!
List!of!figures!............................................................................................................................!ix!
List!of!tables!.............................................................................................................................!x!
List!of!abbreviations!.................................................................................................................!xi!
Acknowledgements!................................................................................................................!xiii!
1! Introduction+...............................................................................................................+1!
1.1! Research!context!....................................................................................................................!3!
1.2! Background!.............................................................................................................................!3!
1.3! Problem!statement!.................................................................................................................!6!
1.4! Aims!and!objectives!................................................................................................................!8!
1.5! READ!product!development!...................................................................................................!9!
1.6! The!scope!and!significance!of!the!research!..........................................................................!11!
1.7! The!DCI!journey!....................................................................................................................!12!
1.7.1! Research!design!overview!.............................................................................................!12!
1.7.2! Research!phases!............................................................................................................!13!
1.7.3! Examinable!DCI!outputs!................................................................................................!14!
1.8! Chapter!overview!.................................................................................................................!14!
1.8.1! Dissertation!roadmap!...................................................................................................!17!
2! Contextual+scan+........................................................................................................+19!
2.1! Community!arts!....................................................................................................................!20!
2.1.1! Community!arts!and!cultural!development!(CACD)!.....................................................!20!
2.1.2! The!rise!of!community!partnerships!............................................................................!21!
2.2! Evaluation!.............................................................................................................................!23!
2.2.1! Program!evaluation!......................................................................................................!24!
2.2.2! Participation*oriented!approaches!to!program!evaluation!..........................................!26!
2.3! Community!arts!evaluation!..................................................................................................!28!
2.3.1! Trends!in!community!arts!evaluation!...........................................................................!29!
2.3.2! Relevant!approaches!and!models!for!the!Nuclear!Futures!site!of!practice!.................!32!
2.4! Conclusion!............................................................................................................................!37!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! vi!
!
3! Research+design+........................................................................................................+39!
3.1! Qualitative!research!methodology!.......................................................................................!40!
3.1.1! Practice*informed!research!paradigm!..........................................................................!41!
3.2! Research!strategy:!forensic!reflective!practice!....................................................................!42!
3.2.1! Theoretical!underpinnings!of!reflective!practice!..........................................................!42!
3.2.2! Methodological!considerations!for!forensic!reflective!practice!...................................!44!
3.3! Data!collection!and!analysis!methods!..................................................................................!45!
3.3.1! Reflective!methods!.......................................................................................................!47!
3.3.2! Interviews!and!consultation!.........................................................................................!52!
3.3.3! Observation!..................................................................................................................!54!
3.3.4! Evaluation!theory!and!READ!product!development!.....................................................!54!
3.4! Research!phases!...................................................................................................................!57!
3.5! Ethical!considerations!..........................................................................................................!57!
3.6! Conclusion!............................................................................................................................!58!
4! Site+of+practice+.........................................................................................................+59!
4.1! Nuclear!Futures!Partnership!Initiative!(Nuclear!Futures)!.....................................................!60!
4.2! Balaklava!Community!Arts!Project!.......................................................................................!64!
4.2.1! Origins!and!inspiration!.................................................................................................!64!
4.2.2! Project!partners!and!stakeholders!...............................................................................!66!
4.2.3! Project!overview!...........................................................................................................!67!
4.2.4! Summary!of!the!Balaklava!community!arts!practice!....................................................!69!
4.3! Conclusion!............................................................................................................................!78!
! !
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! vii!
!
5! The+READ+application,+Balaklava+(2014H15)+...............................................................+79!
A!guide!to!the!READ!application!.............................................................................................................!80!
Part+A:+Envisaging+the+READ+model+and+preliminary+activities+.........................................+81!
5.1! Background!...........................................................................................................................!81!
5.1.1! The!READ!model!...........................................................................................................!81!
5.1.2! A!systems!approach!......................................................................................................!85!
5.1.3! Research!phases!...........................................................................................................!87!
5.2! READ!approach!and!assumptions!.........................................................................................!89!
5.2.1! Context!and!parameters!...............................................................................................!89!
5.2.2! Internal!position!for!READ!facilitation!..........................................................................!90!
5.2.3! Interpretative!stance!....................................................................................................!90!
5.2.4! Participatory*oriented!approach!..................................................................................!91!
5.3! Scoping!.................................................................................................................................!92!
5.3.1! Stakeholders!identification!and!consultation!...............................................................!92!
5.3.2! Methods!and!analysis!...................................................................................................!93!
5.3.3! Results!for!the!scoping!phase!.......................................................................................!95!
Part+B:+READ+methodology+in+action+................................................................................+98!
5.4! Design!and!implementation!.................................................................................................!98!
5.4.1! Support!structures!........................................................................................................!99!
5.4.2! Reflection!...................................................................................................................!101!
5.4.3! Evaluation!...................................................................................................................!108!
5.4.4! Analysis!.......................................................................................................................!116!
5.4.5! Documentation!...........................................................................................................!120!
5.5! READ!outputs,!sharing!and!use!..........................................................................................!125!
5.6! Conclusion!..........................................................................................................................!131!
! !
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! viii!
!
6! Findings+..................................................................................................................+133!
6.1.1! Analysis!of!the!READ!application!................................................................................!134!
6.1.2! Questions!arising!from!forensic!reflective!practice!....................................................!135!
6.2! Findings!and!discussion!......................................................................................................!137!
6.2.1! Modular!and!integrated!approach!.............................................................................!137!
6.2.2! Social!constructivist!stance!........................................................................................!142!
6.2.3! Internal!facilitation!.....................................................................................................!146!
6.2.4! Emergent!evaluation!practice!....................................................................................!149!
6.2.5! Meta*evaluation!.........................................................................................................!153!
6.2.6! Opportunities!for!enriching!evaluation!practice!........................................................!164!
6.3! The!final!READ!product!......................................................................................................!169!
6.3.1! Product!development!.................................................................................................!169!
6.3.2! Profile!for!the!READ!product!......................................................................................!171!
6.3.3! Infographic!.................................................................................................................!172!
6.3.4! Assumptions,!constraints!and!variables!.....................................................................!175!
6.4! Chapter!conclusion!.............................................................................................................!178!
7! Conclusion+..............................................................................................................+181!
7.1! The!DCI!research!................................................................................................................!182!
7.2! Summative!conclusions!......................................................................................................!186!
7.3! Implications!and!significance!of!the!research!....................................................................!193!
7.4! Dissemination!.....................................................................................................................!196!
7.5! Future!research!..................................................................................................................!197!
7.6! Dissertation!conclusion!......................................................................................................!199!
Appendix+1:!Interview!design!for!scoping!interviews…………………………………………………….…………….203!
Bibliography+..................................................................................................................+205!
!
Supplementary!material!
Document+2:+READ%in%Practice+portfolio+(RIPP)+
Companion!document!to!the!dissertation!(164!pages)!–!See!attached!document!
! !
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! ix!
List#of#figures##
Figure!1.1:!The!Doctor!of!Creative!Industries!research!design!............................................................!12!
Figure!1.2:!The!six!research!phases!for!the!DCI!research!.....................................................................!13!
Figure!1.3:!Roadmap!for!dissertation!...................................................................................................!18!
Figure!3.1:!The!Doctor!of!Creative!Industries!research!design!............................................................!41!
Figure!3.2:!Model!for!forensic!reflective!practice!................................................................................!43!
Figure!3.3:!Hierarchy!of!theory!development!via!forensic!reflective!practice!.....................................!55!
Figure!4.1:!Map!of!South!Australia!and!relevant!place*names!............................................................!61!
Figure!4.2:!Map!of!Balaklava!and!Wakefield!Regional!Council!area,!South!Australia!..........................!64!
Figure!5.1:!The!READ!model!as!system!................................................................................................!86!
Figure!5.2!:!The!research!phases!for!the!DCI!........................................................................................!88!
Figure!5.3:!Key!elements!of!Community!Arts!project!to!be!reviewed!.................................................!90!
Figure!5.4:!Mapping!Reflection!within!READ!for!the!Balaklava!Community!Arts!Project!..................!107!
Figure!5.5:!!READ!timeline!of!activities!and!outputs!for!Balaklava!Community!Arts!Project!.............!129!
Figure!6.1:!Extracts!from!reflective!practice!......................................................................................!147!
Figure!6.2:!Evaluation!theory!tree!by!Christie!and!Alkin!(version!3)!..................................................!150!
Figure!6.3:!Structured!design!and!development!for!the!READ!product!.............................................!170!
Figure!6.4:!The!READ!model!infographic!............................................................................................!173!
!
!
Box!6.1:!Reflective!questions!arising!from!the!forensic!reflective!practice!sessions!for!the!DCI!.......!136!
!
! #
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! x!
List#of#tables#
Table!3.1:!Qualitative!methods!employed!for!the!DCI!research!design!(2013*15)!..............................!46!
Table!4.1:!Key!activity!strands!for!the!Nuclear!Futures!program!(2014*16)!........................................!63!
Table!6.1:!Summary!of!READ!model!processes!and!outputs!.............................................................!152!
! #
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! xi!
List#of#abbreviations#
AES! Australasian!Evaluation!Society!
ANVA! Australian!Nuclear!Veterans!Association!
CACD! Community!arts!and!cultural!development!
CCD! Community!cultural!development!
CCPI! Creative!Communities!Partnership!Initiative!
KEC! Key!evaluation!checklist!
KEQ! Key!evaluation!questions!
Nuclear!Futures! Nuclear!Futures!Partnership!Initiative!
READ! The!READ!model:!Reflection,!Evaluation,!Analysis,!
Documentation!
RIPP! READ%in%Practice!portfolio!(companion!document) !
SA! South!Australia!
SLSA! State!Library!of!South!Australia!
SWOT! Strengths,!weaknesses,!opportunities,!threats!(situation!
analysis!tool)!
QUT! Queensland!University!of!Technology!
!
!
!
! #
Ellise Barkley 19th July, 2016
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
xii!
The!work!contained!in!this!dissertation!and!accompanying!portfolio!has!not!been!previously!
submitted!to!meet!requirements!for!an!award!at!this!or!any!other!higher!education!
institution.!To!the!best!of!my!knowledge!and!belief,!the!dissertation!contains!no!material!
previously!published!or!written!by!another!person!except!where!due!reference!is!made.!
QUT Verified Signature
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! xiii!
Acknowledgements#
I!extend!my!appreciation!to!my!supervisory!team:!Professor!Helen!Klaebe!and!Associate!Professor!Sandra!
Gattenhof!for!the!quality!input,!direction!and!expertise!contributed,!as!well!as!their!sunny!encouragement.!I!
thank!Dr!Jess!Dart!for!the!wisdom!and!depth!of!knowledge!she!brought!to!the!research!as!industry!mentor,!
and!her!generosity!in!sharing!her!practitioner!insights.!Thanks!also!to!Adjunct!Professor!Cheryl!Stock!and!Dr!
Lee!McGowan!for!their!leadership!through!the!DCI!process!and!igniting!my!interest!in!reflective!practice.!
Substantial!input!for!this!research!has!been!offered!at!my!site!of!practice,!and!I!am!indebted!to!all!those!who!
have!contributed!to!the!investigation!(directly!and!inadvertently).!First,!my!colleague!Paul!Brown!(Creative!
Producer,!Alphaville)!has!made!significant!intellectual!contribution!and!provided!constructive!commentary!to!
advance!the!research.!There!is!always!something!to!learn!from!Paul!Brown!–!thank!you.!I!am!appreciative!for!
the!ideas,!effort!and!cooperation!of!the!Nuclear!Futures!team!and!partners,!who!worked!on!the!collective!
design!and!application!of!the!READ!model.!In!particular!community!members!Avon!and!Kate!Hudson,!and!
creative!team!members!Jessie!Boylan,!Carly!Friedrichs,!Linda!Dement,!Teresa!Crea!and!John!Romeril!–!what!
delightful!folk!to!work!with.!I!thank!State!Library!of!South!Australia!staff!for!helping!set!the!compass!for!the!
community!archiving!in!the!research,!and!am!grateful!to!the!hundreds!of!participants!who!have!engaged!with!
the!READ!process!through!the!art*making!and!evaluation!practice.!I!also!thank!the!Alphaville!arts!organisation!
for!their!cooperation!during!the!research!and!for!making!Nuclear!Futures!program!content!available.!
Behind!the!scenes,!the!research!has!been!made!possible!by!the!love,!patience!and!humour!of!my!family!and!
friends.!In!particular:!the!unwavering!generosity!of!Peter!Graham!(words!cannot!express),!the!optimism!and!
encouragement!of!daughters!Lily!and!Sophie,!the!thoughtful!input!of!my!mum,!Jan,!and!the!caring!support!of!
Karen!and!Ian.!I!would!like!to!thank!Jaymie!Maley!for!her!graphic!design!services!and!input!during!the!
development!of!the!READ!infographic,!Mel!Rumble!for!copyediting!and!proofreading!Chapters!1,!6!and!7!(in!
accordance!with!the!university’s!editing!guidelines),!and!Matt!Bailey!for!technical!support!with!the!electronic!
compilation!of!the!READ%in%Practice!portfolio!(companion!document!to!this!dissertation);!and!for!the!
energising!friendship!of!all!three!during!these!processes.!
! !
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! xiv!
!
Blank!page!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 1!
1! Introduction#
This!Doctor!of!Creative!Industries!(DCI)!research!was!conceived!through!a!practitioner’s!lens,!devised!
to!address!the!evaluation!needs!at!the!site!of!my!professional!practice!in!community!arts.!The!field!
of!community!arts!encompasses!the!wide!range!of!arts*based!collaborative!practices!run!by,!or!with,!
communities!in!partnership!with!professional!artists!and!arts!workers!(Australia!Council!for!the!Arts,!
2011).!Over!the!past!30!years,!despite!increasing!requirements!and!developments!in!evaluation!in!
this!context,!no!agreed!professional!framework!for!evaluating!community!arts!exists,!and!the!criteria!
for!effective!and!appropriate!evaluation!practice!in!community!arts!remains!highly!contested.1!!
The!focus!of!this!research!is!the!development,!operational!use,!and!critical!review!of!the!READ!
model,!a!dynamic!and!rigorous!model!for!evaluation!of!community!arts.!The!innovation!of!the!model!
lies!in!the!integration!of!four!key!appraisal!and!learning!strategies!*!Reflection,!Evaluation,!Analysis!
and!Documentation!(READ)!*!to!cater!for!the!multi*faceted!demands!of!complex!creative!community!
partnerships.!Prior!to!this!research,!‘READ’!existed!in!concept!form!only,!arising!from!professional!
practice!and!coined!in!2012!by!Sydney*based!arts!organisation!Alphaville!to!meet!the!needs!of!the!
Nuclear!Futures!Partnership!Initiative!(Nuclear!Futures).!I!was!a!key!contributor!in!this!process,!
which!resulted!in!a!one*page!outline!and!statement!of!intent!for!the!to*be*developed!READ!model!
(see!Section!1.5).!The!DCI!offered!a!means!to!advance!the!READ!concept,!create!a!model!of!practice,!
and!test!the!READ!methodology!developed!in!an!industry!setting!to!study!its!efficacy!and!potential.!!
The!research!is!embedded!in,!and!furthers,!my!own!professional!evaluation!practice,!advancing!my!
expertise!in!the!design!and!delivery!of!reflexive!learning!and!appraisal!systems!suited!to!the!creative!
arts!context.!It!also!establishes!seminal!protocols!for!evaluation!practice!at!my!site!of!practice.!My!
role!as!researcher!involved:!engaging!with!project!partners!and!participants!at!my!site!of!practice;!
facilitating!stakeholder!input!and!participation;!leading!the!methodological!design!and!delivery!of!
the!READ!model;!trialling!READ!methods;!consolidating,!articulating!and!analysing!the!model;!and!
creating!communication!materials!such!as!the!READ!infographic!and!partner!resources.2!Through!the!
DCI!I!have!also!designed!a!practitioner*friendly!READ!product!for!future!practice!(see!Section!6.3).!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
1!A!standard!definition!for!evaluation!is:!‘the!systematic!collection!and!analysis!of!information!to!make!judgments,!usually!
about!the!effectiveness,!efficiency!and/or!appropriateness!of!an!activity’!(Australasian!Evaluation!Society,!2010,!p.3).!2!In!the!DCI,!I!refer!to!my!role!as!‘READ!facilitator’!to!define!the!active,!leadership!role!I!played!in!administering!the!
collective!READ!application!at!my!site!of!practice.!This!was!not!a!neutral!role,!and!I!was!the!driving!agent!for!coordinating,!
delegating,!and!enabling!the!collective!input,!skills!development,!and!stakeholder!brokerage!necessary!for!the!model’s!
development.!Further,!beyond!the!collective!process,!I!played!a!major!role!in!designing!the!READ!methodology!presented,!
contributing!significant!intellectual!input!into!the!model’s!development,!and!have!independently!generated!the!original!
findings!and!knowledge!on!the!READ!model,!based!on!my!data!collection!and!analysis,!as!presented!in!Chapter!6!and!7.!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 2!
The!body!of!evidence!presented!in!this!dissertation!represents!the!first!iteration!of!trialling!and!
critically!analysing!the!READ!model’s!development!during!the!initial!phase!of!the!Nuclear!Futures!
initiative!(2013!*!2015),!a!three!year!international!arts!and!cultural!program!funded!by!the!Australia!
Council!for!the!Arts!(2014!*!2016).!The!Nuclear!Futures!program!brings!together!artists!and!atomic!
survivors!to!develop!creative!works!exposing!the!legacies!of!the!atomic!age.!For!the!DCI,!I!studied!
the!first!industry!development!and!application!of!the!READ!concept,!conducted!for!the!Balaklava!
Community!Arts!Project!that!was!delivered!in!South!Australia!as!part!of!Nuclear!Futures.!In!order!to!
study!and!evaluate!the!READ!application!in!the!Nuclear!Futures!setting,!I!employed!a!rigorous!
qualitative!research!methodology!for!the!DCI,!utilising!forensic!reflective!practice!and!methods!such!
as!written!and!video!reflection,!interviews,!digital!storytelling!and!observation!(refer!to!Chapter!3!for!
the!DCI!research!design).!The!DCI!research!design!created!the!critical!lens!to!systematically!review!
my!evaluation!practice,!and!the!READ!application,!to!generate!the!DCI!findings.!
In!addressing!identified!knowledge!gaps!on!effective!community!arts!evaluation,!the!research!
engages!with!the!complexities!of!evaluation!of!creative!partnership!initiatives!and!contributes!to!the!
discourse!within!the!field!on!cultural!value,!offering!an!innovative!stakeholder*oriented!approach!to!
program!evaluation!that!aligns!with!the!ethos!of!community!arts.!The!impact!of!the!research!has!
been!significant!and!transformative,!and!the!READ!model!of!practice!developed!has!extended!and!
enriched!my!own!evaluation!practice,!as!well!as!improved!the!evaluation!practice!at!the!Nuclear!
Futures!site!of!practice.!The!findings!establish!an!important!foundation!for!the!further!development!
of!innovative!evaluation!practices!across!the!three*year!Nuclear!Futures!program,!and!have!
contributed!to!an!enhanced!understanding!of!the!artistic,!theoretical,!academic!and!wider!social!
implications!of!the!Nuclear!Futures!community!arts!practice.!!
In!this!introductory!chapter,!Chapter!1,!I!establish!the!motivation!and!problem!statement!for!the!DCI!
and!state!the!aims!and!objectives!of!the!enquiry.!I!explain!why!the!research!problem!is!significant!for!
my!professional!practice!as!program!manager!and!evaluator,!and!how!the!problem!relates!more!
broadly!as!a!persistent!issue!facing!community!arts!practitioners!and!evaluators.!Chapter!1!
concludes!with!a!synopsis!of!the!dissertation!structure!and!research!progression!(see!Section!1.8.1!
for!the!dissertation!roadmap),!providing!an!overview!of!each!chapter!in!the!dissertation.!
Accompanying!the!dissertation!is!an!important!companion!document,!the!READ%in%Practice!portfolio.!
The!portfolio!includes!examples!from!the!industry*based!evaluation!practice,!and!comprises!
documentation!and!artefacts!demonstrative!of!the!scope!of!the!READ!model!development!and!
application!undertaken!for!the!research.!The!dissertation!and!READ%in%Practice!portfolio!are!
submitted!as!a!set,!and!are!two!interrelated!yet!distinct!research!outputs!resulting!from!the!DCI.!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 3!
1.1! Research#context##
The!research!presented!in!this!dissertation!has!been!completed!as!part!of!the!Doctor!of!Creative!
Industries!(DCI)!in!the!Creative!Industries!Faculty,!Queensland!University!of!Technology!(QUT).!The!
DCI!is!a!professional!research!doctorate!for!industry!practitioners!that!combines!advanced!critical!
reflection!of!practice!with!rigorous!scholarship.!The!research!is!dually!positioned!in!both!academic!
and!industry!contexts!in!the!field!of!community!arts.!Community!arts!encapsulates!the!diverse!range!
of!arts*based!collaborative!practices!run!by,!or!with,!communities!in!partnership!with!professional!
artists!and!art!workers!(MacDowall,!Mulligan,!Panucci,!&!Badham,!2013).!!
The!research!problem!arises!out!of!professional!community!arts!practice,!conceived!through!an!
industry!lens!from!an!evaluator!and!program!manager!perspective.!The!topic!of!research!represents!
an!area!of!professional!interest!(and!challenge)!associated!with!my!vocation!as!manager!and!
evaluator!over!the!past!15!years!in!community!arts!and!cultural!development,!education,!and!
sustainability!across!local!government,!business!and!community!organisation!settings.!The!research!
problem!is!positioned!within!the!unfolding!contemporary!debate!in!the!field!on!cultural!value,!
framed!by!the!trend!toward!cultural!measurement!and!the!rising!conceptualisations!of!evaluation!as!
a!means!to!articulate,!assess!and!quantify!the!impact!of!community!arts!practice.!
1.2! Background#
The!impetus!for!the!research!began!during!the!pre*program!development!of!the!Nuclear!Futures!
Partnership!Initiative!(Nuclear!Futures)!in!2012,!led!by!Sydney*based!arts!production!company!
Alphaville,!while!in!my!role!as!prospective!Nuclear!Futures!program!manager.3!Nuclear!Futures!is!a!
three*year!international!arts!and!cultural!program!grounded!in!linked!community!arts!and!cultural!
development!projects,!delivered!between!2014*16!(Alphaville,!2015).!The!initiative!is!driven!by!
international!partnerships,!including!the!collaboration!of!artists!working!with!atomic!survivor!
communities!to!bear!witness!to!the!legacies!of!the!atomic!age!through!creative!arts.!Across!the!
program,!communities!and!artists!use!theatre,!film,!paintings,!sculpture,!photography,!digital!arts!
and!other!art!forms!to!make!creative!works!that!reflect!both!the!horror!of!living!with!nuclear!
radiation!and!the!resilience!of!affected!communities.!The!latter!part!of!the!program!involves!the!
internationalisation!of!the!community*based!work,!including!the!touring!and!showcasing!of!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
3!Alphaville!is!an!arts!organisation!specialising!in!community!arts,!cultural!development!and!production,!with!a!focus!on!
social!justice!and!sustainability!issues.!I!was!officially!appointed!manager!in!late!2013,!once!funding!was!secured.!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 4!
integrated!artworks!in!multiple!countries.!The!image!below!is!one!example!of!the!creative!works.4!!
!
%%
10%Minutes%to%Midnight!immersive!projection!installation,!part!of!the!Nuclear!Futures!!
showcase!presented!for!the!Balaklava!Community!Arts!Project.!Photo!by!Tania!Safi.!
!
In!order!to!understand!the!background!to!the!emergent!READ!evaluation!practice,!it!is!necessary!to!
provide!a!brief!historical!overview!of!the!development!of!the!Nuclear!Futures!program.!
In!2013,!Nuclear!Futures!received!funding!approval!via!the!then!Creative!Communities!Partnership!
Initiative!(CCPI)!scheme!administered!by!the!Australia!Council!for!the!Arts,!and!has!since!secured!
supplementary!funding!via!other!government,!private!and!philanthropic!sources.!Underpinned!by!a!
community!partnerships!logic!(see!Chapter!2),!the!Nuclear!Futures!program!is!characterised!by!
diverse!and!inter*sectoral!partnerships,!a!multi*arts!and!experimental!creative!practice,!and!has!a!
focus!on!community!development!and!mutual!partnership!benefits.!Like!many!externally!funded!
arts!programs,!evaluation!became!a!priority!for!the!program!as!both!a!prerequisite!for!some!of!the!
funding!agencies!approached,!and!as!a!valued!practice!for!review!and!assessment!by!many!of!the!
stakeholders.!At!a!minimum,!the!Australia!Council!stipulated!that!10!per!cent!of!Nuclear!Futures’!
grant!value!be!dedicated!toward!evaluation,!which!represented!a!significant!investment!given!the!
level!of!funding!secured!through!the!CCPI.!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
4!Full!program!details!and!creative!works!are!listed!on!Alphaville’s!Nuclear!Futures!website:!http://nuclearfutures.org.!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 5!
With!over!20!partners!involved,!Alphaville!was!eager!to!ensure!that!resources!dedicated!to!
evaluation!were!directed!towards!learning!and!evaluative!practices!that!could!offer!wide!and!
relevant!benefits!to!all!partners!and!a!range!of!stakeholders.!Early!consultation!with!partners!and!
stakeholders!had!revealed!that!for!evaluation!to!be!effective!in!this!context,!it!would!need!to:!meet!
an!array!of!program!and!partner*specific!evaluation!needs,!in!ethical!and!culturally!appropriate!
ways;!and!be!based!on!rigorous,!reflexive!and!inclusive!methodologies!designed!to!generate!
partner*preferred!outputs!and!learning!exchange.!As!is!detailed!in!Chapter!5,!there!were!both!
general!and!site*specific!evaluation!challenges!arising!from!the!multiple!and!culturally!diverse!set!of!
partners!participating,!each!having!a!stake!in!how!the!program!evaluation!was!conducted!and!how!
the!community!partnerships!and!artworks!were!to!be!represented.!In!anticipating!the!program’s!
divergent!evaluation!needs,!prior!to!securing!the!Australia!Council!funding,!Alphaville!explored!
potential!options!for!solving!the!problem!of!how!to!evaluate!the!Nuclear!Futures!program.!It!was!
hoped!that!if!an!original!and!effective!approach!for!the!complex!CCPI!was!designed!in*house,!it!
could!be!tailored!to!meet!the!needs!at!the!site!of!practice,!as!well!as!establish!protocols!for!the!
future!evaluation!practice!of!Alphaville!and!partners.!
Through!Alphaville’s!process,!I!was!a!key!member!of!a!team!of!creative!practitioners!who!devised!
then!went!on!to!implement!the!READ!concept,!a!participatory!approach!integrating!Reflection,!
Evaluation,!Analysis!and!Documentation!(READ).5!The!READ!concept!had!been!inspired!by!the!
anticipated!needs!of!the!Nuclear!Futures!CCPI!and!represents!a!practice*led!response!to!the!
challenges!arising!from!community!arts!evaluation.!Conceived!was!a!wide!summative!and!integrated!
framework!to!help!creative!practitioners!and!community!partners!reflect!on,!represent!and!appraise!
their!art*based!activities,!partnerships!and!impact.!As!is!elaborated!in!the!dissertation,!the!READ!
approach!is!grounded!in!the!philosophies!and!art*making!traditions!of!community!arts!and!cultural!
development,!and!spans!across!creative!practice,!program!management!and!evaluation.!Prior!to!the!
DCI!research,!the!concept!for!the!READ!framework!based!on!the!four!overarching!READ!strategies!
had!been!identified,!however!had!not!been!developed!beyond!a!basic!level!(see!Section!1.5).!No!
READ!methodology!or!model!of!practice!existed!or!had!been!tested.!To!take!the!READ!concept!to!
the!next!level,!to!advance!the!hypothesis!into!a!working!model,!I!turned!to!the!Doctor!of!Creative!
Industries!as!the!means!for!completing!this!work!as!part!of!a!process!of!distinct!academic!research.!
The!DCI!professional!doctorate!offered!a!unique!opportunity!for!me!as!an!industry!practitioner!to!
apply!rigorous!scholarship!to!the!task!of!developing!the!READ!model!at!my!site!of!practice.!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
5!This!team!was!led!by!Paul!Brown!as!Creative!Producer!and!myself!as!program!manager!for!Nuclear!Futures,!for!the!host!
organisation!Alphaville.!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 6!
While!the!evaluation!practice!for!Nuclear!Futures!–!and!in!particular,!for!the!Balaklava!Community!
Arts!Project,!which!was!the!sole!focus!of!the!DCI!research!–!was!structured!as!interrelated!with!my!
professional!role!as!the!program!manager,!the!DCI!research!enquiry!was!designed!and!approached!
as!separate!from!this!managerial!role.!Framed!through!my!practitioner!perspective,!the!DCI!research!
fits!within!the!tradition!of!artists,!managers!and!evaluators!seeking!to!address!the!persistent!
epistemological!and!technical!problems!raised!by!community!arts!evaluation!in!practice.!In!doing!so,!
the!research!aimed!to!embrace,!as!well!as!unleash,!the!potential!of!evaluation!within!its!community!
arts!setting!by!designing!and!experimenting!with!solutions!grounded!in!the!creative!arts!practice.!
1.3! Problem#statement#
Practitioners!endeavouring!to!appraise!the!art*making!processes,!outcomes!and/or!impact!of!
community!arts!must!traverse!myriad!philosophical!and!technical!challenges,!particularly!for!large!
and/or!multifaceted!programs!(MacDowall!et!al.,%2013).!As!a!routine!requirement!of!government!
and!philanthropic!funding,!evaluation!has!become!a!recurrent!aspect!of!community!arts!practice!and!
management!(Keating,!2002).!While!substantial!advances!have!been!made!over!the!past!30!years,!
community!arts!evaluation!remains!problematic,!ad!hoc!and!inconsistent!(Goldbard,!2015;!Belfiore!
&!Bennett,!2010).!In!the!literature,!many!areas!remain!unresolved,!such!as!best!practice!standards!
for!evaluating!sizeable!or!complex!programs,!or!an!agreed!professional!framework!for!practice.!!
Community!arts!practitioners!also!face!mounting!pressure!to!demonstrate!the!value!of!their!work!in!
terms!of!the!instrumental!benefits!and!impact!of!the!arts!practice!(Badham,!2015;!Hadley!&!
Gattenhof,!2012).!As!discussed!in!Chapter!2,!the!trend!to!measure!the!social!and!economic!impacts!
of!the!arts!practice,!linked!to!the!instrumentalisation!of!the!arts!and!debate!on!cultural!value,!has!
strongly!influenced!how!contemporary!community!arts!practice!is!framed,!evaluated!and!
represented!(MacDowall,!2015;!Matarasso,!1996).!Arising!are!epistemological!debates!in!the!
literature,!with!some!opponents!arguing!that!evaluation!in!its!current!usage!is!detrimental!and!
contravenes!the!traditional!ethos!of!community!arts!practice!(Goldbard,!2015,!2008;!Belfiore,!2014).!!
In!the!industry!domain,!the!notion!of!what!constitutes!best!practice!and!effective!evaluation!is!also!
elusive!(MacDowall!et!al.,%2013).!While!some!good!practitioner!guides!and!platforms!exist!for!
sharing!the!program!evaluation!models!that!are!currently!available!(see!Chapter!2),!overall,!there!is!
limited!public!access!to!industry*produced!exemplars!(and!critical!analysis!of!these).6!As!a!
practitioner!interested!in!learning!from!evaluations!in!the!public!domain,!the!task!of!ascertaining!the!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
6!Based!on!data!arising!from!interviews!with!industry!stakeholders!and!practitioners!undertaken!for!this!research,!and!from!
my!own!experience!of!seeking!permissions!via!organisations!to!access!exemplars!during!the!contextual!scan!for!the!DCI.!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 7!
quality!of!examples!accessed!is!self*determined,!given!there!is!no!industry!standard!or!benchmark!
for!best!practice!in!the!field.!Questions!also!arise!about!how!representative!industry!exemplars!are!
of!evaluation!and!evaluative!practice!in!the!field,!as!community!arts!evaluation!has!been!
traditionally!orientated!towards!acquittal!and!accountability!agendas,!which!may!not!always!reflect!
the!cultural!or!aesthetic!values!of!all!participating!partners!(Hadley!&!Gattenhof,!2012).!!
Based!on!observation!from!practice,!and!anecdotal!discourse,!there!appears!to!be!a!tendency!for!
community!arts!practitioners!to!feel!pressure!to!apply!a!positivist!and!utilitarian!evaluative!lens,!or!
to!orientate!towards!conservative!or!success*oriented!framing!of!evaluation!findings.7!Practitioners!
adopting!a!cautious!or!narrow!approach!may!do!so!for!myriad!reasons,!such!as!in!response!to:!
perceived!funder!expectations;!the!desire!to!enhance!future!funding!eligibility;!time!and!resource!
constraints;!or!lack!of!evaluation!expertise!(MacDowall!et!al.,%2013;!Fitzpatrick,!Sanders,!&!Worthen,%
2012;!Badham,!2010).!Also,!evaluation!practice!since!the!late!twentieth!century!has!been!influenced!
by!the!global!movement!toward!cultural!and!progress!measurement,!which!has!led!to!increasing!
attempts!to!quantify!cultural!value!and!arts!impact!via!indicators!and!metrics!(MacDowall,!2015).!!
Beyond!the!standard!evaluation!approaches!undertaken!for!acquittal!reporting!in!the!field,!harder!to!
find!are!evaluation!exemplars!from!industry!that!provide!genuine!insight!into!the!internal!and!
nuanced!organisational!learning,!appraisal!and!knowledge*generating!processes!at!play!in!the!
community!arts!practice!and!evaluation,!or!practitioner!reflections!on!experimental!practices!and!
failures!during!art*making!and!evaluation.!For!creative!practitioners,!evaluators!and!managers!
responsible!for!evaluating!community!arts,!the!philosophical!and!technical!challenges!of!evaluation!
in!this!context!must!be!traversed!independently!(Australasian!Evaluation!Society,!2010).!That!is,!in!
the!absence!of!industry*wide!models!or!benchmarks!for!best!practice,!practitioners!must!use!
evaluation!models!according!to!what!they!determine!to!be!most!effective!and!ethical!on!a!project*
to*project!basis.!In!undertaking!the!methodological!design!and!implementation!of!community!arts!
evaluation,!the!challenges!may!include:!addressing!diverse!or!unclear!stakeholder!evaluation!needs;!
partnership!brokerage;!method!selection!to!ensure!rigorous!and!defensible!findings;!and!delivering!
end*utility!and!relevant!outputs!for!partners!(MacDowall,!2015;!Fitzpatrick!et!al.,%2012;%Davidson,!
2005).!With!the!rise!of!participatory*oriented!evaluation!for!community!arts!initiatives,!such!issues!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
7!Based!on!discourse!I!was!privy!to!at!three!industry!events!organised!as!part!of!the!three*year!Australian!Research!Council!
(ARC)!Linkage!Project!Towards%an%integrated%evaluation%framework%for%communityAbased%arts!by!the!Centre!for!Cultural!Partnerships,!University!of!Melbourne!(see!MacDowall!et!al.,!2013).!This!included:!a!two*day!roundtable!with!40!
stakeholders,!a!masterclass!for!postgraduate!students!on!the!topic!of!community!arts!evaluation,!and!network!and!forum!
discussions!with!members!of!the!international!community!of!community!arts!workers!and!evaluators!attending!the!
Spectres%of%Evaluation%conference!held!in!2014!at!Footscray!Arts!Centre,!Melbourne.!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 8!
become!compounded!depending!on!the!scale!and!complexity!of!the!partnerships!involved!(Guba!&!
Lincoln,!2004).!It!is!not!surprising!that!trepidation!of!evaluation!is!evident!in!the!field!(Belfiore,!2014;!
Goldbard,!2008).!
The!epistemological!and!technical!quandaries!arising!from!the!task!of!evaluating!community!arts,!
and!the!broader!debate!on!cultural!value,!forms!the!backdrop!to!the!evaluation!challenges!faced!in!
my!professional!practice.!From!my!experience,!questions!of!what!constitutes!effective!evaluation!
have!been!recurrent!in!my!roles!of!program!manager!and!evaluator.!On!a!project*by*project!basis,!
over!the!many!years!that!I!have!worked!in!these!domains,!I!have!attempted!to!continuously!improve!
my!evaluation!practice!and!build!on!the!mixed!successes!that!have!ensued.!The!scope!and!scale!of!
the!Nuclear!Futures!arts!practice!however!presented!uncharted!ground!from!my!previous!evaluation!
experience.!In!order!to!gain!the!necessary!in*depth!understanding!of!community!arts!evaluation!
relevant!for!tackling!the!Nuclear!Futures!evaluation!problem,!in!2013!I!decided!to!frame!this!
industry*based!challenge!as!a!research!problem!via!the!DCI.!As!a!research!problem,!the!question!of!
how!to!evaluate!the!Nuclear!Futures!program!had!significance!for!my!site!of!practice!and!my!own!
professional!development,!and!strongly!resonated!with!identified!research!and!knowledge!gaps!
currently!present!in!the!field.!
1.4! Aims#and#objectives#
This!DCI!seeks!to!make!an!original!contribution!to!the!discourse!on!effective!evaluation!in!community!
arts,!by!developing!and!examining!a!newly!devised!participation*oriented!model!for!creative!
community!partnerships!and!community!arts!that!adopts!an!integrated!and!wide*lens!approach!to!
evaluation.!!
Specifically,!the!research!aimed!to!develop!the!pre*existing!READ!concept,!coined!during!professional!
practice!prior!to!this!research,!into!an!operational!model!through!an!iterative!industry*based!process!of!
development,!application,!refinement!and!critical!analysis!in!order!to!create!a!substantiated!READ!
product!for!immediate!and!future!evaluation!practice.!!
!
!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 9!
The!DCI!research!addressed!the!overarching!questions:8!
a)! How!does!the!READ!model!contribute!to!program!learning!and!appraisal!for!creative!
community!partnerships!within!community!arts,!based!on!the!Nuclear!Futures!Partnership!
Initiative!experience?!!
b)! What!innovation!does!READ!offer!evaluation!practice!and!program!evaluation?!
The!research!objectives!of!the!DCI!were!to:!
•! Ascertain!the!dominant!conceptualisations!and!challenges!of!evaluation!identified!from!
representatives!in!community!arts.!
•! Identify!and!assess!the!program!evaluation!needs!of!Nuclear!Futures!partners!and!stakeholders.!
•! Examine!current!evaluation!practice!and!models!of!evaluation!for!community!arts!of!relevance!
to!the!Nuclear!Futures!program!evaluation.!
•! Develop!and!implement!the!proposed!READ!model!for!industry!application.!
•! Modify,!improve!and!evaluate!the!READ!model!at!the!site!of!practice.!
•! Refine!and!critique!the!READ!model!product!as!a!methodology!for!community!arts!evaluation.!
•! Contribute!to!the!discourse!on!effective!evaluation!practice,!identifying!areas!of!innovation!and!
wider!practitioner!relevance.!
1.5! READ#product#development#
In!meeting!the!above!aims!and!objectives,!there!are!two!key!ways!the!research!has!advanced!the!
READ!concept.!First,!through!the!process!of!designing,!and!experimenting!with,!an!applied!READ!
methodology!for!the!Balaklava!project!a!model!of!practice!specific!to!that!context!was!developed.!
This!included!the!how!and!why!of!the!approach!as!it!was!designed,!applied!and!refined!(see!Chapter!
5).!Second,!in!parallel!and!interrelated!to!this,!I!examined!the!emergent!evaluation!practice!with!a!
view!to!identify!the!elements!of!the!evaluation!practice!and!methodology!that!might!be!
transferrable!and!relevant!for!a!general!READ!model!of!practice!(applicable!and!reliable!for!other!
projects!across!the!Nuclear!Futures!suite!and!beyond!the!site!of!practice).!!As!explained!in!Chapter!6,!
I!examined!the!theoretical!and!technical!aspects!of!the!developed!model!in!order!to!gain!an!in*depth!
understanding!of!the!READ!approach,!its!operational!framework,!and!the!implications!for!evaluation!
practice.!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
8!To!meet!the!requirements!of!the!DCI!course,!this!research!was!conducted!as!two!one*year!research!projects,!Project!1!
and!Project!2,!in!2014!and!2015!respectively.!Both!Project!1!and!2!focused!on!the!industry*based!Balaklava!Community!
Arts!Project!and!READ!model!application.!For!the!final!DCI!submission,!this!dissertation!and!accompanying!READ%in%Practice!portfolio,!I!present!the!research!aims,!phases!and!findings!of!Project!1!and!2!as!a!complete!(integrated)!body!of!work.!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 10!
Culminating!from!these!processes,!I!developed!an!evaluation!product!for!READ!that!encapsulated!
the!refined!elements!and!principles!of!the!model,!to!present!a!simplified!and!generic!READ!model!of!
practice!communicated!via!a!practitioner*friendly!infographic!and!profile!statement.9!It!was!
envisaged!that!the!READ!product!(presented!in!Chapter!6)!would!be!useful!for!the!future!evaluation!
practice!of!the!Nuclear!Futures!program,!as!well!as!a!tool!for!my!own!professional!evaluation!
practice!and!that!of!other!practitioners!or!organisations.!The!process!of!developing!the!model!of!
practice!and!READ!product!has!led!to!new!understanding!into!previously!unknown!aspects!of!the!
READ!concept,!including!original!knowledge!related!to!the!tested!methodological!design.!!
In!order!to!demonstrate!the!level!to!which!the!READ!concept!has!been!developed!via!this!research,!
it!is!helpful!to!reference!my!starting!point.!Below!is!an!extract!from!Alphaville’s!funding!submission!
to!the!Australia!Council!for!the!Arts!(based!on!ideas!from!2012)!that!illustrates!the!extent!to!which!
the!READ!concept!had!been!formulated!and!articulated!prior!to!this!research.!I!refer!to!this!early!
iteration!of!the!approach!(prior!to!the!research)!as!the!READ!concept!or!conceptual!READ!model:!
We!propose!to!develop!a+new+participatory+model+that+integrates+Reflection,+Evaluation,+
Analysis+and+Documentation!(the!‘READ’!model)!to!evaluate!the!program’s!processes,!
partnerships!and!creative!outputs.!!
•! We!will!use!a!partnership!model!based!on!participatory!evaluation!methodology!and!
reflective!analysis,!with!evaluative!goals!set!by!partners!to!inform!academic*supported!
framework!design!and!implementation.!
•! The!approach!blends!traditional!data!collection!and!analysis!(qualitative!and!quantitative)!
with!emergent!techniques!such!as!‘Most!Significant!Change’!process!(story*based!
evaluation).!It!involves!all!partners!in!asking!the!questions,!data!collection!and!analysis.!(It!
is!socio*dynamic.)!
•! READ!includes!formative!evaluation!and!analysis!for!ongoing!appraisal!and!summative!
evaluation!at!completion,!with!focus!on!CACD!processes,!the!creative!works!and!the!
quality!of!the!partnerships.!
•! Reflective!analysis!originates!in!individual!and!group!practice!within!individual!projects,!
and!proceeds!via!conferences,!seminars!and!publications!that!link!creative!artists,!
academics!and!communities.!!
Strengths:!
•! The!overall!approach!is!documentation*oriented!(written,!visual,!audio,!ephemeral/!rich!
media,!as!well!as!archival),!with!commitment!to!disseminate!READ!findings!to!
communities,!partners!and!industry!to!make!lessons!explicit!&!accessible!to!wider!field.!
Our!on*line!community!is!a!key!delivery!mechanism.!
•! READ!is!evidence*based,!continuous!over!multiple!years,!and!guided!by!socially!inclusive!
principles.!It!is!transferable!and!relevant!for!other!Community*based!Arts!and!Creative!
Industries!practitioners.!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
9!A!product!is!defined!as!‘a!good,!idea,!method,!information,!object!or!service!created!as!a!result!of!a!process!and!serves!a!
need!or!satisfies!a!want.!It!has!a!combination!of!tangible!and!intangible!attributes’!(WebFinances,!2016).!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 11!
•! READ!develops!a!culture!of!learning!and!reflection!that!will!support!professional!
development!and!is!a!key!strategy!for!sustaining!long*term!collaborative!partnerships!and!
capacity!of!communities!and!practitioners.!
•! Evaluation!is!supported!by!an!extensive!‘community!of!researchers’!committed!to!action!
research,!and!therefore!is!likely!to!generate!robust!evaluation!findings.!The!lead!institution!
is!QUT’s!Creative!Industries!Faculty,!which!will!host!two!doctoral!level!research!projects!
dedicated!to!Nuclear!Futures.!
(Extract,!Alphaville!fundraising!documentation,!2012!describing!the!proposed!READ!model)!
Above,!is!the!complete!version!of!what!had!been!written!on!READ!prior!to!the!research.!While!the!
basic!model!concept!and!acronym!were!coined,!no!methodology!or!implementation!strategy!had!
been!conceived.!The!comprehensive!and!sophisticated!READ!findings!presented!in!this!dissertation!
demonstrate!significant!advancement!and!innovation!of!the!READ!model!since!its!early!inception.!!
1.6! The#scope#and#significance#of#the#research#
The!research!has!impacted!my!professional!evaluation!practice!and!established!protocols!for!
evaluation!at!my!site!of!practice.!The!findings!lay!the!foundations!for!the!further!development!of!
innovative!evaluation!practices!across!the!Nuclear!Futures!program,!as!well!as!the!future!trial!and!
application!of!the!new!model!to!other!projects!and!contexts.!For!the!field,!the!research!presents!an!
original!and!practical!contribution!to!community!arts!evaluation!practice,!delivering:!
•! an!integrated!evaluation!method!that!can!be!done!‘in!house’;!
•! a!model!of!evaluation!best!practice!that!embraces!the!ethos!of!community!arts;!
•! a!flexible!framework!for!generating!robust!data,!that!is!linked!with!management!and!
creative!arts!practice!so!that!data!can!be!fed!back,!continuously,!into!the!project!for!real*
time!impact;!
•! original!strategies!for!engagement!and!for!developing!stakeholder!ownership!of!project!
evaluation,!particularly!where!trepidation!or!resistance!to!traditional!evaluation!exists;!
•! possibilities!for!more!general!application!in!(research*led)!creative!management.!
While!the!research!makes!the!above!contributions,!it!is!acknowledged!that!the!body!of!evidence!
presented!constitutes!the!first!of!several!iterations!of!critically!analysing!the!READ!model’s!
development!and!testing,!and!thus!the!DCI!research!does!not!offer!a!complete!critique!of!the!READ!
model.!The!DCI!study!has!been!limited!to!the!first!of!many!READ!applications!for!the!Nuclear!Futures!
program,!which!is!ongoing!beyond!the!timeframe!of!the!research.!Further,!in!identifying!the!trialled!
model’s!strengths!and!constraints,!the!DCI!research!raises!several!questions!that!will!need!to!be!
addressed!for!future!development,!to!maintain!validity!and!credibility!during!more!complex!
applications!such!as!the!Nuclear!Futures!summative!program!evaluation!in!2017.!To!minimise!and!
manage!the!identified!vulnerabilities!of!the!model,!I!make!several!recommendations!in!Chapter!6!
intended!to!improve!future!evaluation!practice!(see!Section!6.2.5).!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 12!
1.7! The#DCI#journey#
1.7.1! Research#design#overview#
In!order!to!critically!analyse!my!evaluation!practice,!the!READ!application!and!the!emergent!READ!
model,!I!employed!a!robust!practice*informed!research!methodology!for!the!DCI!that!created!a!
reflective!lens!on!my!professional!practice.!As!outlined!in!Chapter!3,!the!DCI!research!design!
harnessed!forensic!reflective!practice,!a!knowledge*driven!research!strategy!involving!the!systematic!
examination!of!thinking!and!action!for!critical!enquiry!and!for!the!continuous!improvement!of!
practice!(Schon,!1983).!The!aim!of!the!reflective!enquiry!was!to!generate!original!knowledge!about!
the!READ!approach,!and!my!own!evaluation!practice,!as!well!as!identify!areas!of!innovation!for!
evaluation!practice!more!generally.!Figure!1.1!depicts!the!relationship!between!the!overarching!DCI!
research!methodology!and!the!READ!model!application/!methodology!(as!subject).!!
!
!
Figure!1.1:!The!Doctor!of!Creative!Industries!research!design!
!
!
!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 13!
1.7.2! Research#phases#
It!is!important!to!emphasis!that!the!overarching!DCI!methodology!employed!for!this!research,!
illustrated!in!Figure!1.1,!is!distinct!from!the!READ!methodology!which!was!the!subject!of!the!
research.!The!DCI!research!design!(see!Chapter!3)!created!the!primary!lens!for!the!analysis!of!the!
READ!application!for!the!purpose!of!understanding,!assessing!and!developing!the!model!in!action,!
and!for!the!continuous!improvement!of!evaluation!practice.!Forensic!reflective!practice!prior,!during!
and!following!the!READ!application!enabled!the!critical!investigation,!review!and!iterative!
modification!of!the!developing!READ!model,!and!it!was!via!this!phased!research!progression!that!the!
DCI!findings!and!conclusions!presented!in!Chapter!6!and!7!were!generated.!This!progression!for!the!
DCI!research!consisted!of!the!following!six!research!phases:!
•! A!contextual!scan!covering!the!DCI!research!topic!and!community!arts!context.!
•! Scoping!undertaken!with!Nuclear!Futures!and!industry!stakeholders.!
•! Planning!and!design!of!the!tailored!READ!methodology!(system)!for!Balaklava.!
•! READ!implementation!(data!collection!and!analysis!for!the!READ!application).!
•! Production,!dissemination!and!use!of!READ!outputs!for!the!READ!application.!
•! DCI!research!analysis!of!the!data!collected!through!the!READ!application.!
As!is!represented!in!Figure!1.2,!the!six!research!phases!completed!for!the!DCI!included!four!phases!
associated!with!the!READ!application!for!the!Balaklava!Community!Arts!Project.!While!the!progress!
of!DCI!research!phases!appears!linear!in!Figure!1.2,!they!did!not!conform!to!a!chronological!
sequence!as!the!diagram!suggests,!and!in!reality!were!iterative!research!processes.!
!
Figure!1.2:!The!six!research!phases!for!the!DCI!research!!
!
!
!
Research phases across DCI
Forensic Reflective Practice (ongoing)
Research phases related to READ application (see white boxes)
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 14!
1.7.3! Examinable#DCI#outputs##
The!DCI!research!journey!culminated!in!two!assessable!research!outputs:!
•! A!written!dissertation!(this!document)!–!detailing!the!problem!statement,!research!
methodology,!rich!descriptions!of!evaluation!practice,!key!findings!and!the!implications!of!
the!research!for!the!field.!(See!Section!1.8!for!chapter!overview.)!
•! The!READ%in%Practice!portfolio!(RIPP)!(companion!document!to!the!dissertation)!–!includes!
selected!examples!from!my!industry*based!evaluation!practice!and!READ!outputs.!
The!DCI!research!outputs!have!been!collated!to!illuminate!the!knowledge!claims!made!in!the!
research!findings!(Chapter!6!and!7).!The!curation!of!the!outputs!is!based!on!the!rationale!that!the!
evaluation!practice!undertaken!in!the!professional!context!for!the!DCI!research!constitutes!relevant!
research!deliverables.!Reference!to!and!explanation!of!the!items!included!in!the!accompanying!
practice*based!portfolio!are!detailed!throughout!the!dissertation.!For!cross*referencing!purposes,!I!
have!used!the!icon!in!the!right!margin,!to!signpost!when!original!materials!from!the!READ%in%
Practice!portfolio!(RIPP)!are!relevant!to!the!discussion.!The!article!and!page!numbers!for!RIPP!
items!are!referenced!in!the!dissertation!text.!
The!two!documents!are!submitted!as!companion!research!outputs.!The!dissertation!articulates!the!
evaluation!theory!and!developed!model!of!practice!for!the!READ!approach!and!provides!a!
descriptive!account!of!the!collective!application!at!the!site!of!practice.!The!portfolio!includes!
demonstrations!of!the!industry*based!evaluation!practice.!While!I!cross*reference!from!the!
dissertation!to!the!RIPP,!the!portfolio!is!intended!as!an!accompanying!research!and!documentation!
output.!!
1.8! Chapter#overview#
In!this!Chapter!1,!I!have!introduced!the!Doctor!of!Creative!Industries!research!as!a!scholarly!enquiry!
arising!from!my!professional!practice!as!a!program!manager!and!evaluator,!and!explained!its!dual!
position!in!the!academic!and!industry!contexts!of!community!arts.!In!establishing!the!relevance!of!
the!research!problem,!the!chapter!canvasses!several!of!the!unresolved!and!contested!aspects!of!
evaluation!practice!in!the!field,!and!relates!them!to!the!challenges!faced!at!my!site!of!practice.!The!
research!seeks!to!contribute!to!the!identified!knowledge!gaps!on!effective!evaluation!of!community!
arts,!in!particular!for!complex!creative!partnership!initiatives,!using!a!professional!practitioner!lens!
to!frame!the!research.!!
!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 15!
Central!to!the!research!enquiry!is!the!development!and!critical!review!of!a!newly!devised!
participatory!evaluative!model!that!integrates!Reflection,!Evaluation,!Analysis!and!Documentation!
(READ).!As!explained!in!the!Introduction,!the!model!has!been!designed!via!a!collective!process!for!
the!international!Nuclear!Futures!arts!and!cultural!program,!which!I!led!and!substantially!informed!
as!researcher.!The!research!centres!on!the!reflective!analysis!of!the!READ!model’s!development,!and!
performance!as!a!learning!and!appraisal!framework,!based!on!the!Nuclear!Futures!experience!of!
applying!the!model!in!practice.!In!the!introduction,!I!also!outlined!the!research!phases!of!the!DCI!and!
the!two!DCI!research!outputs!–!a!dissertation!(this!document)!and!a!portfolio!of!materials!generated!
through!the!evaluation!practice!of!the!READ!application!(the!READ%in%Practice!portfolio!–!RIPP).!%
In!Chapter!2,!I!provide!a!comprehensive!contextual!scan.!The!contextual!scan!charts!the!historical!
and!theoretical!trends!relevant!to!the!research!problem!based!on!the!wide*ranging!review!of!
scholarly!and!industry!secondary!sources!from!key!literature!areas!in!community!arts!and!
evaluation.!In!detailing!the!broader!context!of!the!READ!model!application,!I!outline!the!political!and!
cultural!traditions!defining!community!arts!practice!and!overview!the!rapidly!growing!field!of!
evaluation,!honing!in!on!program!evaluation!of!community!arts.!Of!relevance!are!the!contemporary!
participation*oriented!evaluation!approaches!(Sections!2.2.2!and!2.3.2),!where!partnerships!and!
participation!become!central!to!evaluation!practice!and!stakeholder!engagement,!empowerment!
and!capacity!building!get!embedded!in!the!evaluative!processes.!!Equally!significant!for!the!research!
topic!is!the!debate!on!cultural!value,!and!trend!for!evaluation!to!achieve!a!determination!and!
valuation!of!the!impact!of!community!arts!practice,!which!has!had!various!ramifications!for!the!field.!
In!Chapter!3,!I!present!the!overarching!DCI!research!design!and!outline!the!methodology!employed!
for!studying!the!READ!application.!In!order!to!adequately!address!the!research!problem,!a!robust!
qualitative!methodology!was!implemented!that!harnessed!forensic!reflective!practice,!and!this!
utilised!qualitative!methods!such!as!reflection,!digital!storytelling,!interviews!and!observation.!
Chapter!3!explains!the!rationale!of!the!practice*informed!research!approach!and!argues!the!benefits!
afforded!by!the!methods!selected!(Section!3.3).!I!detail!the!systematic!program!of!forensic!reflective!
practice!undertaken,!which!involved!reflection!prior,!during!and!following!the!READ!application,!as!
the!central!research!strategy!for!the!investigation.!Combined,!these!first!three!chapters!set!up!the!
research!enquiry!and!position!it!within!the!field,!ahead!of!the!following!chapters!that!cover!the!
industry*based!evaluation!practice!for!the!Balaklava!Community!Arts!Project.!
Chapter!4!provides!a!rich!descriptive!account!of!the!site!of!practice,!and!the!art*making!defining!the!
Balaklava!Community!Arts!Project.!The!chapter!opens!with!background!information!about!the!
Nuclear!Futures!initiative,!the!multifaceted!international!community!arts!program!from!which!the!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 16!
Balaklava!project!is!delivered.!The!Balaklava!project!encompassed!a!series!of!multi*arts!residencies!
driven!by!multiple!partnerships!and!documentary!art*making!activities!in!the!Wakefield!region!
(north!of!Adelaide)!in!South!Australia.!The!focus!of!the!project!was!to!engage!regional!communities!
in!the!arts!and!work!with!atomic!survivor!community!partners!to!make!story*based!artworks!
exploring!the!legacies!of!the!nuclear!age.!To!showcase!the!diversity!and!complexity!of!the!
community!arts!processes,!and!the!creative!works!produced!for!the!Balaklava!project,!I!use!creative!
documentation!in!this!chapter!to!represent!the!community!arts!practice.!!
In!Chapter!5!I!elaborate!on!my!role!as!the!READ!facilitator,!in!which!I!led!the!collective!design!and!
implementation!process!to!develop!the!READ!concept!into!a!functioning!model!for!the!Balaklava!
project.!This!was!a!dynamic!and!demanding!role,!and!in!Chapter!5!I!provide!in*depth!coverage!of!the!
evaluation!practice!undertaken!and!the!emergent!READ!methodology.!The!chapter!begins!with!
background!information!on!the!READ!model’s!inception,!and!uses!a!basic!systems!approach!to!
explain!the!prospective!model!(Section!5.1).!Six!core!system!elements!of!the!READ!concept!are!
identified,!and!each!is!discussed!in!turn!in!the!chapter.!The!six!elements!charted!are:!the!support!
structures!for!READ!evaluation!practice;!Reflection;!Evaluation;!Analysis;!Documentation;!and!the!
integration,!utilisation!and!dissemination!of!the!READ!outputs.!In!profiling!the!READ!application,!the!
implications!of!the!interpretative!stance,!participatory!ethos!and!internal!positioning!of!the!
evaluation!are!elucidated.!Arising!from!the!scoping!phase!with!stakeholders,!also!presented!in!the!
first!half!of!Chapter!5,!are!the!site*specific!needs!and!challenges!for!evaluating!the!project,!several!of!
which!resonate!with!the!trends!and!issues!identified!in!the!contextual!scan.!Issues!included!a!lack!of!
stakeholder!clarity!about!evaluation!needs,!the!presence!of!divergent!world!views,!methodological!
design!issues!and!for!some!stakeholders,!an!existing!level!of!trepidation!of!evaluation.!!
In!the!second!part!of!Chapter!5!I!detail!each!of!the!four!enquiry!strategies!of!the!developed!READ!
model!(Reflection,!Evaluation,!Analysis!and!Documentation)!to!illuminate!how!each!was!designed!
and!implemented.!This!in*depth!documentation!of!the!READ!application!defines!both!the!technical!
and!epistemological!aspects!of!the!evaluation!practice!for!the!READ!approach!that!I!conducted!for!
the!research.!Finally,!in!Section!5.5,!I!discuss!the!processes!I!employed!for!integrating,!utilising!and!
disseminating!the!READ!data!and!outputs!generated!for!the!Balaklava!project.!Ultimately,!the!
systems!investigation!of!the!READ!design!and!implementation!presented!in!Chapter!5!demonstrates!
the!operations,!functionality!and!outcomes!of!the!applied!READ!approach!during!the!Balaklava!
experience,!evidenced!by!qualitative!data!generated!and!practical!examples!from!the!READ%in%
Practice!portfolio.!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 17!
Next,!in!Chapter!6,!I!present!the!findings!arising!from!READ!application!and!explore!a!number!of!
emergent!themes!about!the!evaluation!practice.!The!DCI!research!methodology!elicited!rich!data!
about!the!READ!application!and!this!chapter!engages!with!the!practical!lessons!that!were!made!
explicit!via!the!reflective!analysis.!The!identified!themes!and!lessons!provide!insight!into!the!efficacy!
and!issues!of!the!developed!READ!model!as!an!applied!learning!and!appraisal!framework,!to!inform!
the!summative!conclusions!presented!in!Chapter!7.!Revealed!is!the!relevance!of!the!internal!
positioning!of!the!READ!facilitation,!the!model’s!versatility!as!a!modular!or!integrated!system,!and!
the!demonstrated!benefits!of!creative!documentation!and!collective!reflection.!The!findings!also!
highlight!the!implications!of!including!legacy!goals!as!a!purpose!of!evaluation!practice,!which!
resulted!in!the!long*term!public!storage!and!distribution!of!several!READ!outputs.!Chapter!6!
culminates!in!the!presentation!of!the!consolidated!READ!product!which!I!designed!via!the!DCI,!that!
articulates!the!collectively!developed!and!refined!model.!For!the!purpose!of!general!presentation,!I!
devised!an!infographic!to!visually!encapsulate!the!developed!READ!model,!as!a!communications!tool!
for!future!applications,!such!as!the!summative!evaluation!of!the!Nuclear!Futures!program!in!2017.!
In!the!final!chapter!of!the!dissertation,!Chapter!7,!I!present!my!summative!conclusions!for!the!DCI!–!
a!final!reflection!on!the!research!problem!and!findings!generated.!Based!on!the!research!findings,!I!
state!the!claims!of!relevance,!effectiveness!and!originality!for!the!READ!model!in!meeting!the!
stakeholder!evaluation!needs!at!the!site!of!practice,!and!elucidate!the!areas!of!innovation!offered!
for!evaluation!practice.!I!argue!that!READ’s!innovation!lies!in!its!deliberate!modular!and!integrated!
approach,!and!that!READ’s!diversified!learning!and!appraisal!strategies!fulfil!multiple!purposes!
relevant!for!community!arts,!including!accountability,!improvement,!broader!learning!and!instituting!
legacy.!In!defining!the!significance!of!the!research,!I!discuss!the!implications!of!the!findings!for!
evaluation!practice!–!on!my!own!practice,!at!the!site!of!practice!and!for!the!field.!In!doing!so,!I!argue!
that!the!research!makes!an!original!contribution!to!the!discourse!and!practice!of!community!arts!
evaluation!and!that!the!research!has!the!potential!to!significantly!impact!the!field.!To!close!the!
chapter,!I!outline!my!intentions!for!research!dissemination!and!directions!for!future!research.!
1.8.1! Dissertation#roadmap#
In!order!to!map!the!chapter!progression!of!the!dissertation,!as!summarised!in!Section!1.8,!I!have!
graphically!represented!the!dissertation’s!structure!and!relationship!to!the!READ%in%Practice%
portfolio!in!Figure!1.3.!As!illustrated,!the!first!three!chapters!(see!below!in!blue)!focus!on!the!context!
and!research!design!for!the!DCI.!Next,!in!Chapters!4!and!5,!I!document!the!industry*based!process!of!
developing!and!testing!the!READ!model!for!the!Nuclear!Futures!site!of!practice!(see!below!in!red).!
These!chapters!have!strong!links!with!the!accompanying!READ%in%Practice!portfolio,!and!I!cross!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 18!
reference!to!the!portfolio’s!practical!examples!throughout!Chapters!4!–!6!to!reveal!the!depth!and!
breadth!of!the!evaluation!practice!conducted.!Next,!I!provide!a!critical!analysis!of!the!READ!model!in!
action,!in!Chapter!6,!that!scrutinises!the!READ!application!based!on!the!findings!of!my!forensic!
reflective!practice!undertaken!before,!during!and!after!the!READ!trial!and!consultation!with!
stakeholders.!The!knowledge!presented!in!Chapter!6!is!then!refined!in!Chapter!7,!to!directly!answer!
the!key!research!questions!for!the!DCI.!
!
!
!
!
#
!
!
!
Figure!1.3:!Roadmap!for!dissertation!
! !
!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 19!
2! Contextual#scan#
The!contextual!scan!presented!in!Chapter!2!examines!the!historical!and!theoretical!background!
relevant!to!community!arts!evaluation!and!the!participation*oriented!approach!of!the!emergent!
READ!model.!Extensive!review!of!scholarly!and!industry!secondary!sources!(from!several!English*
speaking!continents)!has!been!undertaken,!in!the!key!literature!areas!of!community!arts!and!
evaluation,!to!ensure!the!DCI!research!builds!on!existing!knowledge,!scholarship,!and!industry!
practice.!In!order!to!navigate!the!vast!quantity!of!literature!within!these!fields,!I!begin!the!contextual!
scan!broadly!and!then!hone!in!on!the!evaluation!theory!and!methodologies!of!most!relevance!for!
community!arts,!including!an!examination!of!contemporary!debates!and!pertinent!trends.!!
First,!the!political!and!cultural!traditions!defining!community!arts!practice!are!outlined!to!chart!the!
emergence!of!the!community!partnerships!sector!within!community!arts.!It!is!within!this!domain!
that!the!Nuclear!Futures!Partnership!Initiative!(Nuclear!Futures),!my!site!of!practice,!is!positioned.!
Second,!a!theoretical!overview!of!evaluation!is!presented,!with!a!focus!on!program!evaluation.!
While!program!evaluation!is!a!distinct!subset!within!the!evaluation!field,!the!epistemologies!and!
methodologies!applied!for!the!evaluation!of!programs!and!projects!is!extremely!divergent.!
Underlying!the!dominant!approaches!and!models!for!program!evaluation!is!a!wide!spectrum!of!
competing!paradigms,!from!positivist!to!values*oriented,!which!have!significant!bearing!on!how!
evaluation!is!undertaken!and!for!what!purpose!and!outcome.!
Next,!the!dominant!trends!and!challenges!for!community!arts!evaluation!are!discussed,!with!
reference!to!the!pertinent!debates!that!set!the!backdrop!to!this!DCI!research.!Of!relevance!is!the!
contemporary!debate!on!cultural!value,!and!the!increasing!orientation!of!conventional!evaluation!
towards!the!use!of!quantitative!tools!for!impact!measurement!(such!as!cultural!metrics!and!
indicators).!Evident!is!the!extent!to!which!community!arts!evaluation!has!mirrored!broader!
evaluation!trends,!which!has!led!to!the!instrumentalisation!of!the!arts!over!the!past!30!years.!To!
frame!such!debates!in!relation!to!the!research,!I!canvass!the!political,!technical!and!philosophical!
dimensions!of!contemporary!evaluation!practice!and!examine!the!possible!ramifications!of!current!
evaluation!trends,!that!some!scholars!argue!is!highly!detrimental!to!the!field!(Goldbard,!2015).!
Finally,!a!synopsis!of!contemporary!participation*oriented!approaches!and!methods!for!community!
arts!evaluation!is!provided.!For!the!Nuclear!Futures!context,!where!partnerships!and!participation!
are!central!to!the!arts!practice,!exemplars!and!models!that!have!been!able!to!achieve!stakeholder!
engagement,!empowerment!and!capacity!building!as!part!of!the!evaluative!process!were!of!high!
relevance!for!the!collective!design!and!implementation!of!the!READ!model!at!my!site!of!practice.!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 20!
2.1! Community#arts##
In!general!terms,!the!field!of!community!arts!encapsulates!the!diverse!range!of!arts*based!
collaborative!practices!run!by,!or!with,!communities!in!partnership!with!professional!artists!and!arts!
workers!(MacDowall!et!al.,%2013;!Australia!Council!for!the!Arts,!2011).10!!The!field!emerged!out!of!
the!radical,!socially!and!politically!driven!community!arts!movement!in!the!United!Kingdom!in!the!
1960s,!and!has!since!developed!to!encompass!a!broad!range!of!practices!from!grass*roots!political!
arts!activism!(sometimes!referred!to!as!the!movement!for!cultural!democracy)11!to!more!
institutionalised!activities!such!as!community!arts!and!cultural!development!(CACD)!(Van!Erven,!
2001;!Hawkins,!1993).!!
Community!arts!is!widely!recognised!as!a!reflexive!approach!for!community!cultural!development!
and!a!resource!for!previously!unrepresented!or!marginalised!groups!to!express!cultural!self*
determination!(Hawkins,!1993).+Leading!American!community!arts!scholar!Arlene!Goldbard!(1993,!
p.23)!identifies!two!uniting!principles!underlying!community!arts!practice:!
The!first!essential!point!of!definition!is!that!community!arts!practice!is!based!on!the!belief!that!
cultural!meaning,!expression!and!creativity!reside!within!a!community,!that!the!community!
artist's!task!is!to!assist!people!in!freeing!their!imaginations!and!giving!form!to!their!
creativity…The!second!essential!point!of!definition!is!that!collaboration!between!artists!and!
others!is!central!and!necessary!to!the!practice!of!community!arts.!
While!consensus!exists!in!the!literature!that!the!prioritisation!of!participatory!processes!of!art!
creation!are!central!to!community!arts!practice,!Badham!points!out!however,!that!‘(d)espite!a!more!
than!thirty*year!history!of!community!arts!in!Australia,!practitioners!engaging!in!transformative!
practice,!and!government!support!for!projects,!there!is!no!definitive!understanding!or!
comprehensive!theory’!within!the!field!(Badham,!2010,!p.85*6).!!!
2.1.1! Community#arts#and#cultural#development#(CACD)#
In!the!Australian!context,!the!key!shifts!within!community!arts!over!the!past!four!decades!have!been!
strongly!tied!to!the!graduations!in!the!rhetoric,!funding!priorities!and!policy!of!the!Australia!Council!
for!the!Arts,!Australia’s!principal!arts!funding!and!advisory!body!(Hadley!&!Gattenhof,%2012;!
Mulligan!&!Smith,!2009;!Van!Erven,!2001;!Hawkins,!1993).!In!particular,!community!arts!practice!has!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
10!Terminology!in!the!literature!varies,!and!alternative!terms!for!‘community!arts’!are!widely!applied,!such!as!‘community*
based!arts’!and!‘socially*engaged!arts’.!!In!some!cases,!variations!reflect!international!differences!in!rhetoric,!or!
practitioner!preferences.!For!the!purpose!of!this!scan,!such!terms!have!been!considered!to!be!comparable!at!the!general!
level!of!analysis.!While!it!is!beyond!the!scope!of!this!review,!it!is!important!to!note!however!that!subtle!differences!and!
nuances!do!exist!between!the!alternative!terms!when!analysed!in!detail.!11!Sourced!in!Lippard!(1984,!p.341)!with!reference!to!Arlene!Goldbard!and!Don!Adams,!‘From!the!Ground!Up:!Cultural!
Democracy!as!a!National!Movement’,!Upfront,!no.!8!(Winter!1983*4):!6.!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 21!
been!significantly!shaped!by!government*led!CACD!objectives,!formerly!referred!to!as!Community!
Cultural!Development!(CCD).!From!the!mid!1970s,!CCD!became!a!formal!part!of!the!Australia!Council!
for!the!Arts!agenda,!established!to!support!community!arts!projects!working!to!achieve!cultural!
development!and/or!innovative!artistic!outcomes!(Australia!Council!for!the!Arts,!2011).!!
Practitioner!approaches!to!CCD!(and!CACD)!vary!significantly!and!have!been!influenced!over!time!by!
external!factors!such!as!technological!developments,!government!funding!priorities!and!changing!
community!expectations!(MacDowall!et!al.,!2013;!Goldbard,!2006).!The!approaches!and!methods!
documented!in!the!literature!for!community!engagement!and!art*making!practice!in!this!sector!are!
extremely!diverse.!Mulligan!and!Smith!(2009)!define!three!generations!of!CCD!practitioners!shaping!
community!arts!in!Australia:!
There!are!probably!three!generations!of!CCD!practitioners!who!are!still!active:!those!who!
began!their!work!in!the!1970s!or!who!are!heavily!influenced!by!social!and!political!movement!
theory!that!emerged!in!the!1970s;!those!who!were!able!to!take!advantage!of!new!
professional!opportunities!for!CCD!practice!that!emerged!in!the!latter!part!of!the!1980s!and!
early!1990s;!and!those!who!have!taken!advantage!of!new!media!and!new!technologies!that!
have!emerged!within!the!last!decade.!(Mulligan!&!Smith,!2009,!p.12)!
One!recent!paradigm!shift,!led!by!the!Australia!Council!for!the!Arts,!gave!rise!to!‘community!
partnerships’!emerging!as!another!key!sector!within!the!field.12!With!a!focus!on!creative!enterprise,!
sustainability!and!revitalisation!of!the!Arts,!the!CCD!priorities!of!the!1980s!(to!support!socially!
marginal!communities)!were!replaced!by!the!objective!to!move!community!arts!from!solely!
traditional!government!subsidised!funding!models!to!increasingly!shared!partnership!investment!
between!government,!business!and!communities!(Hadley!&!Gattenhof,!2012).!
2.1.2! The#rise#of#community#partnerships#
Following!a!major!scoping!report!in!2006!(see!Dunn,!2006),!the!Australia!Council!for!the!Arts!in!2010!
released!a!three*year!strategic!plan!for!the!community!partnerships!sector!to!better!support!artistic!
practice!and!cultural!engagement!with!a!diversity!of!communities!(Australia!Council!for!the!Arts,!
2011).13!The!rationale!for!this!was!that!‘partnerships!between!the!arts!and!non*arts!sectors!should!
advance!positive!outcomes!for!building!culturally!vibrant!communities!and!provide!working!
opportunities!for!artists’!(Dunn,!2006,!p.19).!Commencing!in!2008,!one!of!the!key!instruments!
employed!by!the!Australia!Council!for!the!Arts!for!advancing!community!arts!and!cultural!
development!has!been!the!Creative!Communities!Partnership!Initiative!(CCPI)!grant!scheme.!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
12!The!Australia!Council!for!the!Arts!uses!both!‘creative!partnerships’!and!‘community!partnerships’!as!interchangeable!
terms!describing!this!agenda!(Hadley!&!Gattenhof,%2012,!p.8).!13!Community!refers!to!‘any!group!of!people!that!identify!with!each!other!through!commonalities!such!as!geographical!
location,!shared!cultural!heritage,!age,!professional,!social!or!recreational!interests’!(Dunn,!2006,!p.39).!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 22!
The!CCPI!operated!as!a!competitive!funding!program!to!provide!significant!and!sustained!funding!
over!a!maximum!of!a!three*year!period!to!partnership!projects!that!aimed!to!deliver!increased!
participation!in!arts!and!cultural!activities,!encourage!innovation,!and!enhance!community!wellbeing!
in!a!number!of!government*led!priority!areas.!The!premise!underpinning!the!CCPI!is!that!deliberate!
cooperative!partnerships!between!arts!and!non*arts!organisations!have!the!potential!to!produce!a!
range!of!benefits!across!the!communities!they!work!with.!!Badham!(2010.!p.93)!elaborates:!
The!new!focus!on!community!partnerships!reflects!a!move!away!from!short*term!‘one*off’!
projects.!There!are!now!new!collaboration!models,!inter*sectoral!partnerships!including!non*
arts!organisations,!new!entrepreneurial!models!with!diversified!and!self!generated!revenue!
sources,!and!longer!term!projects!embedded!in!community!in!order!to!build!local!capacity.!!!
Though!still!aligned!with!many!of!the!principles!of!the!earlier!community!arts!movement,!the!shift!
towards!community!partnerships!reflected!the!influence!of!broader!trends!in!government!policy!and!
industry!practice!during!the!2000s.!In!the!community!partnerships!space,!arts!projects!that!deliver!
and!substantiate!an!‘exchange!of!benefits’!between!the!organisations,!communities!and!individuals!
working!together!to!deliver!positive!social!and!cultural!impacts!are!prioritised!and!funded!(Australia!
Business!Arts!Foundation!&!Australia!Council!for!the!Arts,!2010,!p.1;!MacDowall!et!al.,%2013).!!
While!the!rise!of!community!partnerships!has!been!a!prominent!trend!in!community!arts!funding!
over!the!past!decade,!future!trends!are!less!certain.!In!2014,!the!Australia!Council!for!the!Arts!
announced!its!most!significant!changes!in!funding!in!the!organisation’s!40*year!history,!introducing!a!
streamlined!grants!scheme!and!revised!peer!assessment!process.!The!changes!aimed!to!deliver!a!
simpler,!more!transparent,!and!accessible!grant!application!and!approval!process!for!artists!and!arts!
organisations*!!and!minimised!their!previous!140!categories!to!five!(Australia!Council!for!the!Arts,!
2015).!The!new!funding!model!was!the!result!of!a!two*year!reform!and!consultation!process,!and!
represents!the!most!recent!major!shift!for!how!the!Australia!Council!invests!in!Australian!arts.!While!
community!arts!and!cultural!development!(including!community!partnerships)!remains!a!key!
strategic!goal!for!the!organisation,!previously!dedicated!funding!streams!such!as!the!CCPI!have!been!
disbanded.!At!the!time!of!writing,!the!CACD!and!community!partnerships!sectors!also!face!new!
pressures!(and!uncertainty)!from!the!potential!funding!impacts!of!the!2015*16!Australian!
Government!budget!measures!to!reduce!Australia!Council’s!overall!funding.!
! !
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 23!
2.2! Evaluation#
Determining!the!benefits!of!creative!partnership!exchanges!and!the!impact!of!community!arts!
practice!(cultural,!social,!aesthetic,!economic,!environmental)!demands!that!the!processes,!outputs!
and/or!outcomes!be!articulated!and!valued.!With!no!national!standard!or!peer!agreed!framework!
catering!to!the!breadth!and!complexity!of!community!arts,!practitioners!must!traverse!multiple!
philosophical!and!technical!challenges!when!evaluating!their!processes!and!creative!works!
(MacDowall!et!al.,!2013;!Goldbard,!2006).!To!understand!the!trends!and!challenges!affecting!
community!arts!evaluation,!it!is!necessary!to!first!examine!the!broader!developments!in!application!
and!theory!of!evaluation!practice!more!generally.!
The!Australasian!Evaluation!Society!defines!evaluation!as!encompassing!‘the!systematic!collection!
and!analysis!of!information!to!make!judgments,!usually!about!the!effectiveness,!efficiency!and/or!
appropriateness!of!an!activity’!(Australasian!Evaluation!Society,!2010,!p.3).!While!definitions!vary!
within!the!literature,!most!encapsulate!a!cyclic!process!with!distinct!stages!of!planning!and!decision*
making;!methodology!design;!data!collection;!organisation!of!findings;!evaluative!reasoning!and!
assessment!presentation;!and!reporting!(Davidson,!2005;!Clarke,!1999;!Matarasso,!1996;!Wilde!&!
Sockey,!1995).!Some!scholars!in!evaluation,!such!as!Scriven!(2012)!and!Davidson!(2005),!argue!that!
central!to!the!evaluator’s!work!is!the!systematic!determination!of!merit!or!worth.!
Evidence!of!formal!evaluation!practice!dates!back!over!two!thousand!years!(Fitzpatrick!et!al.,%2012).!
Despite!this!long!history,!the!majority!of!theoretical!developments!and!innovations!in!application!
have!occurred!in!the!last!sixty!years!(Scriven,!2012;!Sharp,!2003).!Influential!in!the!early!scholarship!
in!evaluation!were!scholars!such!as!Scriven!(1967),!Stake!(1967)!and!Weiss!(1972).!Since!then,!
scholarship!in!evaluation!has!expanded!significantly,!amounting!to!a!vast!and!sophisticated!body!of!
literature.!Considered!and!debated!within!the!literature!are!divergent!epistemologies!of!practice!
relating!to!evaluation!paradigms,!design,!methods,!and!strategies!for!stakeholder!engagement!and!
partnership!brokerage!(Fitzpatrick!et!al.,%2012;!Hadley!&!Gattenhof,!2012;!Friedman,!2008;!Owen!&!
Rogers,!1999).!The!breadth!of!evaluation!discourse!is!evident!in!compendiums!such!as!Program%
evaluation:%Alternative%approaches%and%practical%guidelines!(Fitzpatrick!et!al.,%2012),!Handbook%of%
Evaluation!(Shaw,!I.,!Shaw,!I.G.R,!Greene,!&!Mark,!2006),!and!the!Encyclopedia%of%Evaluation!
(Mathison,!2005).!
Historical!accounts!attribute!the!widespread!application!of!evaluation!across!disciplines!and!
countries!to!an!increased!demand!by!policymakers,!managers,!consumers!and!the!general!public!for!
improved!accountability,!efficacy,!and!understanding!of!social!problems!(Fitzpatrick!et!al.,%2012;!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 24!
Mark,!Henry,!&!Julnes,!2000).!Since!the!1990s,!growth!in!the!field!has!also!been!linked!with!trends!
such!as!the!internationalisation!and!professionalisation!of!evaluation!(Fitzpatrick!et!al.,%2012;!House,!
1993).!Documented!in!the!literature!are!increased!numbers!of!professional!evaluation!organisations!
(growing!from!a!handful!in!the!1980s!to!being!established!in!approximately!100!countries!in!2011)!
and!published!scholarly!and!industry!work.!Schwandt!(2009)!cites!research!by!Dahler*Larsen!
estimating!10,000!individuals!at!least!nominally!self*identify!as!evaluators!and!belong!to!one!or!
more!of!the!many!national!and!international!evaluation!associations!around!the!world.!Online,!!a!
diversity!of!evaluation*specific!networks,!resource!guides!and!blogs!are!available!(Scriven,!2012).14!
2.2.1! Program#evaluation#
Within!the!field!of!evaluation,!the!domain!of!relevance!for!this!research!is!program!evaluation.!A!
program!is!defined!as!a!‘group!of!planned!activities!that!contribute!to!a!common!strategic!objective’!
(Australian!Government,!2000,!p.4).!Program!evaluation!is!used!to!ascertain!program!impact,!worth,!
efficacy!and!efficiency!for!the!purpose!of!continuous!improvement,!decision*making,!accountability,!
partnership!development,!and!program!acquittal!(Fitzpatrick!et!al.,!2012;!Davidson,!2005;!Wilde!&!
Sockey,!1995).!!
Program!evaluation!has!its!origins!in!Education!in!the!1950s,!linked!with!the!rise!of!the!standards*
based!education!movement!and!social!experimentation!and!research!(Fitzpatrick!et!al.,%2012).!By!the!
end!of!the!1950s!evaluation!began!to!flourish,!linked!with!the!increase!in!social!research,!and!
systematic!approaches!were!being!developed!using!scientific!experimental!methods!to!rationalise!
the!evaluative!process.!From!the!1950s!to!the!1970s!in!countries!such!as!the!United!States,!Sweden!
and!Germany,!social!inquiry!laid!the!foundations!for!the!‘first!wave’!of!modern!program!evaluation.!
Evaluation!practice!in!this!context!was!linked!to!program!improvement!and!social!experimentation!
to!test!new!interventions!(Fitzpatrick!et!al.,%2012,!p.!50;!Rist,!1999).!In!other!countries,!such!as!the!
United!Kingdom,!a!second!wave!of!evaluation!is!documented!as!being!driven!by!the!rise!in!demand!
for!public!sector!accountability!and!measuring!program!efficacy!(Fitzpatrick!et!al.,%2012).!In!the!
Australian!context,!program!evaluation!(such!as!in!health!and!community!development)!became!a!
major!government!commitment!during!the!1970s!and!has!since!remained!a!priority!(Sharp,!2003).!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
14!Some!examples!of!online!resources/hubs!include!American!Evaluation!Association!
365!|!A!Tip*a*Day!by!and!for!evaluators!http://aea365.org/!;!IDEAS!International!Development!Evaluation!Association!
http://ideas*global.org!;!BetterEvaluation!website!http://betterevaluation.org/!;!Community!Sustainability!Toolbox!
http://evaluationtoolbox.net.au!;!Does!Your!Project!Make!a!Difference?!
http://www.environment.nsw.gov.au/resources/community/projecteval04110.pdf!.!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 25!
Christie!and!Alkin!(2008)!argue!that!such!drives!for!increased!social!inquiry!and!fiscal!control/!
accountability,!constitute!the!central!roots!of!modern!evaluation.!!
The!epistemological!undercurrents!of!program!evaluation!encompass!positivist,!methods*focused!
approaches!(Weis,!1972;!Campbell,!Stanley,!&!Gage,!1963),!to!post*positivist!paradigms!prioritising!
realist!evaluation!(Davidson,!2005;!Scriven,!2003a),!to!models!espousing!the!philosophy!of!
subjectivity!where!it!is!argued!that!there!is!no!one!objective!truth!(Greene,!2005;!Guba!&!Lincoln,!
1985).!15!!These!theoretical!domains!are!described!and!classified!in!different!ways!by!scholars!in!the!
literature.!For!example,!in!theorising!the!different!branches!of!evaluation!Christie!and!Alkin!(2008)!
map!theorists!across!their!tree!of!evaluation!in!three!predominant!orientations!*!valuing,!methods!
and!utility!(the!branches).!Guba!and!Lincoln!(2004)!classify!the!theoretical!developments!by!defining!
distinct!‘generations’!of!program!evaluation.!The!most!recent!is!fourth!generation!evaluation!(FGE)!
based!on!an!epistemology!of!constructivism!based!on!a!radical!critique!of!the!positivist!foundation!
of!traditional!program!evaluation!(Lay!&!Papadopoulos,!2007;!Guba!&!Lincoln,!2004).!
The!wide*ranging!theoretical!orientations,!based!on!the!differing!philosophical!and!ideological!
stances,!have!a!profound!impact!on!the!methodological!preferences!and!guiding!approach!adopted!
by!program!evaluators.!They!drive!what!is!prioritised!and!practiced!at!the!site!of!practice.!Fitzpatrick!
et!al.,!(2012)!for!example,!identifies!four!dominant!evaluation!approaches:!!
•! expertise!and!consumer*oriented!(performance,!cost*analysis)!
•! program*oriented!(logic!models,!theory*based!evaluation,!goal*free)!
•! decision*oriented!(impact,!results*based,!utility!focussed!evaluation)!
•! participant*oriented!(stakeholder,!empowerment,!participatory!evaluation).!
Approaches!to!program!evaluation!are!nuanced,!and!the!emergent!models!of!practice!become!
specialised,!contextualised!and!politicised!based!on!the!theoretical!concepts!guiding!the!approach!
(Edenloff,!2011;!Davidson,!2005).!!
! !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
15!Some!examples!of!the!interpretative!positions!in!evaluation!include!postpositivism,!constructivism,!transformative!and!
pragmatist!paradigms,!with!shifts!mirroring!trends!and!debates!in!social!research!including!both!quantitative!and!
qualitative!stances!(Fitzpatrick!et!al.,!2012,!p.124;!for!analysis!of!social!research!trends!see!Denzin!&!Lincoln,!2011;!Cresswell,!&!Clark,!2007;!Schwandt,!2000;!Alvesson!&!Skoldberg,!1999;!Becker,!1996;!Babbie,!1992).!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 26!
Alkin!(2004)!refers!to!the!plethora!of!evaluation!approaches!and!models!shaping!evaluation!practice!
since!the!1950s!as!prescriptive!theories.!Fitzpatrick!et!al.!(2012,!p.112)!argue!‘(t)he!various!
approaches,!or!theories,!proposed!by!evaluators!make!up!the!content!of!the!field!of!evaluation’!and!
are!testament!to!the!dynamism!involved!in!designing!and!conducting!evaluation.!Consider!for!
example:!Utilisation*Focused!evaluation!(Patton,!2008);!Naturalistic!evaluation!(Guba!&!Lincoln,!
1981);!Empowerment!evaluation!(Fetterman!&!Wandersman,!2005);!Participatory!evaluation!
(Cousins!&!Whitmore,!1998);!and!Feminist!evaluation,!with!its!focus!on!the!gender!inequalities!that!
lead!to!social!injustice!(Hood,!2002;!Sielbeck*Bowen,!Brisolara,!Seigart,!Tischler,!&!Whitmore,!2002).!
Practitioner!guides!and!frameworks!include!the!Key!Evaluation!Checklist!(Scriven,!1991);!
BetterEvaluation!Rainbow!Framework!(RMIT!&!partners*!website);!Real/!Actionable!Evaluation!
(Davidson,!2010);!and!systems!approach!to!program!evaluation!(Darabi,!2002).!!
Program!evaluation!has!also!been!demarcated!and!theorised!in!the!literature!on!the!basis!of!the!
timing!of!implementation!in!relation!to!the!program!being!evaluated.!Scriven!(1967)!was!the!first!to!
identify!that!depending!on!when!an!evaluation!is!undertaken!in!a!program!or!project!cycle!different!
evaluative!functions!may!be!achieved.!Irrespective!of!the!type!of!approach!adopted!or!methods!
employed,!evaluation!may!be!defined!as!being!(or!a!mix!of):!preformative!(prior!to!commencement);!
formative!(during!the!early!stages!of!implementation);!developmental!(ongoing!or!for!uncertain!
and/or!innovation*oriented!contexts);!summative!(once!activity!is!completed);!or!ascriptive!(for!the!
sake!of!the!knowledge!gained)!(Scriven!2012;!Davidson,!2005).!!
2.2.2! ParticipationSoriented#approaches#to#program#evaluation#
Within!the!domain!of!program!evaluation,!it!is!the!participation*oriented!(or!participatory)!
approaches!that!best!align!with!the!CACD!principles!governing!my!site!of!practice,!which!include!
participation,!representation,!democracy,!and!inclusion!(common!attributes!of!a!community!arts!
ethos).!!Participation*oriented!evaluation!is!defined!by!the!level!of!engagement!and!contribution!by!
participants!and!partners!in!the!process,!and!encompasses!a!wide!spectrum!of!applications.!In!the!
literature,!there!are!many!interrelated!models!described!as!participatory!approaches!such!as!
empowerment,!stakeholder*oriented,!action,!and!transformative!evaluation.!What!unites!these!
approaches!is!the!participatory!spirit!of!how!evaluation!is!designed!and!undertaken.!!
Butterfoss,!Francisco,!and!Capwell!(2001,!p.115)!argue!that!evaluation!is!participatory!when!
‘evaluators!collaborate!in!some!way!with!individuals,!groups,!or!communities!who!have!a!stake!in!
the!program!or!project!that!is!being!evaluated’.!To!varying!degrees,!partners!get!involved!or!
collaborate!in!the!formulation!of!evaluation!questions,!goals,!approaches,!as!well!as!with!the!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 27!
delivery!and!implementation.!Evaluators!actively!draw!on!the!knowledge,!skills,!perspectives!or!
participation!of!partners!and!stakeholders!during!the!evaluation!process.16!The!parameters!for!
participatory!practice!are!vast!and!flexible.!
Over!the!past!forty!years,!the!orientation!towards!stakeholder!and!partner!participation!in!
evaluation!has!developed!and!expanded!significantly.!The!origins!of!the!genre!date!back!to!the!
1960s,!when!reaction!to!the!then!dominant!social!science!research!methods!applied!in!the!field!of!
Education,!catalysed!the!turn!to!participation!(Fitzpatrick!et!al.,!2012).!Robert!Stake!(1967)!is!cited!as!
the!first!theorist!to!lay!claim!to!a!participation*oriented!methodology,!when!he!introduced!a!suite!of!
responsive,!qualitative!and!pluralist!approaches!in!his!work!on!Education!evaluation!(Fitzpatrick!et!
al.,!2012).!Another!Influential!era!was!in!the!1980s,!when!Guba!and!Lincoln’s!(1985)!naturalistic!
approach!asserted!a!move!from!quantitative!to!qualitative!methods!such!as!interviews,!observations!
and!case!studies.!Underpinning!the!formative!development!of!participation*oriented!approaches,!
was!the!emergent!position!that!evaluation!needed!to!extend!beyond!measurement,!description!and!
judgment*!to!develop!methodologies!that!cater!for!the!multiple!realities!and!stakeholder!views!
needed!to!produce!authentic!and!representative!findings!(Guijt,!Brouwers,!Kusters,!Prins,!&!
Zeynalova,!2011).!!
Prominent!movements!within!the!participatory!approaches!since!the!1980s!include!Practical!
Participatory!Evaluation!(Cousins!&!Earl,!1995),!Developmental!Evaluation!(Patton,!1994,!2010),!
Empowerment!Evaluation!(Fetterman!&!Wandersman,!2005),!and!Deliberative!Democratic!
Evaluation!(House!&!Howe,!2000).!The!differences!between!participatory!approaches!can!be!defined!
by!applied!elements!such!as!the!control!of!the!evaluation!process!offered!to!stakeholders!via!
participant!ownership,!stakeholder!selection!processes,!depth!of!participation,!and!the!role!played!
by!the!professional!evaluator!in!designing!and!implementing!evaluation!(Fitzpatrick!et!al.,!2012,!
Moreau!&!Cousins,!2012;!Dunst,!Trivette,!&!LaPointe,!1992).!While!the!breadth!of!participatory!
rhetoric!and!application!in!the!field!is!vast,!the!quality,!impact!and!authenticity!of!participation*
oriented!evaluation!practice!varies.!Echoed!across!the!literature!is!the!cautionary!reminder:!
‘Collaboration!is!not!a!guarantee!of!fairness!any!more!than!quantitative!methods!were!once!thought!
to!ensure!objectivity’!(Sielbeck*Bowen!et!al.,!2002,!p.6).!
! !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
16!Related!theories!of!participatory!practice!and!participatory!research!more!broadly!are!also!relevant!(see!Wilcox,!1996;!
Wadsworth,!1998;!Lennie,!2006;!Jackson!et!al.,!2003)!and!community!of!practice!(Lave!&!Wenger,!1991;!Wenger,!
McDermott,!&!Snyder.,!2002;!Hildreth!&!Kimble,!2004;!Cox,!2005).!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 28!
2.3! Community#arts#evaluation#
While!evaluation!practice!in!community!arts!has!been!influenced!by!broader!trends!for!program!
evaluation!in!Australia!during!the!1970s!(such!as!in!health!and!community!development),!it!was!not!
until!the!1990s!that!evaluation!practice!became!prominent!within!the!community!arts!field!
(MacDowall,!Badham,!Blomkamp,!&!Dunphy,!2015).!Prior!to!this!time,!during!the!1970s!and!1980s,!
community!arts!programs!did!not!‘depend!on!detailed!post!hoc!analysis’!and!were!delivered!
‘without!any!particular!managerial,!administrative!or!bureaucratic!pressure!to!provide!facts,!figures!
and!information!after!the!activity’!(Hadley!&!Gattenhof,%2012,!p.5).!The!rise!of!evaluation!practice!in!
community!arts!reflects!a!paradigm!shift!within!the!field,!from!the!mid!1990s,!where!managerial!and!
entrepreneurial!approaches!were!increasingly!integrated!into!programs!and!practice!(MacDowall!et!
al.,!2013).!!
The!proliferation!of!evaluation!in!the!arts!is!evidenced!by!an!increase!in:!funder!stipulation!for!
evaluation;!arts*related!evaluation!conferences,!professional!cohorts,!and!associations;!evaluation!
consultancy;!and!technical!‘How!to’!guides,!tool!boxes,!training!and!resources!aimed!at!developing!
professional!knowledge,!improving!capacity!and!increasing!evaluation!uptake!by!CACD!practitioners!
(Mulligan!&!Smith,!2009;!Jackson,!Harranz,!&!Kabwasa*Green,!2003;!Owen!&!Rogers,!1999;!Williams,!
1997;!Hawe,!Degeling,!&!Hall,!1990).!While!evaluation!in!this!field!is!burgeoning,!the!role!and!
relevance!of!formal!evaluation!in!community!arts!remains!contentious!and!is!the!subject!of!much!
scholarly!and!practitioner!debate.!!Central!are!philosophical!disputes!about!the!notion!of!cultural!
value,!that!is,!how!the!value!and!impact!of!community!arts!is!conceptualised,!defined!and!measured!
(if!at!all).!Related,!is!a!technical!debate!about!the!tools!for!evaluation!and!measurement!*!those!
methodological!approaches!that!can!produce!credible,!appropriate!and!reliable!means!to!analyse,!
evaluate!and!report!on!the!outcomes!and!impact!community!arts!(MacDowall!et!al.,!2015).!!
The!next!section!explores!the!trends!in!community!arts!evaluation!that!have!been!influential!for!this!
research,!particularly!the!epistemological!and!technical!dimensions!of!evaluation!practice!in!this!
context.!In!the!community!partnership!domain!of!community!arts,!where!creative!initiatives!involve!
multiple!and!culturally!diverse!partners,!establishing!common!ground!for!notions!of!cultural!value,!
arts!impact!and!evaluation!practice!presents!challenges.!For!example,!MacDowall!(2015,!p.4)!states!
that!‘increasing!partnerships!between!arts!and!non*arts!agencies!can!mean!differing!frameworks!for!
evaluation,!with!divergent!assumptions!about!success!and!measurement’.!The!following!discussion!
provides!the!contextual!backdrop!for!later!discussions!in!the!dissertation,!on!the!conceptualisations!
and!evaluation!needs!identified!at!the!Nuclear!Futures!site!of!practice!that!have!informed!the!design!
and!development!of!the!READ!model!(see!Chapters!5!and!6).!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 29!
2.3.1! Trends#in#community#arts#evaluation#
Arguably!one!of!the!most!dominant!and!influential!trends!identified!in!the!community!arts!literature!
of!relevance!to!this!research!is!the!instrumentalisation!of!the!arts!over!the!past!40!years.!MacDowall!
et!al.!(2013,!p.4)!describes!the!trend!as!being!‘signalled!by!a!range!of!social!uses!for!art,!and!the!
interest!from!non*art!agencies!and!government!departments!across!all!public!policy!areas,!from!
health!and!urban!renewal,!to!crime!prevention!and!anti*discrimination’.!Resulting,!is!an!increased!
focus!and!resource!allocation!to!the!articulation!and!measurement!of!impact!of!the!social,!cultural,!
economic!and!environmental!benefits!generated!by!community!arts!practice!and!activities!
(Goldbard,!2008,!Mills!&!Brown,!2004;!Keating,!2002;!Matarasso,!1997,!1996).!As!well!as!framing!
contemporary!policy,!arts!practice,!and!partnership!development!in!the!field,!the!era!of!
instrumentalisation!has!influenced!evaluation!practice!(Belfiore,!2014;!Hadley!&!Gattenhof,%2012).!!
Linked!to!this!is!the!more!recent!global!movement!towards!cultural!measurement,!since!the!late!
twentieth!century,!evidenced!by!the!growing!interest!by!governments!worldwide!to!establish!
frameworks!that!quantify!cultural!value!and!progress!using!tools!such!as!indicators!and!metrics!
(MacDowall!et!al.,!2015).!These!trends!fit!within!the!broader!movement!of!global!progress!
measurement,!of!which!culture!is!now!considered!to!be!key!domain!(Woolcock!&!Davern,!2015).!
MacDowall!(2015,!p.2)!defines!cultural!measurement!as!covering!‘a!range!of!systems!for!measuring!
aspects!of!culture,!from!quantified!cultural!indicators!that!lend!themselves!to!comparison!and!
analysis!over!time,!to!qualitative!measures!of!cultural!value!and!techniques!for!the!evaluation!of!arts!
and!cultural!programmes’.!Such!practices!have!been!adopted!by!countries!worldwide,!including!the!
United!Kingdom!and!the!United!States!of!America,!and!are!gaining!increasing!traction!in!Australia!
(Badham,!2015;!Blomkamp,!2015;!Goldbard,!2015).!One!Australian!example!of!the!application!of!
cultural!metrics!is!the!Community!Indicators!Victoria!(CIV)!framework!arising!from!the!Victorian!
Community!Indicators!Project!commissioned!by!VicHealth!in!2005*6,!which!is!used!as!a!state*wide!
system!for!measuring!community!well*being!(MacDowall,!2015;!Woolcock!&!Davern,!2015).17!!
The!interest!in!cultural!measurement!aligns!with!historical!trends!in!evaluation!and!public!policy,!
driven!by!the!demand!for!evidence*based!policy!and!management!by!governments!as!a!means!for!
improving!the!allocation!and!accountability!of!public!funding!and!expenditure!(MacDowall!et!al.,!
2015;!Christie!&!Alkin,!2008).!MacDowall!(2015,!p.1)!asserts!that!‘culture!and!cultural!development!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
17!This!framework!utilises!statics!about!arts!participation!as!a!cultural!indicator!in!the!context!of!community!wellbeing!and!
local!government,!where!culture!is!one!of!five!key!domains.!While!the!cultural!indicators!included!in!the!framework!
constitute!only!a!small!part!of!the!framework,!the!inclusion!of!such!metrics!in!this!type!of!scheme!is!significant!in!the!
Australian!context!(MacDowall,!2015;!Woolcock!&!Davern,!2015).!Accessible!at:!http://www.communityindicators.net.au!!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 30!
have!become!internationally!recognised!as!important!dimensions!of!contemporary!governance!and!
public!policy’.!In!the!arts,!cultural!measurement!is!being!established!for!a!number!of!purposes,!
including!to!quantify!the!impact!of!arts!and!cultural!policies,!activities,!and!organisations;!to!monitor!
progress!or!cultural!vibrancy!of!communities!and!nations!across!time;!and!to!calculate!the!long*term!
and/or!cross*sectoral!benefits!and/or!costs!being!generated!through!investment!in!the!arts!(related!
to!community!development,!economic,!social!well*being,!health,!and!sustainability!benefits)!
(MacDowall,!2015).!In!order!to!undertake!these!types!of!measurement,!criteria!for!cultural!value!
and!impact!need!to!be!determined,!and!this!remains!a!complex!and!controversial!area!of!evaluation!
in!the!arts.!
Traditionally,!the!notion!of!cultural!value!has!been!contested!through!a!binary!lens!where!
discussions!centred!on!the!value!of!the!arts!as!being!intrinsic!(often!linked!with!inherent!and!
aesthetic!worth)!or!instrumental!(often!associated!with!economic!or!social!outcomes)!(Goldbard,!
2008;!Geursen!&!Rentschler,!2003).!This!has!shifted!in!recent!years!and!the!contemporary!discourse!
on!cultural!value!encompasses!much!wider!terms!of!reference.!Blomkamp!(2015)!for!example,!
proposes!a!spectrum!of!four!dimensions!of!cultural!value:!individual,!collective,!commodity!and!
process,!which!can!be!measured!via!cultural!indicators!for!‘economic,!human!rights,!ecologic,!
symbolic,!artistic!and!social!policy’!impact/worth!(Blomkamp,!2015!p.12).!Dunphy’s!(2015)!‘holistic’!
model!for!arts!engagement!is!another!example!of!a!broadened!approach,!drawing!on!frameworks!of!
community!wellbeing!and!sustainability!across!‘personal!wellbeing,!cultural,!social,!economic,!civic!
and!ecological’!domains!(Dunphy,!2015,!p.243).!!
The!trend!towards!the!categorisation!and!quantification!of!cultural!value!can!be!attributed!to!both!
practical!and!symbolic!reasons,!including!that!metrics!hold!the!promise!of!addressing!reoccurring!
issues!within!arts!evaluation!by!turning!what!was!once!intangible,!into!quantitative,!evidence*based!
knowledge!(Blomkamp,!2015;!MacDowall,!2015).!While!international!benchmarks!and!cultural!
indicators!potentially!allow!for!comparative!analysis!and!progress!measurement,!questions!are!
raised!in!the!literature!about!the!efficacy,!politics,!suitability,!and!the!long*term!ramifications!of!
cultural!measurement!in!the!community!arts!context!(MacDowall!et!al.,!2015;!Belfiore,!2014;!
Goldbard,!2008).!Some!scholars!argue,!for!example,!that!the!pursuit!of!cultural!metrics!and!
valuation!is!ineffectual,!and!that!serious!technical!and!methodological!questions!remain,!as!well!as!
persistent!questions!on!the!reliability!of!cultural!data!(Goldbard,!2015;!MacDowall,!2015).!Others!
assert!a!reliance!on!indicators!can!create!limitations!to!focus!on!what!can!be!measured,!as!opposed!
to!what!should!be!measured!(Blomkamp,!2015).!MacDowall!(2015.!p.1)!further!cautions!that!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 31!
systems!of!measurement!can!‘create!rather!than!capture!cultural!value’!and!often!neglects!the!
context!of!the!measurement.!
In!the!broader!discourse!framing!community!arts!evaluation,!there!are!also!ethical,!philosophical!
and!political!questions!raised!on!the!legitimacy!of!the!notion!of!cultural!value!and!measurement.!
Parsons!(2015,!p.53)!argues!that!‘defining!cultural!value!is!not!a!question!of!finding!the!right!
“system”!but!resolving!inherent!flaws!in!the!terms!of!the!debate’.!What!has!ensued!is!an!
epistemological!debate!in!the!field!on!cultural!value!and!the!impact!agenda.!!
Countering!the!instrumentalisation!and!measurement!trend,!are!practitioners!and!scholars!
expressing!trepidation!towards!empirical!evaluation!in!the!community!arts!context!(Belfiore,!2014;!
Jackson,!2009;!Jennings!&!Baldwin,!2010;!Newman!et!al.,!2003).!This!is!evident!in!the!peer*reviewed!
literature!as!well!as!in!the!cross*sectoral!discourse!on!academic!led!blogs!and!professional!forums!
(Donovan,!2012;!Selwood,!2002).!One!of!the!lead!advocates!of!this!counter!position!is!Goldbard!
(2008,!2015)!who!asserts!deep!concerns!about!the!far*reaching!impacts!of!evaluation!on!the!field,!
and!the!perpetuation!of!the!instrumentalisation!of!the!arts.!Goldbard!argues!that!the!‘Metrics!
Syndrome’!(a!term!coined!in!reference!to!her!American!context!of!arts!funding,!evaluation,!and!
cultural!measurement)!is!creating!an!irreconcilable!ideological!crisis!for!community!arts!(Goldbard,!
2015,!2008;!Matarasso,!1996):!
The!trouble!is,!the!very!quest!for!metrics!is!contaminated!with!ideas!and!assumptions!
borrowed!from!worlds!that!have!nothing!in!particular!to!do!with!community!and!creativity.!
The!notion!that!everything!of!value!can!be!weighed!and!measured,!which!is!one!of!the!most!
grotesque!artefacts!of!post*Enlightenment!thinking,!is!antithetical!to!the!deep!values!of!
community!cultural!development.!Indeed,!in!this!domain,!the!search!for!metrics!actually!
harms!what!it!seeks!to!help.!(Goldbard,!2008,!p.1)!
This!argument!is!based!on!the!premise!that!the!social!and!economic!indicators!being!used!for!
cultural!measurement!and!evaluation!in!the!arts!have!been!adopted!from!other!fields,!such!as!
economics,!and!have!inherent!rhetoric,!paradigm!values!and!power!structures!that!are!not!always!
suited!to!the!community!arts!context!(Goldbard!2015,!2008).!!
One!ramification!is!that!there!has!become!an!increased!orientation!toward!the!socio*economic!
impacts!of!the!arts!over!the!past!two!decades,!and!this!is!closely!linked!to!how!public!funding!and!
policy!decisions!are!being!rationalised!and!legitimated!(MacDowall!et!al.,!2015;!Belfiore,!2014).!
Belfiore!(2014,!p.96)!explains!that!‘discussions!of!“value”!have!become!inextricably!linked!to!the!
challenge!of!“making!the!case”!for!the!arts!to!justify!public!cultural!funding!given!increasing!funding!
pressures!facing!public!service!provision,!especially!in!the!West’.!Within!the!discourse!on!cultural!
value,!it!is!also!evident!that!the!concepts!of!value!and!impact!are!increasingly!used!interchangeably!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 32!
(with!a!focus!on!the!measurable)!which!some!scholars!argue!has!obscured!the!debate!on!cultural!
value!and!reduced!its!scope!to!technocratic,!economic!and!instrumental!interpretations!(Goldbard,!
2015;!Belfiore,!2014).!Conceptualising!cultural!value!in!relation!to!public!value!has!implications!for!
arts!practice,!as!art*making!processes!and!creative!works!become!viewed!as!commodities!valued!for!
the!social!return!on!investment!they!deliver!(Blomkamp,!2015).!In!addition,!some!scholars!argue!
that!evaluation!and!benchmarking!of!practice!are!techniques!employed!by!government!as!a!way!of!
exercising!control!over!civil!society,!which!can!prevent!effective!community!engagement!and!
undermine!meaningful!collaboration!(Milbourne,!2009;!Larner!&!Butler,!2005;!Ilcan!&!Basok,!2004).!!
While!the!opponents!to!the!instrumentalisation!of!the!arts!and!the!intensification!of!impact!
measurement!are!not!arguing!for!a!data*free!approach!to!evaluation!in!community!arts,!they!do!
challenge!the!emergent!conventions!of!cultural!measurement!and!urge!practitioners!to!look!beyond!
a!narrow!instrumental!prescription!of!impact*oriented!evaluation!to!justify!arts!practice.!Badham!
(2015,!p.193)!argues!that!‘top*down!hegemonic!benchmarks!tend!to!disregard!the!diverse!cultural!
values!that!make!up!society’!and!advocates!a!more!‘bottom*up!approach’!to!cultural!indicators!
‘through!the!use!of!arts*based!and!participatory!methods!of!community*based!research’.!Goldbard!
(2015)!argues!for!a!shift!away!from!metrics!is!needed,!to!art!and!story*based!methods!as!a!means!to!
elucidate!the!value!of!arts!and!culture,!encouraging!practitioners!to!resist!the!imperative!to!measure!
culture.!Likewise,!Belfiore!(2014,!p.95)!pleas!‘for!a!collaborative!effort!to!resist!the!economic!doxa,!
and!to!reclaim!and!reinvent!the!impact!agenda!as!a!route!towards!the!establishment!of!new!public!
humanities’*!and!for!the!field!to!move!beyond!the!limiting!aspects!of!the!impact!debate!and!
economic!rationalist!framing!to!initiate!a!more!open!and!rich!discourse!on!cultural!value.!
2.3.2! Relevant#approaches#and#models#for#the#Nuclear#Futures#site#of#practice#
Despite!the!significant!developments!in!the!field!relating!to!the!evaluation!of!community!arts,!the!
limitations!of!the!conventional!approaches,!and!their!more!recent!quantitative!and!metrics*driven!
orientation,!are!widely!documented!(Goldbard,!2015;!Belfiore!&!Bennett,!2010;!2008).!In!the!field,!
what!remains!is!a!persistent!demand!for!tested,!effective!and!accepted!techniques!able!to!‘capture!
the!depth,!as!well!as!the!breadth,!of!the!encounter!between!communities!and!the!creative!arts’!
(Newman,!Curtis,!&!Stephens,!2003,!p.13).!This!reflects!more!general!calls!for!improved!
methodologies!in!evaluating!the!arts,!such!as!by!the!Australian!Business!Arts!Foundation!(AbaF)!
assertion!that!the!development!effective!evaluation!methodologies!is!a!key!challenge!facing!the!arts!
sector!(AbaF!&!Australia!Council!for!the!Arts,!2010).!While!it!is!beyond!the!scope!of!this!contextual!
scan!to!comprehensively!review!the!wide!spectrum!of!evaluation!methodologies!and!methods!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 33!
currently!applied!in!community!arts,!a!brief!review!of!methods!identified!in!the!literature!as!relevant!
to!the!research!(and!the!task!of!designing!the!READ!model!for!the!site!of!practice)!is!presented.!
As!part!of!the!contextual!scan!I!examined!a!number!of!existing!models!and!frameworks!developed!
for!evaluation!practice!for!community!arts!in!Australia!and!overseas,!as!well!as!evaluation!reporting!
and!exemplars!provided!by!the!Australia!Council!for!the!Arts!and!available!in!the!public!domain.!In!
the!Australian!context,!one!notable!evaluation!framework!of!relevance!was!Arts!Victoria’s!
Evaluating%Community%Arts%and%Community%Well%Being%(Keating,!2002).!This!practical!guide!was!
developed!for!community!art!practitioners!and!organisations,!providing!a!comprehensive,!step*by*
step!guide!to!evaluating!the!process,!outcomes!and!impacts!of!community!arts!programs!(Edenloff,!
2011).!!
Also!of!interest!for!this!research,!were!models!and!examples!in!the!field!that!demonstrated!effective!
strategies!for!designing!and!implementing!evaluation!using!collaborative!processes.!As!was!
introduced!in!Section!2.1,!the!Nuclear!Futures!site!of!practice!constitutes!a!‘creative!communities!
partnership!initiative’,!a!partner*oriented!niche!of!community!arts!promoted!by!the!Australia!
Council!for!the!Arts.!The!collaborative!nature!of!the!arts!practice,!involving!artists,!communities!of!
interest,!cultural!and!academic!institutions,!government!agencies,!and!not*for*profit!organisations,!
means!that!there!are!diverse!and!numerous!stakeholders!with!an!interest!in!the!program’s!
evaluation.!For!the!evaluators!of!such!complex!programs,!the!contextual!scan!revealed,!that!several!
fundamental!questions!arise!such!as:!How!is!cultural!value!and!evaluation!conceived!and!defined!
across!the!partnerships?!Which!values,!agendas!and!questions!should!drive!evaluation!practice?!
How!is!program!and!partnership!performance,!progress,!quality,!and!success!understood!and!
represented?!What!types!of!data!and!outputs!are!needed!to!meet!partner!expectations!for!
measuring!and!evidencing!impact!or!making!summative!conclusions?!Who!within!the!partnership!
should!be!involved!to!ensure!such!processes!are!representative?!(Swan!&!Atkinson,!2012;!Badham,!
2010;!Donaldson,!Christie,!&!Mark,!2008).!As!revealed!in!Section!2.3.1,!it!is!clear!that!questions!
arising!for!evaluations!involving!multiple!and!culturally!diverse!partners!are!not!merely!technical,!
they!are!philosophical!and!political!as!well.!
Within!the!community!arts!field,!there!is!a!growing!interest!in!how!the!participation*oriented!
evaluation!approaches,!discussed!in!Section!2.2.2,!might!be!applied!to!deal!with!these!issues!
routinely!arising!in!the!community!arts!context!(MacDowall!et!al.,!2013).!Clements!(2007,!p.325)!
argues!that!the!ideal!democratic!participatory!approach!for!community!arts!evaluation!would!
encourage!‘participant!involvement,!empowerment!and!self*management’!as!compared!to!what!he!
describes!as!the!‘normative!bureaucratic!and!autocratic!approaches!steeped!in!top*down!agendas!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 34!
which!may!exacerbate!social!problems’.!Advocates!for!a!‘bottom*up!approach’!argue!that!cultural!
value!and!credibility!of!findings!about!the!possible!impact!of!community!arts!need!to!include,!if!not!
be!wholly!determined!by,!local!knowledge,!expertise!and!ownership!(Badham,!2015;!Mercer!2005).!
Badham!(2015,!p.210)!argues!‘it!is!only!the!people!who!experience!the!community!directly!who!can!
determine!what!progress!means!locally’.!
In!the!international!context,!the!literature!review!revealed!numerous!exemplars!relevant!to!the!site!
of!practice!in!terms!of!how!participation*oriented!approaches!to!evaluation!were!successfully!
adopted!for!partnership*oriented!community!arts!projects.!One!example,!integrating!academic!and!
research!systems!for!learning!and!analysis,!was!the!participatory!evaluation!undertaken!by!the!
Chicago!Arts!Partnership!in!Education!(CAPE).!Based!in!the!USA,!CAPE!aimed!to!improve!education!
and!school!outcomes!(such!as!increased!student!academic!success,!critical!thinking!and!creativity)!
through!research*based,!arts!driven!education.!Central!to!the!CAPE!program!was!an!emphasis!on!
strong!partnership!development,!and!a!commitment!to!inquiry,!documentation,!professional!
development!and!collaborative!action!research!through!evaluation!(Burnaford,!2006).!The!approach!
adopted!by!CAPE’s!longitudinal!partners,!challenges!traditional!non*participatory!models!and!
demonstrated!a!turn!toward!a!culture!of!evidence.!Partners!and!participants!were!enabled!to!join!a!
community!of!researchers!to!collaborate!on!the!evaluation!design!and!implementation,!and!also!
contribute!to!the!generation!of!new!knowledge!in!the!field!of!Arts!Education!(Burnaford,!2006,!p.3).!
While!this!is!an!older!example!from!the!literature,!it!resonated!as!a!case!study!relevant!for!
facilitating!the!READ!application!because!of!the!strong!academic!orientations!of!several!of!the!key!
partners.!(This!will!be!expanded!on!in!Chapters!4!and!Chapter!5!about!the!READ!application.)!
Leading!the!way!in!innovating!applied!participatory!evaluation!practice!in!community!arts!in!
Australia!are!organisations!such!Big!hART!(community!arts!and!production!company),!Somebody’s!
Daughter!Theatre!company,!Arts!Victoria,!Queensland!University!of!Technology!and!the!University!of!
Melbourne.18!Of!particular!relevance!to!the!DCI!research!are!findings!emerging!from!the!three!year!
Australian!Research!Council!(ARC)!Linkage!Project!Towards%an%integrated%evaluation%framework%for%
communityAbased%arts!completed!in!2014!(see!MacDowall!et!al.,!2013).!The!ARC!research!tested!a!
number!of!evaluation!approaches!in!the!community!partnerships!context!based!on!four!industry*
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
18!Examples!from!these!organisations!are!also!referred!to!in!Chapter!5!and!include!(in!order):!Big!hART!(2010),!Palmer!
(2010);!Osmotherly!(2002),!Osfield!Consultants!(2007);!Keating!(2002);!Klaebe,!Ellison,!and!Kirk!(2015),!Gattenhof!and!
Seffrin!(2014);!MacDowall!et!al.!(2013).!I!also!reviewed!several!good!practice!evaluation!reports!and!industry!case!studies!
not!in!the!public!domain!which!were!informative,!with!assistance!from!the!Australia!Council!for!the!Arts!in!facilitating!a!
permissions!process!for!access!to!the!exemplars.!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 35!
embedded!case*studies!exploring!the!themes!of!negative!value,!new!aesthetics,!networked!thinking!
and!dialogic!approach!(MacDowall!et!al.,!2013).!Engagement!with!this!ARC!Linkage!Project!research!
has!been!formative!for!advancing!and!consolidating!an!understanding!of!the!context!of!practice,!and!
provided!unique!access!to!practitioners!working!on!participatory!community!arts!evaluation.!
Individual!exemplars!were!also!relevant,!such!as!Evaluation%of%the%Community%Arts%Development%
Scheme!(Kelaher,!Dunt,!Curry,!Berman,!&!Joubert,!2009).!
Methods%of%relevance%for%the%READ%application%
The!implication!for!evaluators!interested!in!fostering!participation!is!the!need!for!sound!methods!
that!prioritise!equity!amongst!partner!needs!and!utilise!the!strengths!of!the!site!of!practice!and!
context!–!while!achieving!rigorous!and!credible!results!(Hadley!&!Gattenhof,!2012).!Despite!the!
previously!noted!trend!toward!social!impact!methodology,!including!quantification!via!cultural!
indicators!(discussed!in!Section!2.3.1),!participatory!evaluation!methodologies!in!practice!still!rely!
heavily!on!qualitative!methods,!and/or!use!of!mixed!research!methods!to!elicit!qualitative,!
quantitative!and!performative!data!(including!ephemeral!rich!media!such!as!video,!audio!and!
images)!(Belfiore!&!Bennett,!2010;!Clements,!2007;!Haseman,!2006).!!
From!the!contextual!scan!there!were!a!number!of!relevant!methods!studied!for!the!READ!
application!in!relation!to!various!domains!of!evaluation!practice.!These!included!a!mix!of!
conventional!methods!for!data!collection,!analysis,!and!stakeholder!engagement!that!were!suitable!
for!participation*oriented!evaluation.!Included!was!research!on!methods!such!as!collective!
workshops,!focus!groups,!evaluation!specific!tools!(such!as!evaluation!needs!assessment),!
partnership!assessment!techniques,!relevant!ethical!frameworks!(Markiewicz,!2012;!Edenloff,!2011).!
In!addition!to!these!conventional!methods,!I!examined!a!range!of!art*based!methods!for!evaluation!
of!relevance!for!the!Nuclear!Futures!site!of!practice.!Art*based!methods!use!creative!and!artistic!
mediums!(e.g.!video/film,!photography,!narrative/storytelling,!performance,!written!text)!to!
generate!understanding!of!a!subject!and!address!social!research!problems!(Leavy,!2009;!McNiff,!
2008).!Such!methods!are!diametrically!opposed!to!the!quantitative!approaches!described!in!Section!
2.3.1.!Goldbard!(2015,!p.226)!advocates!that!artists!and!cultural!policy*makers!have!the!right!to!
draw!on!‘story,!image,!metaphor,!and!experience’!as!tools!to!convey!cultural!value!and!meaning,!
and!argues!the!relevance!of!these!methods!for!future!evaluation!practice!in!the!community!art!
context.!!
!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 36!
The!following!quote!elaborates!on!how!Goldbard!envisages!the!applications!of!art*based!methods:!
Right!now,!visionary!leaders!are!using!digital!stories!–!short,!first*person!sound*and*image!
video!narratives!–!to!capture!participant’s!experience!and!response.!They!are!employing!the!
methods!of!art!to!convey!art’s!value!and!identify!cultural!development!needs,!by!convening!
story!circles!and!turning!the!collective!wisdom!thus!extracted!into!plays!and!videos!that!can!
be!shared,!sparking!useful!dialogue;!or!by!engaging!community!members!in!envisaging!their!
collective!future!through!a!participatory!photography!project!in!which!images!and!captions!
show!what!needs!preservation!and!what!needs!development.”!(Goldbard,!2015,!p.226)!
While!art*based!methods!are!still!considered!to!be!‘on!the!fringe!of!social!science!research’!
(Edenloff,!2011,!p.!62),!and!are!most!commonly!used!in!conjunction!with!conventional!qualitative!
methods!and!not!as!stand*alone!methods,!supporters!argue!that!they!are!particularly!valuable!
techniques!for!arts*related!evaluation!(Rudkin!&!Davis,!2007).!Leavy!(2009)!argues!that!the!methods!
elicit!different!types!of!information!to!conventional!qualitative!and!quantitative!methods,!nuanced!
by!rich!descriptive!expression,!and!grounded!in!stories,!images!and!feelings.!Arts*based!evaluation!
methods!are!also!identified!as!effective!methods!when!working!with!children!and!youth,!alleviating!
the!reliance!on!written!and!oral!communication!(Edenloff,!2011).!Successful!application!is!also!
demonstrated!with!Indigenous!communities!due!to!the!reflexivity!that!art*based!methods!offer!
(Price,!McCoy,!&!Mafi,!2012).!All!of!these!attributes!of!the!creative!methods!suite!were!deemed!
highly!relevant!for!the!READ!application!(covered!later!in!this!dissertation)!to!supplement!the!more!
formal!and!conventional!evaluation!methods!being!included!in!the!READ!model!development.!
As!the!lead!practitioner!responsible!for!driving!the!design!and!implementation!of!the!READ!model,!
to!take!it!from!the!basic!idea!that!existed!prior!to!the!research!to!developing!a!complete!
methodological!framework!for!implementation,!the!contextual!scan!conducted!for!the!Doctor!of!
Creative!Industries!research!has!proved!extremely!valuable.!As!part!of!my!leadership!role!at!my!site!
of!practice,!I!shared!these!findings!with!my!colleagues!and!community!partners!to!ensure!our!
collective!efforts!towards!the!READ!application!(detailed!in!Chapter!5)!were!informed!by,!and!built!
upon,!existing!theories!and!knowledge!of!evaluation!practice.! !
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 37!
2.4! Conclusion#
The!contextual!scan!involved!the!far*ranging!review!of!scholarly!and!industry!literature!on!
evaluation,!and!exemplars!from!the!community!arts!context.!Investigating!the!key!literature!areas!
related!to!my!professional!practice!and!writing!up!the!findings!in!this!chapter!helped!to!make!sense!
of!the!vast!and!contested!practice!of!community!arts!program!evaluation.!Historical!and!theoretical!
trends!in!evaluation!were!charted,!and!the!cited!academic!analyses!were!revealing!*!particularly!in!
regard!to!the!impact!of!the!instrumentalisation!of!the!arts!on!evaluation!practice!and!the!still!
unfolding!debate!on!cultural!value.!Many!of!the!theorists!I!studied!(such!as!Goldbard,!MacDowall,!
Badham,!and!Belfiore)!resonated!with!my!own!philosophies!developed!through!my!professional!
practice,!and!I!experienced!immediate!benefits!for!my!evaluation!practice!as!I!underwent!an!
iterative!reflective!process!of!learning,!incorporating!new!understandings!into!practice,!and!
reframing!my!concept!of!evaluation!for!community!arts!as!I!delved!deeper!into!the!theory.!
Importantly,!the!scan!provided!the!basis!for!positioning!the!DCI!research!within!the!domain!of!
participation*oriented!program!evaluation!within!community!arts,!and!identified!gaps!in!the!
literature!for!participatory!frameworks!that!eloquently!deal!with!the!complexities!of!creative!
community!initiatives!involving!diverse!partnerships.!The!next!chapter,!Chapter!3,!outlines!the!
overarching!DCI!research!design!and!provides!the!rationale!for!the!qualitative!methodology!and!
methods!I!employed!to!study!and!critically!review!the!READ!model!developed!at!my!site!of!practice.!
!
!
!
!
!
!
!
!
!
Digital+story!–!Reflecting+on+the+problem+of+community+arts+evaluation+
!
As! part! of! the! program! of! forensic! reflective! practice! undertaken! for! the!
research!(detailed!in!the!next!chapter)! I!produced!a!digital!story!exploring!the!
research!problem,!as!a!sensemaking!exercise! in!response!to!themes!emerging!
from! the! contextual! scan.! The!digital! storytelling! process!was! employed! as! a!
research!method,!and!is!not!a!summative!product!of!the!scan.!It!does!however!
demonstrate!the!understanding!I!gained!during!the!early!phase!of!the!research!
in!relation!to!the!knowledge!gaps!for!community!arts!evaluation.!
The!video!(3mins!40sec)!and!script!for!the!digital!story!can!be!viewed!at:!
http://nuclearfutures.org/digital*story*community*arts*evaluation!!!
(Please!use!password:!!READ!)!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 38!
! !
Blank!page!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 39!
3! Research#design##
Chapter!3!outlines!the!research!design!implemented!for!the!Doctor!of!Creative!Industries!(DCI)!to!
study!the!development!and!first!industry!application!of!the!READ!model,!a!newly!devised!evaluation!
model!for!community!arts.!The!READ!approach!arises!from!professional!practice!and!integrates!
Reflection,!Evaluation,!Analysis!and!Documentation.!The!purpose!of!this!chapter!is!to!detail!the!key!
elements!and!rationale!for!the!DCI!research!design!*!the!critical!lens!I!devised!for!evaluating!the!
model,!in!order!to!substantiate!a!final!READ!product!for!evaluation!practice.!Summarised!is!the!
qualitative!methodology,!practice*informed!paradigm,!research!strategy,!and!the!data!collection,!
analysis!and!theory!building!methods!that!I!employed!during!the!DCI!to!meet!the!research!aims.!
One!of!the!significant!influences!on!the!research!design!was!the!context!of!the!enquiry,!and!the!
central!positioning!of!myself!as!researcher!and!lead!Evaluator.!The!research!was!dually!positioned!
within!industry!and!academic!contexts,!in!the!fields!of!community!arts!and!evaluation.!The!site!of!
practice!for!research!was!the!Nuclear!Futures!Partnership!Initiative!(Nuclear!Futures),!a!three*year!
creative!community!partnership!initiative!funded!by!the!Australia!Council!for!the!Arts!(overviewed!in!
Chapter!4).!!The!research!design!I!devised!for!the!DCI!aimed!to!provide!a!framework!for!the!
systematic!analysis!of!the!conceptualised!READ!model!(in!terms!of!performance,!strengths,!
weaknesses,!areas!of!innovation)!based!on!its!development!and!testing!for!the!Balaklava!Community!
Arts!Project!delivered!in!South!Australia!during!2014*5!(see!Chapter!5).!As!well!as!informing!the!
future!evaluation!practice!for!the!Nuclear!Futures!program,!the!research!aimed!to!identify!the!
potential!areas!of!innovation!of!broader!relevance!for!the!fields!of!evaluation!and!community!arts.!
To!address!the!DCI!research!problem,!I!adopted!a!practice*informed!approach!and!designed!a!robust!
qualitative!methodology!that!created!a!critical!and!reflective!lens!for!studying!the!READ!model’s!
development!and!first!industry!application.!Demanded!was!a!methodology!capable!of!capturing!and!
substantiating!the!learnings!arising!from!evaluation!practice!and!enabling!the!investigative!analysis,!
theorisation!and!refinement!of!the!model!based!on!the!Balaklava!experience.!The!primary!research!
strategy!for!the!enquiry!was!forensic!reflective!practice!–!a!knowledge*driven!approach!involving!the!
systematic!examination!of!thinking!and!action!for!critical!enquiry!and!for!the!continuous!
improvement!of!practice!(Schon,!1983).!This!strategy!utilised!reflective!and!qualitative!methods!for!
data!collection!including!journaling,!video!diaries,!interviews!and!observation.!In*depth!analysis!of!
the!data!generated!was!conducted!via!secondary!reflection!for!the!purpose!of!building!evaluation!
theory,!and!utilised!interpretative!techniques!such!as!coded,!textual!and!comparative!analysis,!as!
well!as!creative!and!meta*evaluation!techniques!(to!evaluate!the!evaluation).!! !
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 40!
3.1! Qualitative#research#methodology#
A!qualitative!methodology!was!selected!as!being!most!adept!at!situating!the!practitioner!as!a!
researcher!and!for!delivering!the!interpretative!aims!of!the!research.!The!qualitative!paradigm!
underpinning!the!research!inherently!valued!my!own!context!and!interpretation!as!a!researcher,!and!
allowed!reflexivity!to!emergent!themes!arising!from!the!inductive!research!process!and!reasoning!
(Denzin!&!Lincoln,!2011).!By!utilising!interpretative!research!practices!or!‘representations’!(such!as!
field!notes,!visual!texts,!interviews)!the!qualitative!methodology!allowed!me!‘to!make!sense!of!and!
interpret!phenomena!in!terms!of!the!meanings!people!bring!to!them’!(Denzin!&!Lincoln,!2011,!p.3).!
Specifically,!the!qualitative!research!methodology!endorsed!the!necessary!capture!and!
interpretation!of!meaning!relating!to!convergent!conceptualisations!(Ezzy,!2006;!Becker,!1996)!
which!in!this!case!related!to!catering!for!divergent!definitions!of!evaluation!and!the!READ!approach!
at!the!site!of!practice,!for!the!purpose!of!original!theory!development!of!the!READ!model.!
In!response!to!the!research!aims!(see!Section!1.4),!the!methodology!needed!to!address!the!dilemma!
of!conducting!research!required!to!both!answer!a!research!question!and!fulfil!a!practical!need!
(Blichfeldt!&!Andersen,!2006),!which!involved!‘the!active!and!deliberate!self*involvement!of!the!
researcher!in!the!context!of!his/her!investigation’!(McKay!&!Marshall,!2001,!p.!49).!That!is,!the!
research!sought!to!advance!evaluation!theory!and!practice!for!community!arts!and!this!involved!
facilitating!and!analysing!the!design,!implementation!and!analysis!of!the!READ!model!to!meet!the!
evaluation!needs!at!my!site!of!professional!practice.!As!explained!in!Section!3.2,!the!qualitative!
methodology!needed!to!elicit!data!prior!to,!during!and!following!the!READ!application,!and!this!was!
achieved!through!an!ongoing!schedule!of!reflective!practice.!Figure!3.1!depicts!the!relationship!
between!the!overarching!qualitative!methodology!for!the!DCI!(discussed!in!this!chapter)!and!the!
READ!application!under!review.19!
!
!
!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
19!It!is!important!to!note!that!the!approach!and!methods!documented!in!this!chapter!for!the!overarching!DCI!methodology!
are!separate!to!the!READ!methodology!presented!in!Chapter!5.!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 41!
!
Figure!3.1:!The!Doctor!of!Creative!Industries!research!design!
!
!
The!following!sections!explain!the!key!elements!of!the!DCI!methodology!summarised!in!Figure!3.1.20!
3.1.1! PracticeSinformed#research#paradigm#
Central!to!Queensland!University!of!Technology’s!(QUT)!DCI!program!is!the!dominant!paradigm!of!
practice*led!research.!Practice*led!refers!to!the!emergent!performative!research!paradigm!defining!
artist*as*researcher,!where!artists!embrace!the!dual!roles!of!the!researcher!and!the!researched!
through!creative!practice!(Sullivan,!2010,!2009;!Haseman!&!Mafe,!2009).!In!practice*led!research,!
the!artist*as*researcher!engages!in!the!creation!of!data!through!performative!or!creative!methods!
and!outputs!(Smith!&!Dean,!2009;!Gray,!1996).!Creative!practice%is!the!research!activity!(Haseman,!
2006).!In!most!cases,!DCI!enquiry!becomes!a!form!of!research!that!will!incorporate!creative!or!
design!practice!in!the!research!methodology!and!outputs.!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
20!This!figure!is!included!in!Chapter!1!as!Figure!1.1.!I!have!included!the!diagram!again!in!Chapter!3!as!it!is!an!important!
reference!figure!demonstrating!the!difference!between!the!overall!DCI!methodology!and!the!trialed!READ!methodology!
(‘the!application’).!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 42!
For!this!research,!my!role!in!the!professional!context!was!as!evaluator!for!a!creative!arts!program,!
and!not!as!a!creative!artist/!practitioner.!While!this!research!design!for!the!DCI!is!aligned!with!the!
practice*led!paradigm,!the!research!undertaken!was!non*performative!(in!terms!of!generating!
creative!arts)!and!better!described!as!practice*informed.21!The!practice*informed!approach!has!
nuances!of!the!practice*led!paradigm,!however!differs!in!that!the!research!is!undertaken!in!
relationship!to!creative!practice,!not!through!it.!!A!major!part!of!the!practice*informed!research!
methodology!employed!was!built!around!the!research!and!outputs!generated!through!the!
evaluation!practice.!While!not!primarily!geared!toward!creative!work,!it!is!important!to!note!that!
several!of!the!evaluation*related!outputs!utilised!arts*based!methods!to!produce!original!creative!
documentation.!
3.2! Research#strategy:#forensic#reflective#practice#
Over!the!past!30!years,!reflective!practice!has!gained!standing!as!a!research!strategy!for!critical!
enquiry!across!many!disciplines!including!Nursing,!Education!and!the!Creative!Industries!(Ryan!&!
Cousins,!2009;!Finlay,!2008;!Johns,!2006;!Alvesson!&!Skoldberg,!1999;!Gibbs,!1988).!Reflective!
practice!is!of!particular!relevance!for!uncertain!and/or!complex!social!and!technical!contexts!
(Bourner,!2003;!Schon,!1983)!and!highly!adaptable!for!practice*led!and!practice*informed!research!
(Haseman!&!Mafe,!2009).!For!the!DCI,!the!Creative!Industries!Faculty!at!QUT!developed!a!robust!and!
highly!critical,!knowledge*generating!application!of!reflective!practice!–!!forensic!reflective!practice!–!
which!is!the!recommended!research!strategy!for!DCI!candidates!(Stock,!2013).!The!next!section!
presents!the!theoretical!underpinnings!of!forensic!reflective!practice,!ahead!the!discussion!on!
methods!selected!for!this!DCI,!presented!in!Section!3.3.!
3.2.1! Theoretical#underpinnings#of#reflective#practice#
Central!to!reflective!practice!is!the!concept!of!reflection,!referring!to!the!active,!deliberate!and!in*
depth!examination!of!thoughts!and!action,!as!a!means!to!develop!or!improve!practice!(Bolton,!2010;!
Leitch!&!Day,!2000). Whether!directed!towards!thought!processes,!activities,!experiences,!events!or!
problematic!situations,!reflection!is!used!to!bring!a!subject!into!focus!and!generate!new!insights!and!
perspectives!about!the!subject!(Bolton,!2010;!Boud,!2001).!While!definitions!vary,!most!chart!a!
process!of!problem!identification,!reframing!(exploring!alternative!views)!and!analysis,!followed!by!
structured!action!(Loughran,!2002).!Figure!3.2!summarises!the!forensic!reflective!practice!process.!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
21!The!term!practice*informed!is!not!cited!from!the!literature!and!was!sourced!from!personal!communication!with!Assoc.!
Prof.!Sandra!Gattenhof,!with!input!from!Prof.!Helen!Klaebe!(on!28.5.13).!It!is!based!on!early!concepts!of!practice*based!
research!strategies!emerging!in!the!United!Kingdom!in!the!mid!1970s.!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 43!
!
+
!
!
!
!
!
!
!
!
!
!
!
!
Figure!3.2:!Model!for!forensic!reflective!practice!!
(Illustration!adapted!from!Scalan!&!Chernomas!(1997)!*!Three!Stage!Model)!
!
!
!
Early!scholarship!by!Dewey!(1933)!depicts!reflection!as!a!precise!and!rigorous!practice!using!
systematic!methods!of!observation,!description,!analysis,!theory!building!and!experimentation!to!
construct!meaning!(Rodgers,!2002).!Dewey!also!considered!the!practice!to!have!moral!ends,!
promoting!reflection!as!a!means!for!personal!and!intellectual!growth!of!self!and!others!(Rodgers,!
2002;!Dewey,!1933).!It!was!not!until!the!more!recent!era!of!theorisation!however!(from!the!1980s!
onwards),!introduced!by!Schon!(1983)!following!his!earlier!work!with!Argyris!in!the!1970s,!that!
reflective!practice!gained!traction!as!a!strategy!for!critical!enquiry!(Bourner,!2003).!!
Central!to!Schon’s!reflective!practice!theory!is!the!premise!that!‘our!knowing!is!in!our!action’!(Schon,!
1983,!p.49).!Schon!argues!that!reflection*on*action!(reflection!after!the!event)!and!reflection*in*
action!(reflection!during!the!event)!are!legitimate!and!valuable!sources!of!knowledge.!Bolton!
describes!the!reflective!practitioner’s!process!as!‘paying!critical!attention!to!the!practical!values!and!
theories!which!inform!everyday!actions,!by!examining!practice!reflectively!and!reflexively’!(Bolton,!
2010,!p.!xix).!Through!reflective!practice!the!implicit!knowledge!embedded!in!professional!practice!is!
able!to!be!unearthed,!critically!examined,!theorised,!tested!and!made!applicable!as!new!knowledge!
(Bourner,!2003;!Schon,!1983).!Bourner!asserts!that!it!is!the!process!of!critical!examination!through!
probing!questions!that!distinguishes!‘reflective’!from!‘unreflective’!thinking!(Bourner,!2003,!p.269).!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 44!
The!rise!of!reflective!practice!has!been!prominent!in!health,!teacher,!and!professional!education!
across!the!United!Kingdom,!North!America!and!Australia!(Ryan!&!Cousins,!2009;!Greenwood,!1993).!
Since!Schon’s!seminal!work,!numerous!models!to!operationalise!reflective!practice!have!been!
developed!across!various!fields,!including!in!the!Creative!Industries!(see!Smith!&!Dean,!2009;!
Doloughan,!2002).!The!now!widely!cited!domain!of!reflective!practice!includes!a!spectrum!of!models,!
each!differing!in!rigour!and!critical!reflection.!As!is!described!in!the!following!sections,!forensic!
reflective!practice!constitutes!a!significant!nexus!between!critical!reflection,!professional!practice,!
and!research!enquiry!(Boud,!2001;!Newman,!1999).!
3.2.2! Methodological#considerations#for#forensic#reflective#practice#
For!this!research,!the!forensic!reflective!practice!strategy!has!been!designed!to!be!rigorous!while!
guiding!a!flexible!and!socially!responsive!approach!to!the!critical!review!of!practice,!through!
deepening!cycles!of!forensic!reflection.!As!well!as!framing!the!research,!reflective!practice!was!used!
as!an!evaluator’s!tool!for!improving!evaluation!practice,!which!is!argued!in!the!literature!to!help!
make!explicit!some!of!the!espoused!values!and!assumptions!that!commonly!get!embedded!within!
program!evaluation!design!and!implementation!(MacDowall!et!al.,%2013;!Bolton,!2010).!!
In!planning!the!program!of!forensic!reflective!practice!several!methodological!issues!were!
addressed.!First,!in!considering!the!interaction!between!researcher!and!research!material,!it!is!
acknowledged!that!through!the!reflective!practice!process!‘knowledge!is!negotiated’!as!a!result!of!
personal!construction!(Gray,!1996,!p.13).!While!traditional,!positivist!opponents!may!argue!that!
subjectivity!and/or!the!centrality!of!the!practitioner!in!the!research!context!‘stands!to!infect!the!
objective!‘truth’!and!universal!applicability!of!research!findings’!(Haseman!&!Mafe,!2009,!p.212),!the!
credential!of!the!reflective!practice!strategy!is!that!it!adequately!accommodates!and!makes!meaning!
from!the!value*based!terrain!of!practice*informed!research!to!unearth!valuable!tacit!knowledge.!
Second,!for!the!research!strategy!to!be!forensic,!knowledge*directed!and!produce!communicable!
findings,!the!methodology!for!the!collection!and!analysis!of!empirical!material!during!the!reflective!
process!needed!to!be!systematic!and!rigorous!(Stock,!2013;!Archer,!1995).!Schon!argues:!
The!dilemma!of!rigor!or!relevance!may!be!dissolved!if!we!can!develop!an!epistemology!of!
practice!which!places!technical!problem*solving!within!a!broader!context!of!reflective!inquiry,!
shows!how!reflection*in*action!may!be!rigorous!in!its!own!right,!and!links!to!the!art!of!practice!
in!uncertainty!and!uniqueness!to!the!scientist's!art!of!research.!(Schon,!1983,!p.69)!
Specifically,!scholarship!on!reflective!practice!suggests!that!the!type!and!quality!of!claims!to!
knowledge!able!to!be!made!as!a!reflective!practitioner!will!be!strongly!influenced!by!the!methods!
selected!and!the!effectiveness!of!implementation!(Stock,!2013;!Loughran,!2002).!!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 45!
In!response,!recognising!the!implicit!and!explicit!influences!of!myself!as!researcher!during!the!
reflective!enquiry!demanded!accurate!representation!of!the!tensions!created!from!‘the!necessary!
interrelationship!of!(pre*existing)!materiality!and!subjectivity!as!they!interact!in!a!dynamic!and!
motivated!context’!(Doloughan,!2002,!p.61).!Techniques!to!promote!self*questioning,!dialogic!
reflection!and!metacognition!that!would!allow!me!to!stand!back!from!the!event,!consider!alternative!
perspectives,!explore!the!emotional!influences!present,!and!review!my!reflective!processes!were!
incorporated!based!on!practices!evidenced!in!the!literature!(such!as!Moon,!2001).!Comprehensive!
documentation!aided!to!make!these!tensions!explicit,!and!during!the!READ!application!I!attempted!
to!keep!a!balance!between!the!READ!practice!and!the!forensic!reflective!practice.!Scholars!such!as!
Haseman!and!Mafe!(2009)!and!Finlay!(2008)!argue!that!managing!the!flux!between!research!and!
practice,!a!balance!between!rigour!and!moderation!needs!to!be!achieved!so!that!the!reflective!
process!becomes!systematic!without!overbearing!the!practice!(which!is!in!my!case!relates!to!my!
evaluation!practice),!to!ensure!an!adequate!level!of!reflexive!detachment!is!maintained.!
3.3! Data#collection#and#analysis#methods#
The!suite!of!data!collection!and!analysis!methods!included!in!the!research!design!was!informed!by!a!
literature!review!of!qualitative!research!methods!(informed!by!scholars!such!as!Denzin!&!Lincoln,!
2011;!Cresswell!&!Clarke,!2007;!Ezzy,!2006;!Becker,!1996).!(See!Table!3.1!for!a!summary.)!!While!the!
methods!are!categorised!separately!in!Table!3.1,!the!phases!of!data!collection,!analysis!and!theory!
development!were!iterative!and!cyclic!processes!as!part!of!the!forensic!reflective!practice.22!!
The!primary!methods!for!data!collection!were!a!mix!of!tailored!reflective!methods!(journaling,!video!
diary,!structured!exercises)!and!standard!qualitative!methods!(interviews!and!observation).!
Secondary!reflection!for!the!purpose!of!in*depth!analysis!and!theory!building!(to!develop!a!READ!
model!of!practice)!utilised!mixed!integrative!techniques!and!creative!methods,!refining!data!and!
consolidating!emergent!themes.!Reflective!methods!included!conventional!data!capture!activities!as!
well!as!practice*related!methods!involving!the!generation!and!curation!of!creative!documentation!–!
where!critical!analysis!is!undertaken!through!the!compilation!process.!Throughout,!emergent!
findings!were!fed!into!the!testing!and!theory!development!of!the!READ!model!during!and!following!
the!Balaklava*based!application,!in!an!iterative!process!of!data!collection,!organisation,!analysis,!and!
testing!guided!by!deepening!cycles!of!reflective!review.!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
22!In!the!evaluation!field,!reference!to!theory!development!differs!from!the!conventional!theory!building!of!other!fields!
that!adhere!to!an!academic!or!theoretical!standard,!and!tend!to!be!exclusively!prescriptive!(not!predictive)!offering:!‘a!set!
of!rules,!prescriptions!and!prohibitions!that!specify!what!a!good!or!effective!evaluation!study!is!and!how!an!evaluation!
study!should!be!conducted’!(Christie!and!Alkin,!2008,!p.132;!Alkin,!2004).!In!the!evaluation!field,!models!of!practice!
contribute!to!evaluation!theory.!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 46!
Table!3.1:!Qualitative!methods!employed!for!the!DCI!research!design!(2013*15)!!
QUALITATIVE!METHODS! RATIONALE!
!
!
!
!
!
Data+collection+
(primary)+
++
Prior,!during!and!
post!the!READ!
application!!
!
Forensic!reflective!practice:!
•! journaling!(written)!
•! video!diary!
•! semi*structured!exercises!
•! collective!sessions!
•! creative!documentation!!
•! enabled!original!data!generation!via!individual/!collective!
reflection!using!multiple!methods!(Bolton,!2010)!
•! unlocked!tacit!knowledge!(Schon,!1983)!
•! responsive!to!the!emergent!research!themes!and!data!from!other!
(Archer,!1995)!
•! creative!documentation!(photo!books/essays!and!digital!
storytelling)!acted!as!dual!data!collection!and!analysis!methods!
Interviews!and!consultation:!
•! semi*structured!and!
unstructured!!
•! individual!and!group!face*to*
face!and!telephone/!Skype!
•! informal!stakeholder!
consultation!and!liaison!!
•! made!accessible!the!interpretations,!perceptions!and!experiences!
of!individuals!considered!to!have!a!detailed!knowledge!of!the!
research!topic!(Mason,!2002)!
•! allowed!for!mix!of!sample!and!question!design!techniques!
•! flexible!for!planned!and/or!spontaneous!interviews/!feedback!!
•! produced!rich!source!for!data!collection!and!was!used!for!
clarification!for!theory!and!product!development!(Charmaz,!2000)!
Observation!as!researcher!and!
READ!facilitator:!
•! field!notes!
•! anecdotal!notes!!
•! utilised!researcher/!practitioner!observations!!
•! provided!thick!and!rich!descriptions!of!context!and!key!project!
milestones!(Denzin!and!Lincoln,!2011)!
•! important!bridge!between!program!management!and!evaluation!
!
Data+collection+
(secondary)+
Contextual!Scan:!
•! academic!and!industry!
sources,!referred!exemplars!
•! online!and!social!media!
•! utilised!a!variety!of!data!sources!such!as!academic!literature,!
industry!reports/!exemplars,!website!and!online!forums!!
•! supported!the!theory!development!process,!and!informed!coding!
and!analysis!
!
!
!
!
!
!
!
Data+analysis+
and+product+
development+
Reflective!creative!documentation:!
•! digital!storytelling!•! photomedia!collation!
•! communications!
•! method!for!analysis!and!content!generation!(Burgess,!2006)!
•! reflection!used!as!communication!tools!(Couldry,!2008)!
•! the!process!of!analysis!and!producing!creative!documentation!
involved!a!hierarchy!of!reflection!from!descriptive!to!dialogic!to!
critical!enquiry!Hatton!and!Smith!(1995)!
Analysis!techniques!included:!
•! thematic!coding,!sorting,!and!
categorisation!!
•! coded,!interpretative,!
comparative,!textual,!
collective,!reflective!analysis!!
•! meta*evaluation!
•! well!suited!for!culturally!sensitive!research!topic!(Charmaz,!2000)!
•! micro!level!analysis!uncovered!how!stakeholder!constructs!of!
meaning!and!deeper!level!analysis!teased!out!the!underlying!
discourse/!ideologies!feeding!into!READ!constructions!
•! manual!approach!used!for!managing,!sorting,!and!interpreting!
information!to!deepen!reflective!practice!(i.e.!not!software)!
•! meta*evaluation!used!to!evaluate!the!evaluation!(Scriven,!2012)!
Theory!and!product!development!!
•! information!management!!
•! theoretical!sampling!
•! inductive!reasoning!
•! critical!reflection!
•! design!and!communication!
•! identified!interesting!patterns/!themes!(Braun!&!Clarke!2006).!
•! inductive!approaches!were!tailored!to!suit!theory!development!
(Lehmann,!2001;!Charmaz,!2000)!
•! the!process!involved!deep!and!critical!analysis!(Dobson,!1999)!•! constructivist!approaches!suited!to!the!qualitative!research!at!
hand!(Charmaz,!2000)!
•! structured!design!and!product!development!
!
Concurrent!with!the!primary!research!was!a!comprehensive!contextual!scan!(see!Chapter!2).!The!
scan!comprised!a!literature!review!in!evaluation!(particularly!in!the!domain!of!participatory!
evaluation)!and!community!arts!(including!community!partnerships)!and!review!of!practice*based!
resources!(including!industry!reports!and!documentation,!online!and!social!media!platforms,!peer*
recommended!evaluation!exemplars,!and!seminar/!conference!papers).!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 47!
The!following!sections!provide!descriptions!of!the!key!research!methods!utilised!as!part!of!the!
forensic!reflective!practice!strategy.!The!methods!were!employed!for!data!collection!and!analysis!for!
my!process!of!appraising!the!performance!and!outcomes!of!the!READ!application,!and!facilitating!
the!collective!development!of!the!model!for!future!use.!!
3.3.1! Reflective#methods##
A!cognitive!program!for!forensic!reflective!practice!was!designed,!utilising!scheduled!and!reflexive!
sessions!for!data!collection,!analysis!and!knowledge!generation.!The!methods!selected!supported!
the!reflective!practice!cycle!shown!in!Figure!3.2,!and!were!chosen!based!on!the!quality!and!depth!of!
the!reflection!they!could!deliver,!which!is!argued!by!Moon!(2001)!to!be!of!critical!importance.!The!
multimodal!inquiry!approach!I!devised!was!inspired!from!the!Stepping%Outside%the%Circle!online!
resource,!a!reflective!practice!framework!for!creative!facilitators!created!by!Moffatt!(2014)!geared!
toward!supporting!transformative!learning!and!action.23!Across!the!READ!application!I!employed!
reflective!methods!at!various!times,!such!as!before,!during!or!post*events!or!key!evaluation!
milestones.!Loughran!(2002,!p.35)!states!that!a!mix!of!reflective!methods!employed!as!‘anticipatory,!
retrospective!and!contemporaneous!reflection’!achieves!Schon’s!reflection*in*action!and!reflection*
on*action!principles,!as!well!as!incorporates!reflection*before*action!to!rationalise!pre*existing!
judgments,!prejudices!and!actions!(Kotzee,!2012;!Finlay,!2008;!Loughran,!2002;!Schon,!1983).!!
The!documentation!of!reflective!practice!aimed!to!capture!progress,!learning,!problematic!blocks,!
failures,!strengths,!and!shifts!to!articulate!the!taken*for*granted!aspects!of!practice.!As!argued!by!
Boud!(2001,!p.9),!reflective!documentation!functions!as!‘a!lens!for!learning’.!Informed!by!techniques!
of!scholars!such!as!Bolton!(2010),!Gray!(1996)!and!Burgess!(2006),!the!methods!I!employed!included!
unstructured!and!structured!reflective!journaling!(20+!entries),!video!diaries!(10+!entries),!
production!of!one!digital!story,!anecdotal!notes,!online!posts,!and!archiving!of!visual!and!audio!
material.!The!systematic!use!of!such!methods!is!argued!to!result!in!grounded!research!analysis!and!
interpretative!learning!(Stock,!2013),!and!produce!an!emergent!language!able!to!blend!process!and!
product,!form!and!content,!creativity!and!rationality!(Doloughan,!2002).!
! !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
23!Moffat’s!framework!can!be!retrieved!from:!http://www.steppingoutsidethecircle.com/#!build*your*own/c1qr4!!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 48!
Journaling*and*video*diaries*
The!two!principal!reflective!methods!I!employed!were!written!reflective!journaling!and!recording!a!
video!diary.!Guiding!the!forensic!reflective!practice!for!both!of!these!methods!were!strategic/!
directed!questions!and!activities,!as!well!as!an!unstructured!approach.!Boud!(2001,!p.12)!describes!
the!unstructured!approach!as!being!characterised!by!a!‘focus!on!uncertainty,!perplexing!events,!and!
exploration!without!necessarily!knowing!where!it!will!lead’.!Throughout!my!reflective!practice,!
searching!questions!(reflective!questions)!were!used!to!stimulate!reflective!thinking!and!scrutinise!
my!practitioner!experience.!This!is!a!technique!supported!by!Bourner!(2003).!!
Asking!questions!of!ourselves!as!practitioner*researchers!not!only!produces!new!information!and!
perspectives,!but!can!stimulate!creative!and!innovative!shifts!in!practice!(Tharp,!2003;!Gray!1996).!
Haseman!and!Mafe!(2009,!p.225)!argue!‘questioning!is!the!key!to!deepening!reflexivity’.!Utilised!
were!emergent!questions!as!well!as!selected!structured!and!semi*structured!exercises/!templates/!
question*sets!designed!to!guide!my!reflective!exploration,!promote!skills!development!and!advance!
practice!(Bourner,!2003).24!The!results!of!this!type!of!reflective!practice!are!documented!in!Chapter!
6.!Combined,!the!journaling!and!video!diary!sessions!and!exercises!produced!vast!and!rich!data.!
Interestingly,!they!also!produced!many!questions!which!compelled!the!reflective!practice.!!To!assist!
with!organising!and!re*evaluation!through!the!reflective!practice,!reframing!techniques!were!used!
involving!the!consideration!and!analysis!of!alternative!perspectives!via!techniques!such!as!viewing!
data!through!the!lens!of!different!stakeholder!positions!and!the!meta*evaluation!techniques!
discussed!below.!
Critical*buddy*and*shared*reflective*discussion**
Self*evaluation!was!enhanced!through!critical!exchange!and!shared/collective!reflection!sessions!
with!selected!peers!and!colleagues!working!at!the!site!of!practice.!Such!sessions!are!defined!in!the!
literature!as!critical!friend!or!buddy!sessions!and!involve!review,!feedback!and/or!critical!discussion!
on!practice!between!individuals!(Bambino,!2002).!The!technique!of!sharing!reflective!dialogue!with!a!
critical!friend!(or!buddy)!is!argued!by!Lamont!(2011,!p.64)!to!help!‘colleagues!to!make!sense!of!their!
world!by!being!non*judgmental!and!trying!to!understand!their!perspective’.!During!the!Balaklava!
project,!I!conducted!critical!buddy!sessions!with!three!members!from!the!creative!team,!and!this!
proved!useful!given!the!issues!arising!from!the!evolving!READ!model!theorisation!and!application!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
24!Templates/!exercises!included!an!evaluators’!competency!framework,!evaluation!design!exercises!based!on!Davidson’s!
Actionable!evaluation!framework!and!meta*evaluation!criteria!(Davidson,!2005),!and!Michael!Scriven’s!meta*evaluation!
checklist!(Scriven,!2011).!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 49!
transcended!my!own!individual!practice!and!affected!multiple!stakeholders!and!the!program!
management!and!art!practice!of!the!Nuclear!Futures!initiative.!!
Data!was!also!collected!from!small!and!medium!group!reflections!(ranging!from!3!*8!people),!and!
there!were!many!of!these!sessions!integrated!into!the!creative!residency!program!associated!with!
the!READ!application.!These!sessions!are!an!example!of!a!collective!reflection!method,!and!data!was!
captured!via!note!taking,!photographs,!and!observation.!The!sessions!were!a!mix!of!structured!
(facilitated)!and!self*governing!(un*facilitated)!reflective!dialogues.!
Digital*storytelling**
Digital!storytelling!was!utilised!early!during!the!research!as!a!media!production!process!for!analysis.!
In!general!terms,!digital!storytelling!is!a!short!multi*media!production!method!used!for!telling!stories!
digitally,!often!with!an!autobiographical!or!mini*documentary!narrative!(Centre!for!Digital!
Storytelling,!2014;!Lambert,!2012;!Burgess,!2006).!Research!demonstrates!digital!storytelling!
applications!can!be!an!effective!for!learning,!professional!reflection,!knowledge*generation!and!
communication!(Couldry,!2008).25!!For!the!research,!I!employed!digital!storytelling!as!means!to!
understand!and!articulate!the!complexities!I!faced!as!an!evaluator!practitioner!at!the!site!of!practice,!
which!was!central!to!the!work!involved!in!developing!the!READ!model.!In!particular,!I!utilised!the!
method!as!a!sensemaking!process!at!the!completion!of!the!contextual!scan.!The!digital!story!and!
transcript!can!be!viewed!at:!http://nuclearfutures.org/digital*story*community*arts*evaluation!
(Password:!READ).26!!
The!digital!storytelling!technique!used!for!the!research!is!an!adaptation!from!the!original!model!
introduced!by!the!Centre!for!Digital!Storytelling!(CDS)!–!a!model!involving!the!co*creative!practice!
where!ordinary!people!use!digital!media!tools!(images,!voiceover,!music)!to!produce!a!2*3!minute!
movie!of!an!autobiographical!story,!in!partnership!with!experienced!professionals!(Centre!for!Digital!
Storytelling,!2014;!Lambert,!2012;!Lundby,!2008;!Watkins!&!Tacchi,!2008).!Like!the!CDS!model,!the!
digital!storytelling!method!used!for!the!reflective!practice!placed!emphasis!on!process,!and!utilised!
techniques!such!as!the!personal!voiceover!and!combined!visual!and!audio!material!to!convey!an!
overarching!narrative!(Lambert,!2010;!Drotner,!2008).!The!application!for!reflective!practice!
however!aligns!more!closely!to!successful!digital!story!adaptations!developed!for!Action!Research!
and!Ethnography!(as!represented!by!Burgess!&!Klaebe,!2009),!where!digital!storytelling!is!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
25!See!Hlubinka!(2003)!for!an!example!of!a!successful!education*related!application!of!digital!storytelling;!or!Kathy!
Schrock’s!extensive!list!of!applications!across!curriculum!at!http://www.schrockguide.net/digital*storytelling.html!!26!The!digital!story!was!created!as!a!private!product!of!the!reflective!practice,!rather!than!produced!for!public!audiences.!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 50!
renegotiated!from!a!workshop*based!learning!model!to!a!research!and!documentation!method!for!
knowledge*generation!(Hartley!&!McWilliam,!2009).!!While!still!using!a!self*representational!
narrative,!the!adaptation!is!not!strictly!autobiographical!and!uses!a!systematic!and!analytical!
approach!to!the!narrative,!with!the!researcher!centrally!positioned!in!this!process!(Hertzberg!Kaare!
&!Lundby,!2008).!!
The!digital!story!produced!is!an!example!of!how!forensic!reflective!practice!involved!cycles!of!
secondary!reflection!as!a!method!of!analysis.!Experimenting!with!the!digital!story!format!allowed!me!
to!re*examine!(reflect!upon)!the!research!problem!in!response!to!the!information!arising!from!the!
contextual!scan!and!preliminary!interviews!conducted.!As!a!secondary!reflective!activity,!the!digital!
storytelling!helped!me!organise!and!synthesise!data,!identify!relevant!themes!emerging!from!the!
scholarship,!industry!documentation!and!research!participants,!and!undertake!analysis!via!scripting!
the!narrative!and!editing!the!multi!media!content!such!as!photomedia,!film!and!sound!samples.!
Critical!reflection!and!analysis!became!creative!and!iterative!processes!leading!to!the!deep!
interrogation!of!both!primary!and!secondary!research!material,!and!the!research!topic.!!
Meta8evaluation**
Given!one!of!the!central!tasks!of!the!research!was!to!critically!review!the!READ!application,!and!
assess!the!emergent!READ!approach!as!a!potential!model!of!practice!for!community!arts!evaluation,!
the!forensic!reflective!practice!needed!to!generate!data!that!would!make!explicit!the!practical!
insights!and!learnings!arising!from!the!READ!trial.!This!related!to!understanding!and!assessing!data!
generated!about!my!own!evaluation!practice!as!READ!facilitator,!as!well!as!the!collective!evaluation!
practice!and!experience!of!the!READ!model!at!the!site!of!practice!with!stakeholders.!To!generate!
knowledge!about!the!model’s!functionality!and!to!determine!the!strengths,!weaknesses!and!
potential!areas!of!innovation!demonstrated!from!the!READ!approach,!I!employed!meta*evaluation!
and!situation!analysis!methods!for!critical,!reflective!appraisal.!
The!term!meta*evaluation,!in!the!field!of!evaluation,!refers!to!the!process!of!evaluating!the!
evaluation!(Scriven,!1991).!Davidson!(2005,!p.205)!describes!meta*evaluation!as!the!‘determination!
of!the!quality!and/or!value!of!an!evaluation’.!As!is!the!case!for!evaluation!(see!Chapter!2),!meta*
evaluation!can!also!be!ambiguous!and!contentious,!however!it!does!offer!a!means!to!appraise!
evaluation!in!reasoned!and!meaningful!ways!(Davidson,!2005).!!
!
!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 51!
Davidson!(2005,!p.215)!outlines!the!basic!process!for!meta*evaluation!as!follows:!
At!a!bare!minimum,!meta*evaluation!should!involve!checking!in!regularly!with!clients!to!see!
how!useful,!valid,!credible,!cost*effective,!and!appropriately!conducted!the!evaluations!have!
been!to!them.!Also!helpful!for!self*directed!meta*evaluation!is!the!exercise!of!running!
through!one!or!more!evaluation!or!meta*evaluation!checklists!to!see!how!well!each!of!the!
relevant!checkpoints!or!standards!has!been!addressed.!The!meta*evaluation!checkpoint!in!
this!case!should!consist!of!an!honest!critique!of!the!strengths!and!limitations!of!the!
evaluation!and!what!can!and!cannot!be!inferred!from!it.!
The!above!text!is!also!summative!of!the!approach!I!adopted!for!the!research!problem!(see!Chapter!6!
for!findings!and!further!explanation!of!the!meta*evaluation!criteria!used).!!
Meta*evaluation!and!situation!analysis!techniques/methods!used!in!the!reflection!sessions!included:!
•! Davidson’s!(2005)!meta*evaluation!criteria:!(a)!validity!of!conclusions,!(b)!utility!to!relevant!
stakeholders,!(c)!conduct,!(d)!credibility,!and!(e)!cost.!!
•! Scriven’s!(2003b,!2011)!Key!Evaluation!Checklist!(KEC)*!a!prescriptive!evaluation!matrix!with!
sequenced!criteria!defined!by!Scriven!as!elements!for!evaluation!best!practice.!
•! Reflective!questions!arising!at!the!site!of!practice!(informed!by!stakeholder!perspectives),!
such!as!asking!why?!or!why!not?!areas!of!READ!worked!or!didn’t,!and!what!could!have!been!
done!differently!or!better?!(see!Chapter!6,!Section!6.1.2).!
•! SWOT,!a!situation!analysis!tool!to!examine!the!strengths,!weaknesses,!opportunities!and!
threats!(SWOT)!of!the!READ!model!(an!established!business!and!project!management!
technique!that!enables!the!user!to!take!different!perspectives!on!a!product/!idea!for!the!
purpose!of!development,!assessment,!and!improvement).!!
Reflective!exercises!based!on!the!above*stated!methods!and!techniques!systematised!and!
structured!the!reflective!analysis.!With!the!data!generated,!I!fed!the!results!and!emergent!ideas!into!
the!continuous!development!and!refinement!of!the!READ!model,!striving!to!overcome!identified!
weaknesses!or!limitations!in!the!conceptualised!and/or!Balaklava*based!iteration!of!the!model.!
Likewise,!I!sought!to!identify!those!attributes!of!the!developed!model!and!its!functioning!which!are!
most!likely!to!become!successful!innovations!for!community!arts!evaluation,!scrutinising!which!
would!be!relevant!for!future!applications!for!the!Nuclear!Futures!program!and/or!beyond.!As!part!of!
the!full!complement!of!techniques!used!during!reflective!practice,!meta*evaluation!allowed!me!to!
draw!both!on!academic!scholarship!and!tacit!knowledge!from!my!professional!practice!to!enable!the!
determination!of!effective!practice!for!the!Balaklava!application,!in!order!to!clarify,!explain,!and!
justify!the!findings!extrapolated!(see!Chapter!6!and!7).!
!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 52!
3.3.2! Interviews#and#consultation##
The!interviews!and!in*depth!consultation!with!partners!aimed!to!generate!data!specific!to!the!
stakeholder!collective!and!the!Nuclear!Futures!initiative,!as!well!as!identify!general!themes!relevant!
to!the!community!arts!context.!The!interviews!and!consultation!sessions!were!undertaken!before,!
during!and!after!the!application!and!involved!thirteen!participants.!!
Data!was!collected!for!the!following!three!purposes:!
•! Gaining!understanding!of!the!broader!context!of!community!arts!and!challenges!for!
evaluation!(not!related!to!the!READ!application!or!Nuclear!Futures!site!of!practice).!
•! Scoping!with!stakeholders!to!identify!the!evaluation!needs!for!the!Balaklava!Community!Arts!
Project!specifically,!within!the!broader!Nuclear!Futures!program!(prior!and!during!the!READ!
application).!
•! Obtaining!feedback!and!analysis!from!stakeholders!participating!in!the!READ!application!
about!the!emergent!READ!model!and!their!experience!of!participation.!
The!interviews!and!consultation!conducted!generated!data!for!the!overarching!DCI!methodology!
and!as!an!embedded!part!of!the!READ!application!(see!Chapter!5).27!Interviews!and!consultation!
sessions!took!on!a!variety!of!modes!across!several!phases!of!the!READ!application,!with!varying!
numbers!of!questions,!and!adaptations!to!accommodate!interviewees!(an!approach!supported!by!
Rowley,!2012).!The!sessions!conducted!to!collect!feedback!about!the!READ!application!generated!
valuable!data!for!validating!content!arising!from!the!reflective!methods!(for!triangulation).!Sessions!
were!face*to*face!or!via!Skype,!and!were!either!recorded!and!transcribed,!and/or!done!with!note!
taking.!Approximately!five!group!sessions,!each!attended!by!2*8!participants,!were!facilitated!
(corresponding!with!interview!questions).!!
Sampling**
Research!participants!were!sourced!using!techniques!including!purposeful!sampling!(i.e.!involving!
well!regarded!industry!practitioners/experts),!snowball!sampling!(i.e.!referred!research!informants)!
and!criterion!sampling!(i.e.!Nuclear!Futures/Balaklava!project!partners!and!stakeholders)!to!ensure!
participants!were!site!relevant!and/or!had!a!detailed!understanding!of!the!field/context.!Sampling!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
27!Given!that!the!content!collected!in!these!interviews!spanned!both!the!DCI!and!READ!methodologies!I!have!included!
information!about!the!interviews!in!the!dissertation!sections!related!to!both!(see!Chapter!5!for!READ!application!details).!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 53!
techniques!were!informed!by!research!by!DiCicco*Bloom!&!Crabtree!(2006),!Babbie!(1992)!and!
McNeill!(1985).!Theoretical!sampling!was!used!during!the!theory!development!process!where!it!was!
necessary!to!clarify!ambiguities!as!the!READ!model!developed,!involving!consultation,!interviews,!
and!feedback!processes!with!Alphaville!(host!organisation!for!Nuclear!Futures)!and!stakeholders.!!
The!research!participants!interviewed!were!representatives!from:!
•! Government!arts!and!funding!bodies!(state!and!federal)!!
•! Peak!arts!organisations!and!cultural!institutions!
•! Community!arts!practitioners/!creative!artists/!arts!workers/!administrators!
•! Community!partners!and!community!arts!participants!
•! Evaluator!practitioners!
•! Industry!leaders/!experts/!specialists!in!the!research!area!
•! Academic!researchers.!
Question*design*for*interviews,*consultation*and*group*sessions*
Consideration!was!given!to!the!structure!and!type!of!questions!used!in!the!sessions!and!their!
implications!for!the!information!gathered!in!response!(in!terms!of!accuracy!and!validity).!The!
questions!aimed!to!collect!individual!articulations!of!evaluation!to!provide!insight!into!the!
underlying!constructs!of!participant!definitions,!and!identify!evaluation!needs!and!interests.!This!
responds!to!evidence!from!the!contextual!scan!that!for!evaluations!to!be!utilised,!they!need!to!be!
able!to!answer!the!questions!of!interest!for!the!intended!users!(Kotvojs!&!Hurworth,!2013;!Patton,!
2008).!Information!was!also!collected!on!stakeholder!experiences!of!the!READ!approach!and!the!
utility!of!the!READ!outputs!for!their!needs.!Techniques!for!interview!design!included!funnelling!
(where!interviews!begin!with!general!or!open!questions!and!move!to!more!probing!ones),!story*
telling!(using!questions!that!elicit!stories!and!examples)!and!filtering!(so!respondents!are!directed!
only!to!the!pertinent!questions)!(Kendall!&!Kendall,!2001).!!!
Question!schedules!were!modified!as!necessary!to!suit!respondents!and!setting.!Probing!and!cross!
checking!during!the!interviews!(where!necessary)!allowed!the!perspectives!being!offered!to!be!
verified!for!consistency!with!responses!with!earlier!questions.!Having!a!consistent,!base!question!
schedule!also!aided!the!later!comparative!analysis!of!interview!responses!(see!Appendix!1!for!an!
example!of!a!question!schedule!used).!(RIPP:!Article!2,!p.8!also!includes!a!summarised!rationale!
and!example!question!schedule!as!related!to!the!READ!application.)!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 54!
3.3.3! Observation##
The!observation!employed!was!inspired!by!ethnographic!data!collection!applications!and!was!used!
during!field!trips,!evaluation!practice!and!in!my!role!as!READ!facilitator.!Field!trips!related!to!the!
READ!model!application!were!conducted!in!Balaklava!(SA),!Sydney!(NSW),!and!Adelaide!(SA);!with!
benefit!arising!from!concurrent!Nuclear!Futures!READ*related!projects!in!Yalata!(SA)!and!Brisbane!
(QLD).!Field!observation!was!useful!for!corroborating!the!consistency!of!participant!interview!
responses!and!reflective!practice!findings.!For!example,!observation!during!evaluation*related!
meetings!played!an!important!role!for!verifying!stakeholder!evaluation!needs,!values!and!priorities.!
In!addition!to!producing!valuable!data!for!the!research,!the!method!generated!insights!relevant!to!
broader!partnership!development!and!project!management,!which!informed!the!approach!I!adopted!
for!partner!brokerage!and!project!management.!This!conduct!aligned!with!research!by!Hartley!(2008)!
on!effective!processes!for!project!management.!
3.3.4! Evaluation#theory#and#READ#product#development#
Reflective!analysis!undertaken!during!the!research!also!involved!the!process!of!examining!emergent!
evaluation!theory!related!to!the!READ!application,!and!via!my!own!journey!as!practitioner.!This!was!
an!iterative!process!of!data!collection,!organisation,!analysis,!theoretical!sampling!and!product!
development!–!which!was!enhanced!by!deepening!cycles!of!reflective!review.!The!aim!of!developing!
a!theory!of!practice!for!the!emergent!READ!model!was!to!establish!a!transferable!and!generalised!
READ!product!suitable!for!future!evaluation!practice!for!the!next!phases!of!the!Nuclear!Futures!
program.28!!Through!reflective!practice!and!continuous!improvement!during!and!following!the!READ!
application,!evaluation!theory!at!the!site!of!practice,!and!for!my!own!professional!practice,!was!
advanced.!These!findings!are!presented!in!Chapter!6!and!7.!
Figure!3.3!shows!the!simplified!hierarchy!of!the!theory!development!process!in!relationship!to!the!
levels!of!forensic!reflective!practice!employed!during!the!research.!Across!the!diagram!(Figure!3.3),!
each!level!represents!the!refinement!of!evaluation!theory!as!I!worked!towards!developing!the!READ!
model!of!practice,!with!data!being!progressively!examined!through!increasingly!critical!stages!of!
reflection.!As!data!was!analysed!and!refined!using!qualitative!and!reflective!methods,!through!the!
process!of!secondary!reflection!and!coding,!themes!and!theoretical!concepts!emerged.!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
28!A!product!is!defined!as!‘a!good,!idea,!method,!information,!object!or!service!created!as!a!result!of!a!process!and!serves!a!
need!or!satisfies!a!want.!It!has!a!combination!of!tangible!and!intangible!attributes’!(WebFinances,!2016).!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 55!
The!transitioning!quality!of!reflection!can!be!explained!by!Hatton!and!Smith’s!(1995)!four!stage!
taxonomy,!where!reflection!progressively!deepens!from:!
1.!Descriptive:!where!focus!of!reflection!is!limited!to!description!of!events/!activities.!
2.!Descriptive!reflective:!predominantly!descriptive,!with!deeper!consideration!of!events.!!
3.!Dialogic!reflection:!stepping!back!occurs,!evaluation!of!judgments/!actions,!and!reflective!analysis!
links!factors!and!alternative!perspectives.!
4.!Critical!reflection:!frame!of!reference!for!reflection!is!responsive!based!on!new!information,!
emergent!insights!and!time!passing,!it!considers!the!views/!motives!of!others,!in!relation!to!the!
reflective!practitioner’s!own!judgments!and!experience.!!
!
!
Figure!3.3:!Hierarchy!of!theory!development!via!forensic!reflective!practice!
!
! !
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 56!
Through!the!theory!building!process!for!the!READ!model!and!for!advancing!my!own!practice!
(summarised!in!Figure!3.3),!large!quantities!of!qualitative!data!were!focused!and!integrated.!Critical!
reflection!and!in*depth!analysis!were!based!on!a!number!of!interpretative!techniques!(such!as!
coded,!textual!and!comparative!approaches)!to!identify!recurrent!threads!across!the!data!collected.!
One!of!the!key!integrative!techniques!employed!was!categorisation!of!information!by!emergent!
themes,!where!data!from!reflective!methods,!interviews,!and!observation!were!coded,!organised!
and!analysed!for!iterative!development!of!the!READ!model!and!evaluation!theory.!As!results!were!
produced,!I!continued!to!question!and!analyse!findings!through!the!deepening!levels!of!reflection.!
To!assist!with!organising,!testing!and!re*evaluation!during!the!reflective!practice!cycle,!analytical!
frameworks!and!reframing!techniques!were!used!involving!the!consideration!and!analysis!of!
alternative!perspectives!and!interpretations,!based!on!proof!of!concept!discussions!with!partners.!
Practice!and!site!specific!methods!were!also!utilised!to!enhance!critical!reflection!during!theory!
building!and!product!development!(extended!from!the!READ!application!itself),!as!well!as!the!
application!of!several!key!principles!from!systems!theory!to!help!determine!the!inter*relationships!
and!boundaries!between!elements!of!the!READ!model!as!it!developed.!The!systems!theory!approach!
that!I!employed!to!understand!and!improve!the!model!is!explained!in!Chapter!5,!and!can!be!
generally!defined!as!an!approach!to!‘describe!and!interpret!complex!interactive!phenomena…!
emphasizing!the!properties!of!the!relationships!between!the!phenomena!to!be!as!significant!as!the!
properties!of!the!phenomena!themselves’!(Davidson!&!Rowe,!2009,!p.562).29!!!
Finally,!the!timing!and!delivery!schedule!for!the!reflective!theoretical!development!of!the!READ!
model!proved!important.!Scheduling!was!important!for!ensuring!a!systematic!means!of!investigation,!
and!that!adequate!intellectual!space!was!allocated!between!practice!and!reflection.!Breaks!from!
both!reflection!and!practice!were!also!beneficial,!as!sometimes!significant,!spontaneous!insights!
would!arise!in!the!hours!following!a!reflective!session,!while!I!was!doing!non*research!related!
activities,!suggesting!the!value!of!complimentary!unconscious!and!semi*conscious!unstructured!
processes!of!reflection!and!theory!building.!!
! !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
29!The!general!systems!movement!developed!in!the!early!1950s!and!was!influenced!by!leading!thinkers!such!as!Kenneth!
Boulding,!Ludwig!von!Bertalanffy,!Talcott!Parsons,!and!Anatol!Rapoport!amongst!others!(Richardson,!2004).!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 57!
3.4! Research#phases#
The!DCI!research!methodology!outlined!in!this!chapter!(illustrated!in!Figure!3.1)!supported!my!
evaluation!practice!for!the!READ!application!and!enabled!the!critical!reflective!investigation,!review,!
and!iterative!modification!of!the!READ!model!based!on!its!performance!for!the!Balaklava!
Community!Arts!Project.!When!implemented,!the!research!methodology!spanned!a!number!of!
distinct!phases!across!the!two!years!of!the!DCI.!The!progression!of!the!DCI!consisted!of!the!following!
six!research!phases:!
1.! A!contextual!scan!covering!the!DCI!research!topic!and!community!arts!context!(Chapter!2).!
2.! Scoping!undertaken!with!Nuclear!Futures!and!industry!stakeholders!(Chapter!5).!
3.! Planning!and!design!of!the!tailored!READ!methodology!(system)!for!Balaklava!(Chapter!5).!
4.! READ!implementation!(data!collection!and!analysis!for!the!READ!application)!(Chapter!5).!
5.! Production,!dissemination!and!use!of!READ!outputs!for!the!READ!application!(Chapter!5).!
6.! DCI!research!analysis!of!the!data!collected!on!the!READ!application!(Chapter!6).!!
Of!these!six!research!phases,!the!middle!four!phases!(phases!2*5)!were!grounded!in!the!READ!
application!and!related!to!the!planning!and!the!collective!design!and!implementation!of!a!tailored!
READ!framework!at!the!site!of!practice.30!
3.5! Ethical#considerations#
The!research!was!classified!as!low!risk!and!was!implemented!under!QUT!Ethics!Approval!Number!
1400000001.!In!implementing!the!methodology,!measures!were!taken!to!limit!the!ethical!impacts!of!
the!research!(particularly!during!interviews!and!secondary!reflection)!such!as:!maintaining!
participant!confidentiality!and/or!anonymity!where!necessary;!developing!ethical!participation!
consent!and!withdrawal!procedures;!following!approved!protocols!(individual!and!group);!and!
ensuring!participant!safety!and!comfort.!
! !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
30!See!Figure!1.2!in!Chapter!1!for!a!visual!representation!of!the!research!phases.!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 58!
3.6! Conclusion#
For!the!DCI,!a!robust!practice*informed!research!methodology!was!designed!to!enable!data!
collection!and!analysis!relating!to!the!development!and!first!industry!application!of!the!READ!model.!
Forensic!reflective!practice!was!harnessed!as!the!principal!research!strategy,!and!with!a!tailored!and!
systematic!approach!devised!that!involved!reflective!journaling!and!video!diary!entries,!critical!
buddy!sessions,!digital!storytelling,!interviews,!and!observation.!!Guiding!the!critical!review!of!the!
emergent!READ!model!via!these!methods!were!a!number!of!meta*evaluation!and!situation!analysis!
techniques,!employed!to!assess!the!strengths,!weaknesses!and!areas!of!innovation!demonstrated!by!
the!model.!Data!analysis!utilised!theory!development!and!creative!documentation!processes,!in!
order!to!address!the!research!problem!of!the!DCI!and!develop!a!refined!READ!product!of!relevance!
for!future!evaluation!practice.!
The!DCI!methodology!outlined!in!Chapter!3!was!implemented!throughout!the!DCI!and!formed!the!
critical!lens!for!understanding,!developing!and!evaluating!the!READ!model!in!action.!The!following!
chapters!turn!to!focus!on!the!evaluation!practice!undertaken!for!the!READ!application.!!
In!Chapter!4,!I!provide!a!descriptive!account!of!the!site!of!practice!for!the!READ!application.!Profiled!
is!the!Balaklava!Community!Arts!Project,!the!first!in!the!series!of!Nuclear!Futures!projects!
implemented!by!Alphaville!arts!organisation!with!atomic!survivor!communities.!Given!the!context*
specific!nature!of!evaluation!practice,!Chapter!4!provides!the!essential!background!information!on!
the!project!and!partnerships!ahead!of!the!documented!READ!model!development!and!application!in!
Chapter!5.!The!thorough!account!in!Chapter!5!of!the!stakeholder*oriented!design!of!the!READ!
methodology!reveals!the!theoretical!underpinnings!and!the!diverse!suite!of!methods!implemented!
and!trialled!for!the!READ!application!during!the!Balaklava!Community!Arts!Project!that!respond!to!
the!defining!project!elements,!as!canvassed!next!in!Chapter!4.!!
!
! !
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 59!
4! Site#of#practice#
The!DCI!research!methodology!in!Chapter!3!was!implemented!at!the!site!of!my!professional!practice,!
negotiated!across!the!dual!roles!of!READ!facilitator!and!program!manager!for!the!Nuclear!Futures!
Partnership!Initiative!(Nuclear!Futures).!The!READ!model!(integrating!Reflection,!Evaluation,!Analysis!and!
Documentation)!is!central!to!the!research!enquiry!and!is!a!framework!for!learning!and!appraisal!that!was!
devised!in!response!to!the!evaluation!needs!of!the!Nuclear!Futures!program!and!partners.!Nuclear!
Futures!is!a!dynamic,!international!arts!and!cultural!program!using!community!arts!processes!to!develop!
and!showcase!original!creative!work!in!partnership!with!communities!of!interest,!enabled!by!multiple!
strategic!partnerships.!The!communities!of!interest!for!the!Nuclear!Futures!program!are!atomic!survivor!
communities!(affected!by!nuclear!weapons!testing,!and!use)!in!Australia!and!several!countries!overseas.!!
While!the!READ!approach!will!be!designed!and!tested!across!the!suite!of!Nuclear!Futures!projects!
(approximately!10!sub*projects!in!total),!the!scope!for!the!DCI!research!is!contained!to!the!critical!
examination!of!READ!application!for!the!first!Nuclear!Futures!project!*!a!twelve*month!community!arts!
initiative!delivered!in!regional!Balaklava,!South!Australia,!during!2014*15.!The!findings!from!research!on!
the!Balaklava!project!will!inform!the!ongoing!evaluation!practice!for!the!Nuclear!Futures!program.!
This!chapter!provides!an!overview!of!Nuclear!Futures!(the!site!of!practice)!and!describes!the!Balaklava!
Community!Arts!Project,!detailing!the!arts!activities,!relationships!and!creative!works!under!review!by!the!
READ!model.!The!Balaklava!project!aimed!to!promote!community!participation!and!increase!opportunities!
for!residents!within!the!Wakefield!region,!north!of!Adelaide,!to!engage!with!the!arts,!and!to!create,!
develop!and!present!high!quality!creative!works!on!community*led!and!atomic!legacy!themes.!The!result!
was!a!multi*arts!creative!residency!program!with!visiting!and!local!artists!that!culminated!in!a!creative!arts!
showcase!for!the!Adelaide!Fringe!Festival!in!February!2015.!
Given!the!significant!role!that!the!site!of!practice!has!played!in!defining!the!parameters!and!
considerations!of!the!READ!application,!which!transformed!the!original!READ!idea!into!a!working!model,!
an!understanding!of!both!the!Nuclear!Futures!program!and!the!Balaklava*specific!art*making!activities!is!
necessary!in!order!to!contextualise!the!research.!In!telling!this!story,!creative!documentation!from!the!
Nuclear!Futures!community!arts!practice!is!used!in!this!chapter!to!provide!rich!representations!of!the!
community!arts!processes!and!outputs!for!which!the!READ!model!has!been!devised!to!review.!
! !
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 60!
4.1! Nuclear#Futures#Partnership#Initiative#(Nuclear#Futures)#
4.1.1! Background#
Nuclear!Futures!is!a!three*year!collaboration!across!six!countries!involving!linked!multi*arts!and!
cultural!activity!with!atomic!survivor!communities,!to!create!original!story*based!artworks!exposing!
the!legacies!of!the!atomic!age!(Alphaville,!2015).31!!It!is!coordinated!by!Alphaville,!a!Sydney*based!
company!specialising!in!multi*platform!creative!arts,!originating!in!community!development!
processes.!Alphaville’s!program!documentation!describes!Nuclear!Futures!as!follows:!!
The!initiative!creates!new!partnerships!with!arts!organisations,!and!with!educational!and!
cultural!institutions,!research!facilities,!campaign!groups!and!atomic!survivor!and!nuclear!
veteran!communities!in!Australia,!Britain,!Japan,!Marshall!Islands,!Kazakhstan!and!India.!
Through!the!program,!communities!and!artists!use!theatre,!film,!paintings,!sculpture,!
photography,!digital!arts!and!other!art!forms!to!make!creative!works!that!reflect!both!the!
horror!of!living!with!nuclear!radiation,!and!the!resilience!of!communities!as!they!face!the!
nuclear!future.!(Extract!from!Nuclear!Futures!program!documentation,!courtesy!of!Alphaville)!
Led!by!Creative!Producer!Paul!Brown,!Alphaville!secured!support!for!Nuclear!Futures!in!2013!from!
the!Australia!Council!for!the!Arts!via!Creative!Communities!Partnership!Initiatives!(CCPI)!funding!for!
the!three*year!period!2014*2016!(see!Section!2.1.2).!Positioned!within!the!community!partnerships!
sector!of!the!community!arts!field,!the!funding!was!linked!to!program!objectives!to!promote:32!
!! individuals!and!communities!to!actively!participate!in!excellent!artistic!practices�!!! long!term!outcomes!through!arts!and!non*arts!partnerships!
!! capacity!building!in!communities�!
!! innovation!across!agencies!and!communities!in!the!use!of!the!arts!for!community!well!being�!!! understanding!of!the!contribution!of!the!community!arts!and!cultural!development!sector!
(CACD)!through!effective!evaluation!and!dissemination.!
Program!funding!was!also!secured!from!Arts!South!Australia!and!philanthropy,!and!extensive!in*kind!
support!was!received!from!arts!and!non*arts!organisations,!local!government,!industry,!and!
academic!institutions.!Partnering!organisations!for!Nuclear!Futures!includes!Maralinga!Tjarutja!
Council,!Yalata!Aboriginal!Community,!Global!Hibakusha!Project,!Atomic!Photographers!Guild,!British!
Nuclear!Test!Veterans!Association,!Australian!Nuclear!Veterans!Association!network,!Wakefield!
Regional!Council,!State!Library!of!South!Australia,!Allen!&!Unwin!publishing,!In!Place!of!War,!
Manchester!University,!Winchester!University,!Murdoch!University,!Queensland!University!of!
Technology,!University!of!Western!Sydney,!and!University!of!New!South!Wales!(Alphaville,!2015).!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
31!See!promotional!video!for!a!4!minute!summary!of!the!Nuclear!Futures!program:!http://nuclearfutures.org!!
32!This!content!is!sourced!from!program!documentation,!courtesy!of!Alphaville.!Please!note:!some!Alphaville!content!from!
Nuclear!Futures!documentation!and!in!the!public!domain!(website)!used!in!this!dissertation!has!been!authored!by!myself,!
on!behalf!of!Alphaville!in!my!role!as!program!manager,!and/or!in!collaboration!Paul!Brown,!the!other!major!author!of!
Alphaville!content.!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 61!
4.1.2! Exposing#the#legacies#of#the#atomic#age#through#the#creative#arts##
While!international!themes,!activities!and!linkages!are!prominent!across!the!Nuclear!Futures!
projects,!the!program!is!primarily!focused!on!developing!new!creative!works!that!share!community!
stories!of!the!legacies!arising!from!atomic!testing!in!Australia!in!the!1950s!and!60s,!a!history!that!
remains!largely!unknown!to!most!Australians.!!Between!1952!and!1967,!with!the!agreement!of!Prime!
Minister!Menzies,!twelve!atomic!bomb!tests!and!hundreds!of!toxic!minor!trials!were!carried!out!by!
British!scientists!in!Australia!(McClelland,!1985).!The!British!run!tests!were!performed!in!the!Monte!
Bello!islands!off!Western!Australia,!and!at!Emu!Field!and!Maralinga!in!South!Australia!(National!
Archives!of!Australia,!2013)!(see!Figure!4.1).!The!impacts!of!these!tests!continue!to!play!out!in!the!ill!
health!of!the!Indigenous,!nuclear!veteran,!and!British!scientific!communities!exposed!to!or!involved!
in!the!tests,!and!have!left!a!legacy!of!long!term!environmental!contamination!(Yalata!&!Oak!Valley!
Communities!&!Mattingley,!2009;!Cross!&!Hudson,!2006).!!
!
!
Figure!4.1:!Map!of!South!Australia!and!relevant!place*names,!!
including!Emu!Junction!and!Maralinga!atomic!test!sites!
!
(Source:!Yalata!and!Oak!Valley!Communities!and!Mattingley,!2009,!p.68)!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 62!
4.1.3! Nuclear#Futures#program#rationale##
The!impulse!to!work!with!atomic!survivor!communities!via!Nuclear!Futures!arises!from!the!urgency!
to!work!with!the!last!surviving!eye!witnesses!to!the!world’s!atomic!tests,!and!the!desire!to!contribute!
towards!capacity!building!and!recognition!for!nuclear!affected!communities.!It!fits!into!a!long!
tradition!of!artists!responding!to!nuclear!legacies,!atomic!weaponry,!and!issues!from!the!atomic!age!
(internationally!and!within!Australia)33.!The!Nuclear!Futures!initiative!also!builds!on!earlier!work!by!
Alphaville,!namely!a!verbatim!theatre!and!oral!history!project!with!nuclear!veterans!in!Britain!and!
Australia!that!culminated!in!the!script!development!of!the!play!Half%a%Life,!and!performances!in!
Australia!and!the!United!Kingdom!in!2005!and!2006!(see!Brown,!2010,!2007,!2006;!Arvanitakis,!2008).!!
As!stated!in!Alphaville’s!program!documentation,!the!Nuclear!Futures!initiative!has!the!following!
broad!aims:!
(i)!! To!make!the!story!go!far!
(ii)!! To!develop!the!storytelling!networks,!sharing!stories!of!resilience!
(iii)! To!make!the!link!between!‘white’!and!‘black’!experience!
(iv)! To!collaborative!with!and!support!initiatives!by!nuclear!veterans!and!their!families!
(v)! To!dramatise!the!role!of!British!Scientists!
(vi)!! To!build!new!international!collaboration!and!join!the!Global!Hibakusha!project!
(vii)!!To!explore!areas!of!interest!within!the!CACD!field!
(viii)!To!undertake!independent!evaluation!through!data!collection;!qualitative!and!rich!media;!
structured!feedback!and!review;!critical!reflection;!regular,!rich!reporting.!
Creative!Producer!Paul!Brown!explains,!‘Nuclear!Futures!seeks!to!explore!how!experience!of!the!
bomb!translates!into!community!development!and!international!campaigns!for!peace!and!
disarmament,!and!responds!to!the!atomic!legacies!through!community!arts!and!partnerships’.34!
Central!to!the!art*making!is!the!question:!‘What!creative!art!works!and!processes!can!endure!across!
the!deep*time!of!the!nuclear!future?’!(Alphaville,!2015).!
For!delivery,!Nuclear!Futures!was!devised!as!a!sequence!of!four!inter*related!CACD!activities:!
•! community!and!place!based!residencies!(in!Balaklava!and!Yalata/Oak!Valley,!South!Australia)!
•! integrating!arts!and!cultural!projects!across!the!life!of!the!initiative!(international)!!
•! showcases!of!the!program’s!linked!creative!works,!with!associated!events!(international)!
•! support!structures!for!program!engagement,!planning,!coordination!and!evaluation.!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
33!There!is!a!significant!collection!of!creative!works!previously!produced!responding!to!atomic!issues.!Two!well*known!
Australian!examples!include!the!theatre!play!Ngapartji%Ngapartji!(Big!hART,!see!http://www.ngapartji.org/)!and!Maralinga!Sculpture!by!Lin!Onus!(Art!Gallery!of!Western!Australia). 34!Sourced!from!program!documentation,!courtesy!of!Alphaville.!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 63!
These!four!key!program!activity!strands!are!summarised!in!Table!4.1,!with!the!Balaklava!Community!
Arts!Project!highlighted!in!relation!to!the!other!Nuclear!Futures!activities.!!
!
Table!4.1:!Key!activity!strands!for!the!Nuclear!Futures!program!(2014*16)!
!
NUCLEAR+FUTURES+++H+++KEY+ACTIVITY+STRANDS+
Community+and+place+
based+residencies+
Integrating+arts+and+
cultural+projects+
Public+presentation+
and+showcasing++
Support+structures+
Balaklava+project+
Regional!South!Australia+
(CACD!residencies,!
photography,!script!
development,!creative!
development!for!showcase)!
International+youth+
leadership+workshops++
(Digital!storytelling!skills!
development!and!!exchange!!
for!3rd!generation!atomic!
communities!participants)!
Community+
presentations+
(Presentations!of!creative!
works!to!and!in!communities!
of!interest)!
Nuclear+Futures+hub+
(Management!systems!for!
creative!team,!steering!and!
working!groups,!cultural!
liaisons/!advisers)!
Maralinga+project+
Yalata!&!Oak!Valley,!SA+
(CACD!residencies,!cultural!
exchange,!storytelling,!digital!
arts,!sculptures,!creative!
development!for!showcase)!
Online+community+
(Involves!social!forum!and!
exchange,!film/!webisodes,!
website,!games,!transmedia)!
Public+multimedia+events+
(International!workshops,!
film!screenings,!launches,!
festivals,!exhibitions,!
installations)+
Collaborative+networks+
(Communities!of!practice,!
and!circle!of!advisers,!
academics,!creative!artists!
and!other!researchers)!
!
SpinHoff+projects+
(Nagasaki!Peace!Park!
sculpture!gifting,!play!script!
development!with!Monte!
Bello!focus,!radio!play!with!
British!veterans,!Maralinga!
cultural!tourism)!
The+archive+
(Collection!development!
with!cultural!institutions!and!
creative!documentation)!
Integrated+showcases+
(National!and!international!
touring!of!curated!creative!
works/!showcases!to!
regional!and!metropolitan!
audiences)+
Program+and+partnership+
development+
(Includes!building!creative!
community!partnerships!
with!arts!and!non*arts!
partners!and!fundraising)!
! ! Scholarly+&+industry+
presentations+
(Conferences,!seminars,!
forums)!
READ+
(Implementation!of!the!
READ!model!integrating!
Reflection,!Evaluation,!
Analysis,!Documentation)!
!(Table!by!Ellise!Barkley.!Sourced!courtesy!of!Alphaville)!
Through!the!four!activity!strands!in!Table!4.1,!Nuclear!Futures!was!designed!to!produce!a!suite!of!
new!multi*platform!creative!works,!representing!a!register!of!voices!from!various!atomic!survivor!
communities.!New!creative!works!produced!as!a!result!of!the!program!will!include!digital!stories,!
theatre!plays,!books,!a!radio!play,!exhibitions/!installations,!sculpture,!online!community,!integrated!
showcase!and!an!enduring!archive!of!historical!and!creative!materials.!Community!artworks!and!
creative!works!produced!in!partnership!with,!and!independently!by,!professional!artists!will!be!
curated!into!an!international!showcase!planned!for!2016!onwards,!following!the!presentations!in!
communities!completed!in!2015.!(See!www.nuclearfutures.org!for!further!program!information.)!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 64!
4.2! Balaklava#Community#Arts#Project#
The!Balaklava!Community!Arts!Project!was!the!first!of!two!major!community!and!place*based!
projects!implemented!for!Nuclear!Futures.!The!second!was!a!series!of!creative!projects!undertaken!
with!the!Yalata!Anangu!Aboriginal!community!in!far!west!South!Australia!(see!Figure!4.1).!!
4.2.1! Origins#and#inspiration#
Research!for!the!Balaklava!Community!Arts!Project!began!in!December!2013!with!nuclear!veteran!
Avon!Hudson!and!his!daughter!Kate,!both!residents!of!the!small!regional!community!of!Balaklava.!
Balaklava!is!an!agricultural!township!in!the!Mid!North!Region!of!South!Australia,!situated!92!km!
north!of!Adelaide.!It!has!a!population!of!under!2000!residents!and!is!a!cultural!hub!within!the!
surrounding!Wakefield!regional!area!(see!Figure!4.2)!(Wakefield!Regional!Council,!2015).!!
!
!
!
!
!
!
!
!
!
!
!
!
!
Figure!4.2:!Map!of!Balaklava!and!Wakefield!Regional!Council!area,!South!Australia!
!
(Source:!http://profile.id.com.au/wakefield/about?WebID=10)!!
!
!
!
Balaklava!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 65!
The!project!began!in!Avon’s!Balaklava!living!room!amidst!his!private!collection!of!rich!and!rare!
archival!materials!relating!to!the!Australian!atomic!tests,!amassed!from!decades!of!social!justice!
campaigning!on!this!issue!(see!image!below).!Avon!had!served!in!the!Royal!Australian!Air!Force!
(RAAF)!at!Maralinga!in!the!early!1960s,!and!blew!the!whistle!in!the!1970s!when!he!publicly!revealed!
Maralinga’s!deadly!legacy,!an!act!that!contributed!to!the!establishment!of!the!Royal!Commission!into!
British!Nuclear!Tests!in!Australia!(1984*85).!Over!sixty!years,!Avon!has!been!a!prominent!antinuclear!
campaigner!and!advocate!for!Australia’s!nuclear!veteran!community,!and!was!a!leader!within!the!
Australian!Nuclear!Veterans!Association!(ANVA).!His!nationally!significant!private!archive!remains!an!
important!legacy!of!his!activism!and!was!a!cornerstone!resource!for!the!community!arts!project.!
!
Balaklava!community!partner,!nuclear!veteran!and!whistleblower!Avon!Hudson.!!
Photo!by!Jessie!Boylan.!
!
It!was!in!Balaklava,!Avon’s!hometown,!that!the!CACD!activities!and!community!showcase!for!the!
project!were!delivered.!Avon’s!energy!for!participating!in!the!proposed!project,!his!extensive!
knowledge!of!nuclear!veteran!related!subject!matter!and!networks,!and!his!generosity!for!sharing!his!
testimonies!via!oral!history,!were!formative!factors!for!grounding!the!project!in!Balaklava.!!
+
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 66!
4.2.2! Project#partners#and#stakeholders##
The!process!of!developing!the!creative!community!partnerships!for!the!Balaklava!Community!Arts!
Project!was!facilitated!by!arts!company!Alphaville!and!targeted!relationships!with:!relevant!partner!
organisations!and!communities!of!interest!(to!be!directly!associated!and!involved!in!the!development!
and!delivery!of!the!project);!funding!partners!(contributing!financially!to!the!project);!stakeholders!
(those!potentially!affected!by!or!with!a!stake!in!the!project!in!some!way);!and!potential!project!
champions/!allies!(strategic!alliances!to!help!promote,!support,!and!endorse!the!project).!The!project!
was!financially!supported!by!the!Australia!Council!for!the!Arts!and!Arts!SA,!and!involved!core!
partnerships!with!community!representatives!within!the!Australian!nuclear!veteran!network!in!South!
Australia!and!organisations!such!as!the!Wakefield!Regional!Council,!the!Council’s!Youth!Advisory!
Committee,!and!State!Library!of!South!Australia!(RIPP:!Article!28,!p.106!is!a!partner!summary).!!
Community!consultation!with!stakeholder!groups!such!as!local!government,!regional!arts,!
progress!committees,!and!local!cultural!and!youth!service!organisations!was!undertaken!during!
the!planning!phase!so!that!the!arts!activities!devised!aligned!with!regional!strategic!objectives!and!
local!priorities.!Contact!was!also!made!with!nuclear!veteran!communities!in!Australia!and!overseas,!
via!known!Australian!networks!and!the!British!Nuclear!Test!Veteran!Association.!It!was!established!
that!the!project!would!target!two!key!communities!of!interest:!
1.! South!Australian!nuclear!veteran!networks,!with!Avon!and!Kate!as!the!key!community!
partners.!
2.! Regional!participants!and!the!Balaklava!community,!with!a!focus!on!engaging!young!people!
in!the!Wakefield!region.!!
During!implementation,!partnership!brokerage!and!stakeholder!engagement!continued!across!all!
project!phases.!The!partnerships!with!local!and!regional!arts!and!non*arts!organisations!brought!
significant!in*kind!contribution,!with!an!estimated!value!of!$40,000.!This!included:!expertise!in!
community!cultural!development;!event!management;!oral!history;!archiving;!mentoring/training;!
and!access!to!local!arts!and!cultural!programs,!networks,!community!groups,!equipment!and!venues.!
In!addition!to!the!core!partners!listed!above,!organisations!contributing!in*kind!support!included!the!
Balaklava!Courthouse!Gallery,!Balaklava!Town!Hall,!Balaklava!Men’s!Shed,!Vitalstatistix!(Port!
Adelaide),!University!of!Adelaide,!British!Nuclear!Test!Veteran!Association,!and!the!Blackman!Gallery!
(Clare,!SA).!Avon!Hudson’s!established!veteran!and!media!networks!also!proved!very!valuable!for!
extending!partnership!development!and!advancing!networking!with!several!South!Australian!nuclear!
veteran!families!(initially!unknown!to!the!project).!!
!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 67!
4.2.3! Project#overview#
The!Balaklava!project!aimed!to!promote!community!participation!and!increase!opportunities!for!
residents!within!the!Wakefield!region!to!engage!with!the!arts,!achieve!long*term!outcomes,!and!
encourage!innovation!between!arts!and!non*arts!partners!(see!objectives,!p.60).!For!the!project,!this!
involved!creating,!developing!and!presenting!quality!creative!works!on!community*led!and!atomic!
legacy!themes!(particularly!nuclear!veteran!themes!relating!to!South!Australia’s!atomic!history).!As!
per!the!funding!and!partner!agreements,!the!Balaklava!Community!Arts!Project!set!out!to!deliver:!
•! An!arts!residency!using!CACD!processes,!to!foster!local!cultural!development!
•! Collaborative!and!community*driven!art*making!!
•! Showings!of!the!work*in*progress!to!communities!and!stakeholders!
•! A!community!showcase!of!created!work!–!a!contemporary!multi*arts!work!of!excellence!
•! An!accessible!and!enduring!archive!of!the!evidence!on!which!creative!works!are!based!
•! Findings!based!on!Reflection,!Evaluation,!Analysis!and!Documentation!(READ).!
A!core!creative!team!was!assembled!to!work!with!the!communities!of!interest.!The!Nuclear!Futures!
team!included!Director!Teresa!Crea,!Photomedia!Artist!Jessie!Boylan,!Playwright!John!Romeril,!Digital!
Artist!Linda!Dement,!Projection!Designer!Nic!Mollison,!Sound!Designer!Luke!Harrald,!Creative!
Producer!Paul!Brown,!Production!Manager!Ellise!Barkley,!Audio*visual/Projection!Technician!Rowan!
Lee!and!Local!Coordinator!Carly!Friedrichs.!!
Conceived!were!a!mix!of!creative!residencies!to!be!based!in!Balaklava!involving!visiting!and!local!
artists,!as!well!as!creative!development!and!project!work!conducted!in!Adelaide!and!Sydney!to!
support!the!residencies!and!art*making.!Five!parallel!community!arts!initiatives!proceeded:!!
1.!Photography!skills!development!workshops!and!a!community!exhibition,!facilitated!by!
Photomedia!Artist!Jessie!Boylan!and!local!Photographer!Lisa!Redpath.!
2.!Script!development!for!a!new!play,!led!by!playwright!John!Romeril.!
3.!A!multi*media!immersive!projection!installation,!led!by!Director!Teresa!Crea.!
4.!A!co*curated!photomedia!and!archival!exhibition,!including!artworks!by!Jessie!Boylan.!
5.!Archive!development!and!oral!history!collection!with!State!Library!of!South!Australia!(SLSA).!
Across!March!2014!*!February!2015,!fifteen!professional!creative!personnel!were!contracted!for!the!
CACD!residencies!and!final!showcase!in!Balaklava!(nine!of!the!artists!and!technicians!were!South!
Australian).!Additional!personnel!were!also!involved!via!partner!organisations,!such!as!staff!from!
Wakefield!Regional!Council!and!Collection!Development,!SLSA.!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 68!
The!creative!partnership!established!between!participating!artists!and!community!developed!over!six!
week*long!residencies,!as!well!as!extended!rehearsal!and!production!periods.!The!artist!in!residence!
program!utilised!community!arts!processes!such!as!co*creative!art*making,!storytelling,!public!
engagement,!skills!development,!community!research,!network!building,!mentoring!and!professional!
development*!involving!participants!of!all!ages.!Creative!development!was!undertaken!via!
workshops,!meetings,!oral!history!collection,!interviews,!field!research,!networking!and!immersion!in!
Avon!Hudson’s!private!archives.!Representatives!from!Australia’s!nuclear!veteran!community,!young!
people,!the!wider!Wakefield!community,!and!professional!artists!from!Balaklava,!South!Australia!and!
other!states,!were!engaged.!
The!creative!works!developed!were!presented!back!to!communities!of!interest,!and!publicly!to!
general!audiences.!Presentation!of!new!works!included!two!gallery!exhibitions:!Field%of%View%and!
Portrait%of%a%Whistleblower;!and!a!dynamic!multi*arts!showcase!10%Minutes%to%Midnight%for!the!2015!
Adelaide!Fringe!Festival,!featuring!an!assemblage!of!immersive!projection,!digital!artwork,!
photomedia,!sound!design,!archival!material,!interwoven!with!a!lively!speakers’!program.!Linked!to!
this!was!a!two*day!schools!program!to!engage!local!students.!In!Balaklava,!the!project!engaged!
approximately!400!residents!as!participants!and/or!audience.!!
The!aforementioned!project!activities,!outputs!and!partnerships,!resulted!in!myriad!personal,!social!
and!cultural!outcomes!and!impacts!via!the!community!arts!processes!and!exchange.!The!socio*
dynamic!pulse!underpinning!the!collaborative!creative!practice!was!driven!by!an!iterative!process!of!
learning,!sharing!and!responding!to!the!sensitive!political!and!emotional!subject!matter!of!Australia’s!
atomic!test!legacies,!as!well!as!reflection!on!locally!relevant,!place*based!themes!as!part!of!Field%of%
View.!The!result!was!a!sliding!scale!of!transformative!and!personal!impacts!for!participants!and!the!
creative!team,!as!well!as!benefits!for!the!wider!Balaklava!community.!
Across!the!project,!media!coverage!profiled!significant!milestones!and!events!(in!publications!such!
as!BAside%Magazine,!The%Advertiser,%Plains%Producer,%Northern%Angus,%The%Clothesline!(online)!and!on!
Radio!Adelaide!and!ABC!891).!Mainstream!media,!social!media!and!community!newsletters!were!
used!to!profile!the!creative!development!process,!regional!arts!events,!participant!stories,!and!
showcase!highlights.!Overall,!the!media!coverage!extended!the!public!storytelling!platform!for!
Nuclear!Futures’!community!partners!and!artists,!and!included!feature!articles!on!individual!
community!and!team!members,!and!journalism!on!Australian!atomic!test!history!and!its!
contemporary!socio*political!context.!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 69!
4.2.4! Summary#of#the#Balaklava#community#arts#practice##
This!section!elaborates!on!the!art*making!processes,!activities!and!partnerships!at!the!site!of!practice!
in!relation!to!the!key!project!elements!for!the!Balaklava!Community!Arts!Project.!Creative!
documentation!from!the!arts!practice!of!visiting!and!local!artists!is!included.!
Summarised!over!the!following!pages!are!snapshots!of:!
I.! What’s%the%Story?!Creative!arts!pop*up!
II.! Photography!skills!development!and!community!exhibition!
III.! Script!development!
IV.! Portrait%of%a%Whistleblower,%curation!and!exhibition!
V.! 10%Minutes%to%Midnight!installation!
VI.! Public!showcase:!Adelaide!Fringe!Festival!!
VII.! Secondary!schools!program!
VIII.! Archive!development!and!oral!history!collection.!
+
!
! !
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 70!
i.! What’s+the+Story?+Creative+arts+popHup+(17H25+March,+2014)++
Activities!included!a!storytelling!workshop,!film!screening,!school!visits/!workshops!with!
students,!shop*front!art!displays,!community!archive!mentoring,!and!a!local!business!portrait!
series!where!professional!photos!were!taken!and!gifted!back!to!participants.!The!residency!
was!delivered!by!4!artists/art!workers,!community!partners,!and!SLSA!staff.!The!activities!
were!designed!to!initiate!community!engagement!via!meet*the*artists!events,!relationship!
building, skills!development!and!community!visioning.!
!
A.!Playwright!John!Romeril!with!senior!
Drama!students!at!Balaklava!High!School.!!!
B.!Example!from!the!pop*up!exhibition!of!
atomic!artwork!by!Jessie!Boylan!displayed!in!
Balaklava!shopfronts.!
C.!Creative!arts!pop*up!branding.!
Photos!by!Jessie!Boylan.!
!
!
!
A.+ B.+
C.+
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 71!
ii.! Photography+skill+development+and+community+exhibition+(June+–+November,+2014)+
Photographers!Jessie!Boylan!(visiting!artist)!and!Lisa!Redpath!(Balaklava)!teamed!up!to!run!
two!free!workshops!with!27!participants!for!photography!skills!development.!An!evening!
workshop!targeted!general!community!participants,!and!an!all!day!field!trip!for!young!people!
was!also!held.!Workshop!participants!were!invited!to!submit!entries!for!a!community!
photography!exhibition,!and!artists!Boylan!and!Redpath!co*curated!the!selected!works!
culminating!in!the!Field%of%View!exhibition!held!at!the!Courthouse!Gallery,!25th!September!*!
9th!November,!2014.!The!original!photography!by!local!and!young!creatives!included!diverse!
and!stunning!images,!responding!to!sites!that!have!been!part!of!their!field!of!view!while!
living,!or!growing!up!in!the!Wakefield!region.!The!exhibition!was!officially!opened!by!the!
Mayor!of!Wakefield!Regional!Council!and!was!strongly!supported!by!Wakefield!Regional!
Council!and!their!Youth!Advisory!Committee.!!
!
!
!
! !
A.!Excerpt!from!promotional!poster,!Field%of%View!photography!exhibition.!!Design!by!Ellise!Barkley.!
B.!Workshop!#1!participants.!!
Photo!by!Lisa!Redpath.!
C.!Workshop#2!participants!during!field!
excursion.!Photo!by!Carly!Friedrichs.!
!
!
!
A.+ B.+
C.+
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 72!
iii.! Script+development+(December+2013+–+March+2015)+
!
!
!
!
!
!
!
!
!
!
Writer!John!Romeril!and!Avon!Hudson!worked!together!over!several!residencies!in!2014!at!
Avon's!home!in!Balaklava!as!part!of!the!creative!research!and!script!development!process.!As!
part!of!the!fieldwork,!Romeril!accompanied!Hudson!on!the!2014!Radioactive!Exposure!Tour!
(organised!by!Friends!of!the!Earth)!of!nuclear!sites!in!remote!South!Australia,!to!hear!Hudson!
as!guest!presenter!sharing!his!testimony!and!extensive!historical!knowledge!of!nuclear!
testing!in!Australia.!Director!Teresa!Crea!and!artist!Jessie!Boylan!were!involved!in!the!early!
concept!development!for!the!script,!and!Creative!Producer!Paul!Brown!was!a!key!
collaborator!across!the!drafting!process.!In!developing!the!original!dramatisation,!
documentary,!verbatim!and!multi*media!art!techniques!were!used.!
The!draft!script!developed!was!a!dynamic,!operatic!piece,!inspired!by!the!personal!and!
intergenerational!stories!of!nuclear!veterans!and!their!families.!Early!iterations!of!the!script!
were!formative!in!the!concept!development!for!the!10%Minutes%to%Midnight!projection!
installation!(see!below),!and!closely!linked!with!the!SLSA!supported!community!archive!
activities.!The!script!was!workshopped!in!August!2015,!with!two!actors!and!a!Director,!
involving!Drama!students!and!staff!at!Queensland!University!of!Technology!(QUT)!as!part!of!a!
Nuclear!Futures!exhibition!and!event!program!at!The!Block,!QUT.!
!Creative!research!during!an!artist!residency!in!Balaklava.!Photo!by!Jessie!Boylan.!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 73!
iv.! Portrait'of'a'Whistleblower,+curation+and+exhibition+(October+2014+–+February+2015)+%
%
%
%
%
%
%
%
%
%
%
%
%
% %
%
Portrait%of%a%Whistleblower!chronicles!Avon!Hudson’s!life,!combining!contemporary!
photographs!and!video!works!by!artist!Jessie!Boylan!with!pieces!from!Avon’s!private!archive!
of!rare!documentation,!reports,!testimonies,!news!clippings,!photographs,!maps!and!
artefacts!that!make!up!his!life’s!work.!The!exhibition!was!co*curated!over!several!residencies!
by!Jessie!Boylan,!Ellise!Barkley,!Avon!Hudson!and!Paul!Brown,!and!was!programmed!within!
the!2015!Adelaide!Fringe!Festival.!Officially!opened!on!Friday!13th!February,!2015,!the!
exhibition!ran!over!six!weeks!until!29th!March!and!was!well!attended!by!local,!regional!and!
interstate!audiences.!The!exhibition!is!planned!to!tour!in!Australia!and!internationally!from!
2015!onwards!via!the!Nuclear!Futures!program.!
! !
!
A.!Example!of!archival!
material!included!in!the!
exhibition.!Photo!of!Avon!
Hudson!in!front!of!
Lockheed!Neptune!
aircraft,!Darwin!Airbase,!
NT,!1958.!Photographer!
unknown.!
!
B.!Photomedia!Artist!
Jessie!Boylan!and!Avon!
Hudson!in!the!
Courthouse!Gallery,!
Balaklava,!sitting!
amongst!Portrait%of%a%Whistleblower.!Photo!by!Tania!Safi.!
!
!
!
A.+
B.+
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 74!
v.! 10'Minutes'to'Midnight+projection+installation+(July+2014+–+February+2015)+!
!
!
!
!
!
! !!
A.!Inside!the!immersive!
projection!installation.!Digital!
artwork!by!Linda!Dement.!!
Photo!by!Jessie!Boylan.!
B.!Projection!mapping!and!
technical!set*up.!!
Photo!by!Nic!Mollison.!
C.!External!view!of!custom!built,!
surround!projection!screen.!!
Photo!by!Tania!Safi.!
D.!Projected!video!art!by!Jessie!
Boylan.!Photo!by!Tania!Safi.!
!
! ! !
A.+
B.+ C.+ D.+
A!multi*media!projection!installation!was!developed!to!offer!audiences!an!immersive!
experience!of!South!Australia’s!atomic!test!stories,!integrating!rare!archival!footage!and!oral!
history!with!original!digital!artwork,!projection!and!sound!design.!The!24*minute!creative!
work!applied!cutting!edge!immersive!techniques!to!create!a!dramatic!‘countdown’!to!the!
atomic!bomb!tests!conducted!at!Maralinga!and!Emu!Field.!The!projection!installation!was!
developed!over!seven!months!with!a!team!of!leading!Australian!artists,!at!Balaklava!and!at!
Vitalstatistix!in!Port!Adelaide.!Creative!development!was!led!by!Teresa!Crea,!a!specialist!in!
audience!immersive!techniques,!and!evolved!in!association!with!the!script!development!and!
community!archive!projects.!!
The!creative!team!involved!Teresa!Crea!(Director,!Dramaturg),!Nic!Mollison!(Projection,!
Lighting!&!Set!Designer),!Linda!Dement!(Digital!&!Multi*media!Artist),!Luke!Harrald!(Sound!
Designer!&!Composer),!Jessie!Boylan!(Photomedia!&!Video!Artist),!John!Romeril!(Research)!
and!Avon!Hudson!(Community!Partner).!Production!team!included!Paul!Brown!(Creative!
Producer),!Ellise!Barkley!(Production!&!Event!Manager),!Carly!Friedrichs!(Local!Coordinator),!
Rowan!Lee!(Programmer!&!Technician),!Craig!Clifford!(Set),!and!Amanda!Brown!(Seamstress).!
The!work!created!constitutes!a!core!element!of!the!future!Nuclear!Futures!showcase.!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 75!
vi.! Public+showcase:+Adelaide+Fringe+Festival+(13H14+February,+2015)++
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
A!large*scale!community!showcase!for!
Wakefield!regional!audiences!was!organised!
to!present!back!to!community!the!story*based!creative!works!developed!through!the!
Balaklava!Community!Arts!Project.!Titled!10%Minutes%to%Midnight,!the!2015!Adelaide!Fringe!
Festival!ticketed!event!ran!across!two!venues!and!included!the!immersive!projection!
installation,!Portrait%of%a%Whistleblower!exhibition!and!a!Speakers!Corner!involving!
representatives!from!atomic!survivor!communities,!experts!and!creative!artists.!The!
assemblage!of!original!creative!works!was!the!culmination!of!the!twelve*month!partnership!
between!the!creative!team,!community!partners!and!participating!organisations!across!all!of!
the!Balaklava!projects.!10%Minutes%to%Midnight!was!attended!by!over!140!people,!and!
attracted!local,!Adelaide!and!interstate!audiences.!The!event!provided!the!opportunity!for!
the!new!creative!works!to!be!presented!to!mainstream!community!audiences,!and!was!a!rare!
networking!opportunity!for!attending!atomic!survivors!and!families.! !
!
! !
!
!
A.!Speaker!panel.!
B.!Speakers!Corner!session.!
C.!Audience!visiting!Courthouse!Gallery.!
D.!Nuclear!veteran!and!descendent!exchange.!
E.!Archive!materials!were!brought!by!nuclear!
veteran!audience!members!for!sharing.!
Photos!by!Tania!Safi.!
!
A.+ B.+
C.+ D.+
E.+
D.+
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 76!
vii.! Secondary+schools+program+(10/12+February,+2015)+
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
Over!two!full!days,!a!tailored!education!and!engagement!program!was!offered!for!local!
secondary!schools.!The!program!engaged!220!local!students!from!two!Balaklava!high!schools,!
and!linked!with!the!curriculum!for!Design!and!Technology,!History,!Art!and!Tourism.!Students!
had!the!opportunity!to!view!the!projection!installation,!learn!about!Australia’s!nuclear!test!
history,!have!a!guided!tour!of!the!Whistleblower%exhibition,!hear!eyewitness!accounts!of!the!
atomic!experiments,!and/or!meet!members!of!the!creative!team!to!discuss!the!art*making!
and!technical!aspects!of!the!showcase.!Discussion,!questions!and!feedback!indicated!that!for!
many!students!this!was!the!first!time!they!had!heard!of!Australia's!atomic!test!history.!
!
!
Local!secondary!school!students!attending!10%Minutes%to%Midnight.!Photos!by!Tania!Safi.!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 77!
viii.! Archive+development+and+oral+history+collection+(March+–+May+2015)+
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
Running!in!parallel!with!the!afore*mentioned!arts!activities!was!a!community!archives!
project,!supported!and!mentored!by!SLSA.!The!archiving!and!oral!history!work!initiated!the!
process!of!curating!Hudson’s!private!collection!to!determine!pathways!for!the!future!public!
storage!of!the!material.!It!was!one!of!several!legacy!initiatives!within!the!Nuclear!Futures!
program,!to!support!public!and!long*term!access!to!the!significant!material!and!resources!
unearthed!during!the!creative!research.!The!archives!project!facilitated!the!establishment!of!
a!new!collection!at!SLSA!to!house!material!donated!by!the!nuclear!veteran!community!and!
established!a!relationship!between!SLSA!and!the!Hudson!family!for!future!donation.!
!
!
A.!Avon!Hudson!with!State!Library!of!South!
Australia!and!Nuclear!Futures!staff.!!
B.!Slides!of!Maralinga!from!Avon!Hudson’s!
private!archive.!!
Photos!by!Jessie!Boylan.!
A.+
B.+
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 78!
4.3! Conclusion#
The!Balaklava!Community!Arts!Project!aimed!to!promote!community!participation!and!increased!
opportunities!for!residents!in!the!region!to!engage!with!the!arts.!Across!2014*15!the!project!
delivered!an!arts!residency!program!to!foster!cultural!development!and!collaborative,!community*
driven!art*making.!Local!activities!included!public!events,!school!visits,!skill!development!workshops,!
script!development,!community!archiving,!oral!history!collection!and!the!production!of!original!
creative!works!sharing!community!stories.!Public!presentation!of!new!works!included!two!gallery!
exhibitions:!Field%of%View%and!Portrait%of%a%Whistleblower%and!a!dynamic!multi*arts!showcase!as!part!
of!the!2015!Adelaide!Fringe!Festival.!The!showcase!was!an!assemblage!of!digital!artwork,!photo!and!
video!media,!sound!design,!rare!archival!material,!interwoven!with!lively!panel!and!speaker!sessions.!
Featured!was!the!20*minute!immersive!projection!installation!10%Minutes%to%Midnight.!
The!project!engaged!an!estimated!400!people!in!the!region,!via!workshop!participation!and!event!
attendance.!Media!coverage!extended!the!public!storytelling!platform!for!community!partners!and!
artists,!and!included!feature!articles!on!individual!community!and!team!members,!and!journalism!on!
Australian!atomic!test!history!and!its!contemporary!socio*political!context.!Some!project!outcomes!
have!long!term!implications,!such!as!the!resultant!State!Library!of!South!Australia!collection!on!
South!Australia’s!atomic!test!history.!Many!of!the!story*based!creative!works!generated!from!the!
project!will!also!tour!in!future!showcases!as!part!of!Nuclear!Futures!in!Australia!and!overseas.!
In!order!to!understand!and!appraise!the!activities,!outputs!and!impact!of!the!Balaklava!Community!
Arts!Project,!we!set!out!to!develop!an!integrated!methodology!for!our!evaluation!practice,!to!fit!the!
needs!of!the!project!and!the!Nuclear!Futures!program!more!broadly.!The!READ!model,!integrated!
Reflection,!Evaluation,!Analysis!and!Documentation!(READ)!to!meet!the!stakeholder!requirements!for!
accountability*driven!evaluation!findings!as!well!as!a!means!to!reveal!the!broader!implications!of!the!
creative!practice.!While!the!focus!of!this!research!is!the!Balaklava!application!of!the!READ!concept,!it!
was!envisaged!that!the!model!developed!during!the!DCI!would!be!applicable!for!the!suite!of!inter*
related!CACD!activities!across!Nuclear!Futures.!Different!READ!methodologies!have!been,!and!will!be,!
designed!and!implemented!per!community*based!project,!as!well!as!for!the!international!activities.!!
Next,!in!Chapter!5,!an!in*depth!account!of!the!READ!framework!designed!and!implemented!for!the!
Balaklava!arts!project!is!provided.!In!order!to!be!effective,!the!READ!methodology!needed!to!be!
engaging!and!credible,!dealing!with!the!epistemological!and!technical!challenges!raised!by!the!
complex!multi*arts!practice!and!partnerships,!as!well!as!extend!its!critical!lens!beyond!the!project!
processes!and!outcomes!so!as!to!generate!findings!on!the!impact!of!the!arts*based!activities.!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 79!
5! The#READ#application,#Balaklava#(2014S15)#
“Feed!on!reality,!so!you!can!take!an!impression!of!it,!!
!so!we!can!stand!back!and!say!‘yeah!that!is!how!it!was’.”!
John!Romeril!(Playwright,!Nuclear!Futures!creative!team!member!for!Balaklava!Community!Arts!Project)!!
Chapter!5!examines!the!stakeholder*oriented!evaluation!practice!undertaken!for!the!Nuclear!
Futures!Partnership!Initiative!(Nuclear!Futures),!an!international!arts!and!cultural!program!
partnering!artists!with!atomic!survivor!communities,!funded!by!the!Australia!Council!for!the!Arts.!
Detailed!in!this!chapter!are!the!collaborative!processes,!activities!and!methods!that!were!developed!
and!implemented!for!the!first!industry!application!of!the!READ!model,!for!which!I!was!the!key!expert!
and!facilitator!in!my!role!as!researcher!for!the!Doctor!of!Creative!Industries!(DCI).!Arising!from!
professional!practice,!the!conceptualised!READ!model!aimed!to!provide!a!wide!summative!lens!for!
evaluating!community!arts,!by!extending!evaluation!practice!across!four!broader!strategies!of!
Reflection,!Evaluation,!Analysis!and!Documentation!(READ).!It!was!envisaged!that!an!integrated!
approach!to!evaluation!would!better!equip!practitioners!and!community!participants!to!reflect!on,!
represent,!and!appraise!their!art*based!activities!and!partnerships,!for!shared!learning!in!context.!
My!role!as!researcher!involved!leading!the!design!and!implementation!of!the!tailored!READ!
approach!for!the!Balaklava!Community!Arts!Project!with!stakeholders!at!the!site!of!practice.!In!this!
role,!I!was!responsible!for!the!brokering!of!collective!input,!consolidating!the!methodology,!
certifying!the!validity!and!robustness!of!the!emergent!methodology,!and!ensuring!ethical!and!
appropriate!conduct.!While!I!refer!to!myself!as!the!READ!facilitator!in!the!research,!it!is!important!to!
state!that!this!was!not!a!neutral!role,!and!was!rather!one!of!active!direction!and!participation,!
involving!significant!intellectual!contribution!from!me!as!the!key!evaluation!expert!and!practitioner*
researcher.!Further,!formulating!the!READ!methodology!based!on!stakeholder!input!required!me!to!
span!the!domains!of!creative!practice,!program!management!and!evaluation!practice.!
Documented!here!are!the!‘what’,!‘why’!and!‘how’!aspects!of!the!developed!READ!model!in!action.!
To!articulate!the!READ!model!as!it!was!enacted!for!the!Balaklava!project,!I!describe!and!systematise!
the!extensive!body!of!work!resulting!from!the!trial!and!explain!the!key!developments!in!the!model!
of!practice!developed,!based!on!the!observation,!stakeholder!consultation,!reflection,!and!analysis!I!
conducted!as!READ!facilitator/!DCI!researcher.!In!consolidating!the!first!READ!methodology!of!its!
kind,!I!was!able!to!establish!an!original!and!substantial!model!of!practice,!and!address!some!of!the!
persistent!issues!for!evaluation!of!community!arts,!as!identified!in!the!Contextual!Scan!(Chapter!2),!
by!exploring!the!integration!of!diversified,!mix!method!approaches!to!identify!and!represent!impact.!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 80!
A*guide*to*the*READ*application*
Chapter!5!is!divided!into!two!parts.!Part!A!overviews!the!preliminary!activities!from!the!initial!phase!
of!the!READ!application!for!the!Balaklava!Community!Arts!Project,!and!Part!B!covers!the!tailored!
design!and!implementation!of!the!READ!approach!as!a!model,!collaboratively!devised!at!the!site!of!
practice.!Accompanying!the!chapter!is!the!READ%in%Practice!portfolio,!a!compilation!of!original!
documentation!from!the!evaluation!practice.!Combined,!the!chapter!and!READ%in%Practice%portfolio!
provide!insight!into!the!integrated!READ!model’s!scope!and!capability!in!the!community!arts!setting.!!
Part+A+–+Envisaging+the+READ+model+and+preliminary+activities+
The!chapter!begins!with!the!background!story!of!the!READ!concept!prior!to!this!research,!and!
introduces!a!basic!systems!explanation!of!the!READ!model!that!I!developed!in!preparation!for!the!
application!(Section!5.1).!Six!core!elements!of!the!READ!concept!are!identified,!and!these!become!
the!roadmap!for!the!chapter!as!each!is!discussed!in!turn!in!Part!B.!The!six!READ!system!elements!
charted!are:!the!READ!support!structures;!Reflection;!Evaluation;!Analysis;!Documentation;!and!the!
integration,!utilisation!and!dissemination!of!READ!outputs.!The!discussion!includes!preliminary!
assumptions!and!approaches!determined!for!the!Balaklava*based!application,!including!the!
interpretative!stance!and!the!internal!positioning!of!the!evaluation!practice!within!the!arts!program.!!
Part!A!also!positions!the!READ!application!within!the!overall!DCI!research!progression,!to!provide!a!
guiding!narrative!on!the!research!phases!that!underpinned!the!journey!of!my!evaluation!practice!as!
READ!facilitator.!The!section!concludes!with!a!discussion!on!the!findings!arising!from!the!first!phase!
of!the!READ!application,!the!process!of!scoping!with!stakeholders,!that!critically!informed!the!
collective!design!and!delivery!of!READ!in!action.!
Part+B+–+READ+methodology+in+action++
Part!B!covers!the!systems!investigation!of!the!applied!READ!model,!and!provides!rich!descriptions!
and!practical!examples!of!the!emergent!methodology,!methods,!and!deliverables!for!the!READ!
application.!Throughout!the!chapter!I!cross*reference!to!examples!in!the!suite!of!original!materials!
in!the!READ%in%Practice!portfolio!(RIPP),!sign*posted!by!the!icon!to!the!right.!Alternatively,!the!RIPP!
can!be!viewed!separately!as!a!standalone!document!of!industry!examples.!In!studying!the!READ!
model!in!action,!Part!B!opens!with!an!examination!of!the!organisational!scaffolding!necessary!for!
the!model’s!implementation!in!the!Balaklava!context,!which!I!refer!to!as!the!READ!support!
structures.!Next,!each!of!the!four!enquiry!strategies!for!READ!are!described!to!illuminate!how!each!
was!implemented!for!the!project.!Lastly,!the!processes!employed!for!integrating,!utilising!and!
disseminating!the!READ!data!and!outputs!are!canvassed.!Revealed!in!Part!B!are!the!operations,!
functionality!and!outcomes!of!the!READ!model!during!the!Balaklava!experience.!!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 81!
Part#A:#Envisaging#the#READ#model#and#preliminary#activities#
5.1! Background#
5.1.1! The#READ#model##
The!story!of!the!READ!model’s!inception!starts!at!the!site!of!my!professional!practice,!twelve!months!
prior!to!the!commencement!of!the!Balaklava!Community!Arts!Project!(South!Australia).!The!
integrated!READ!concept!arose!from!the!community!arts!practice!of!Alphaville,!a!Sydney*based!arts!
company!specialising!in!community!arts,!cultural!development!and!production,!with!a!focus!on!
social!justice!and!sustainability!issues.!The!READ!concept!was!collaboratively!devised!and!
implemented!by!Alphaville’s!creative!team,!and!in!particular!through!developmental!work!by!Paul!
Brown!as!Creative!Producer!and!myself!as!Program!Manager.!This!took!place!in!2012!while!working!
together!on!the!program!and!partnership!development!for!the!Nuclear!Futures!Partnership!Initiative!
(Nuclear!Futures),!an!international!program!encompassing!a!suite!of!arts!projects.!It!was!during!the!
formative!stages!of!securing!funding!from!the!Australia!Council!for!the!Arts’!Creative!Communities!
Partnership!Initiative!(CCPI),!that!Brown!and!I!commenced!our!early!work!on!the!READ!approach.!
Nuclear!Futures!was!conceived!by!Brown!as!an!extension!of!Alphaville’s!previous!Half%a%Life%project!
which!developed!a!verbatim!theatre!play!with!nuclear!veterans!(See!Brown!&!the!Maralinga!
Research!Group,!2006).!Brown!and!I!had!collaborated!on!Half%a%Life!and!on!several!environmental!
education!initiatives!at!the!University!of!New!South!Wales!(UNSW).!Nuclear!Futures!built!on!
Alphaville’s!existing!networks!and!was!inspired!by!the!urgency!to!continue!work!with!the!last!
surviving!eye!witnesses!of!the!British!led!atomic!tests!in!Australia,!and!by!renewed!efforts!of!nuclear!
affected!communities!seeking!recognition.!It!developed!as!a!series!of!interrelated!arts!and!cultural!
activities!across!three!years,!partnering!international!atomic!survivor!communities!with!professional!
artists!to!create!an!original!showcase!of!story*based!artworks,!with!the!support!and!engagement!of!
a!broad!partnership!collective!involving!both!arts!and!non!arts!organisations!(see!Chapter!4).!!
In!the!planning!and!development!phase!of!Nuclear!Futures,!Brown!and!I!worked!with!program!
partners!to!design!an!international!program!of!creative!arts,!as!well!as!the!necessary!administration!
and!management!structures!to!support!the!activities!and!partnerships.!Included!in!this!
developmental!phase!was!the!identified!need!for!evaluation.!The!early!premise!was!that!program!
evaluation!of!Nuclear!Futures!would!need!to!service!the!stipulations!of!the!primary!government!
funding!agency,!the!Australia!Council!for!the!Arts,!as!well!as!embrace!the!varied!interests!and!values!
of!community!partners,!so!as!to!be!representative!and!meaningful!for!the!mix!of!culturally!diverse!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 82!
participants!and!organisations.!The!evaluation!approach!also!needed!to!be!reflexive!and!
participatory!to!reflect!the!program’s!creative!community!partnerships!and!community!arts!setting.!!
In!the!development!of!the!READ!approach,!I!reviewed!many!inspiring!evaluation!methodologies!
including!the!rigorous!academic!and!analytical!approach!of!Somebody’s!Daughter!Theatre!
evaluations!conducted!by!Jan!Osmotherly,!the!practice*led!examples!from!Queensland!University!of!
Technology,!and!the!creative!documentation!style!of!Big!hART!for!Ngapartji%Ngapartji!(complete!with!
a!treasure!trove!box!of!stories,!symbolic!objects!and!novelties)35.!Participatory!oriented!and!arts*
relevant!frameworks!such!as!Arts!Victoria’s!Evaluation!Guide,!the!Most!Significant!Change!technique,!
and!the!BetterEvaluation!Rainbow!framework!were!useful!for!providing!methodological!guides,!some!
of!which!had!adopted!a!design*your*own!toolbox!approach.36!While!such!exemplars!were!insightful,!
my!realisation!culminating!from!the!contextual!scan!was!that!there!was!no!one!approach!readily!
applicable!to!suit!the!needs!and!scope!of!the!site!of!practice,!and!the!multi*platform!evaluative!
outputs!Alphaville!envisaged.!Further,!some!of!the!practitioner!guides!consulted!tended!to!demand!
over*simplification!of!evaluation!or!were!underpinned!by!empirical!methodologies!borrowed!from!
the!social!sciences!more!so!than!the!humanities!and!creative!arts!(Belfiore!&!Bennett,!2010).!
Consistently,!the!research!demonstrated!that!the!task!of!designing!effective!program!evaluation!for!
community!arts!is!best!done!using!a!tailored!approach,!on!a!project*by*project!basis!(Keating,!2002).!
In!conjunction!with!the!contextual!scan!to!identify!best!practice,!Brown!and!I!ruminated!on!our!own!
professional!experiences,!on!the!failures,!successes!and!lessons!from!past!evaluations,!and!reflected!
on!the!types!of!evaluative!enquiry!that!may!be!relevant!for!understanding!and!advancing!the!
creative!arts!practice!central!to!Nuclear!Futures.!We!also!considered!the!expertise,!arts!orientation!
and!strengths!of!the!Nuclear!Futures!creative!team!and!prospective!partners.!While!the!creative!
team!operated!across!a!range!of!art!forms!and!practices,!collectively!there!was!extensive!experience!
in!documentary!art*making.!Such!considerations!led!Brown!and!I!to!adopt!a!wide!frame!of!reference!
for!defining!what!might!constitute!Alphaville’s!evaluation!opportunity,!and!we!explored!a!number!of!
complementary!learning,!appraisal!and!exchange!methods!that!would!potentially!support!and!enrich!
evaluation,!program!management!and!creative!practice.!In!creating!an!original!concept!model!for!the!
Nuclear!Futures!program,!there!was!also!an!opportunity!to!consider!some!of!the!broader!issues!and!
problems!of!community!arts!evaluation!identified!within!the!field.!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
35!In!respective!order,!examples!include:!Osmotherly!(2002),!Osfield!Consultants!(2007)!and!general!information!at!
http://www.somebodysdaughtertheatre.com;!Klaebe!et!al.!(2015)!and!Gattenhof!&!Seffrin!(2014);!and!Big!hART!(2010),!
Palmer!(2010)!and!general!information!at!http://www.ngapartji.org.!36!In!respective!order,!references!for!the!listed!examples!include:!Keating!(2002);!Dart!&!Davies!(2003);!and!
BetterEvaluation!(2013).!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 83!
Collectively!Brown!and!I!had!decades!of!experience!in!arts!management!and!evaluation!to!draw!upon!
during!these!formative!conversations.!Over!three!decades,!Brown!has!amassed!a!wealth!of!
experience!in!evaluation!in!institutional!and!community!contexts,!from!the!perspectives!of!manager,!
producer,!and!creative!practitioner.!Brown’s!evaluation!experience!includes:!
•! Program!evaluation!and!staff!reviews!whilst!Head!of!the!School!of!Humanities!at!UNSW.!
•! Evaluation!of!the!impact!of!the!Arts!in!sectors!such!as!Health,!as!co*author!Art%and%Wellbeing!(Mills!&!Brown,!2004).!!!
•! Conducting!a!major!evaluation!of!Australia’s!hazardous!waste!management,!using!survey!and!
key!informant!interview!techniques.!
•! Assessment!and!peer!review!of!project!funding!applications!for!the!Australia!Council!for!the!
Arts.!
•! Reflective!practice/!evaluation!as!an!established!arts!practitioner!(playwright,!producer,!
filmmaker).!
In!complement!to!Brown’s!expertise!is!my!own!professional!experience!in!program!management!and!
evaluation!over!fifteen!years!in!community!development,!creative!arts,!education!and!sustainability.!
My!evaluation!experience!includes:!
•! Research!and!evaluation!of!a!sustainability!education!and!engagement!program!for!1,100!
staff,!in!a!NSW!local!government.!
•! Corporate!and!project!evaluations!as!Manager!for!the!Comboyne!Community!Technology!
Centre!(rural!NSW).!!
•! Program!evaluation!of!a!national!design!and!innovation!program!at!UNSW!involving!900!high!
schools,!and!staff!performance!appraisals!as!the!Program!Manager.!
•! Providing!strategic!evaluation!advice!and!planning!with!businesses!and!organisations,!as!a!
project!consultant!through!my!sole!trader!business!High%Ideals.!
Taking!inspiration!from!Alphaville’s!creative!arts!practice!we!explored!an!integrated!approach,!where!
a!series!of!complementary!review!strategies!relevant!to!community!arts!and!cultural!development!
(CACD),!might!work!together!to!extend!the!relevance!and!benefits!generated!during!program!
development!and!appraisal.!In!the!division!of!labour!for!Nuclear!Futures,!it!became!my!role!to!lead!
the!evaluation!elements!of!the!program,!and!one!of!my!early!tasks!was!to!enunciate!the!scope!and!
intent!of!READ.!Consultation!was!undertaken!with!early!partners!and!stakeholders!for!feedback,!
which!verified!interest!in!an!evaluative!approach!that!could!reach!beyond!an!evaluation!and!acquittal!
reporting!format.!We!also!undertook!negotiation!and!confirmation!with!funding!agency!partners.!!
Across!many!conversations,!the!collective!input!and!visioning!for!creating!a!fresh!approach!to!
evaluation!at!the!site!of!practice!contributed!significantly!to!the!emergent!READ!concept.!!
! !
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 84!
The!early!READ!concept,!as!included!in!the!Australia!Council!for!the!Arts!funding!application,!was:!
We!propose!to!develop!a+new+participatory+model+that+integrates+Reflection,+Evaluation,+
Analysis+and+Documentation!(the!‘READ’!model)!to!evaluate!the!program’s!processes,!
partnerships!and!creative!outputs.!!
•! We!will!use!a!partnership!model!based!on!participatory!evaluation!methodology!and!
reflective!analysis,!with!evaluative!goals!set!by!partners!to!inform!academic*supported!
framework!design!and!implementation.!
•! The!approach!blends!traditional!data!collection!and!analysis!(qualitative!and!quantitative)!
with!emergent!techniques!such!as!‘Most!Significant!Change’!process!(story*based!
evaluation).!It!involves!all!partners!in!asking!the!questions,!data!collection!and!analysis.!(It!
is!socio*dynamic.)!
•! READ!includes!formative!evaluation!and!analysis!for!ongoing!appraisal!and!summative!
evaluation!at!completion,!with!focus!on!CACD!processes,!the!creative!works!and!the!
quality!of!the!partnerships.!
•! Reflective!analysis!originates!in!individual!and!group!practice!within!individual!projects,!
and!proceeds!via!conferences,!seminars!and!publications!that!link!creative!artists,!
academics!and!communities.!!
Strengths:!
•! The!overall!approach!is!documentation*oriented!(written,!visual,!audio,!ephemeral/!rich!
media,!as!well!as!archival),!with!commitment!to!disseminate!READ!findings!to!
communities,!partners!and!industry!to!make!lessons!explicit!&!accessible!to!wider!field.!
Our!on*line!community!is!a!key!delivery!mechanism.!
•! READ!is!evidence*based,!continuous!over!multiple!years,!and!guided!by!socially!inclusive!
principles.!It!is!transferable!and!relevant!for!other!Community*based!Arts!and!Creative!
Industries!practitioners.!
•! READ!develops!a!culture!of!learning!and!reflection!that!will!support!professional!
development!and!is!a!key!strategy!for!sustaining!long*term!collaborative!partnerships!and!
capacity!of!communities!and!practitioners.!
•! Evaluation!is!supported!by!an!extensive!‘community!of!researchers’!committed!to!action!
research,!and!therefore!is!likely!to!generate!robust!evaluation!findings.!The!lead!institution!
is!QUT’s!Creative!Industries!Faculty,!which!will!host!two!doctoral!level!research!projects!
dedicated!to!Nuclear!Futures.!
(Extract!from!Alphaville!fundraising!documentation,!2012!describing!the!READ!model)!
As!outlined!in!Section!1.5,!the!summary!above!demonstrates!the!extent!to!which!the!READ!approach!
had!been!articulated!by!Alphaville!prior!to!this!research.!Brown!and!I!had!surmised!the!basic!frame!
and!overarching!strategies,!however!had!not!established!the!methodology!or!proof!of!concept.!In!
order!to!develop!the!early!READ!concept!into!a!model!of!practice!(as!an!applied/!working!system)!I!
turned!to!the!DCI!as!the!means!for!completing!this!work!as!academic!research!(distinct!from!Nuclear!
Futures!and!CCPI!funding).!The!DCI!professional!doctorate!offered!a!unique!opportunity!for!me!as!an!
industry!practitioner!to!develop!a!READ!product!and!to!substantiate!its!potential!value!using!
advanced!critical!reflection!of!practice!and!rigorous!scholarship.!The!following!sections!document!my!
process!of!leading!the!READ!model!application,!a!collective!design!and!implementation!endeavour!at!
the!site!of!practice,!to!advance!the!epistemology!and!methodology!of!the!READ!model.!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 85!
5.1.2! A#systems#approach#
To!visualise!and!examine!the!conceptualised!READ!model,!at!the!commencement!of!the!research,!I!
used!a!systems!approach!to!identify!and!isolate!its!core!elements!and!boundary,!and!to!map!the!
anticipated!inter*relationships!of!the!key!elements!(Darabi,!2002).37!A!system!is!defined!by!Raynal!
and!Rieunier!as!‘a!set!of!elements!related!both!to!each!other!and!to!their!environment!and!organised!
in!accordance!with!a!goal’!(UNESCO!&!ICCROM,!2015).38!Exploring!the!model!as!a!system!was!the!
first!step!I!took!in!the!process!of!leading!the!development!of!a!READ!prototype!and!methodological!
structure!for!the!READ!Balaklava!application.!
The!six!core!system!elements!of!the!proposed!READ!model!I!identified!were:!
1.! Support!structures!–!administrative!and!operational!structures!developed!to!link!READ!
implementation!with!program!development!and!management,!and!guide!partnership!
brokerage!and!stakeholder!engagement/!participation.!
2.! Reflection!–!community!members!and!artists!reflect!on!projects!and!practice!through!
journaling,!interviews!and!other!personal!records!(individual/collective!reflective!practice).!
3.! Evaluation!–!a!range!of!formal!instruments!are!devised!and!implemented!in!
consultation/collaboration!with!program!partners!in!response!to!key!evaluation!questions.!
4.! Analysis!–!a!cohort!of!academics!work!with!artists!and!community!members!to!develop!
scholarly!publications,!reports!and!presentations,!for!broader!learning!and!dissemination.!
5.! Documentation!–!filmmakers,!multi*media!artists!and!participants!produce!multi*platform!
documentation!and!archives,!and!extracts!from!project!documentation!are!collated.!
6.! Integration,!utilisation!&!dissemination!–!!Data!from!each!READ!element!is!synthesised!to!
produce!a!portfolio!of!evaluative!and!non*evaluative!outputs!to!meet!end*needs!and!
preferred!formats,!in!consultation!with!partners,!artists!and!communities.!
The!above!six!elements!work!together!within!the!READ!system.!Figure:!5.1!Illustrates!how!I!visualised!
the!model!as!a!system!and!its!core!elements.!During!the!chapter,!each!key!element!will!be!discussed.!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
37!Systems!theory!can!be!defined!an!approach!to!‘describe!and!interpret!complex!interactive!phenomena…!emphasizing!
the!properties!of!the!relationships!between!the!phenomena!to!be!as!significant!as!the!properties!of!the!phenomena!
themselves’!(Davidson!and!Rowe,!2009,!p.562).!While!elaboration!on!systems!theory!and!system!approach!to!program!
evaluation!(Darabi,!2002)!is!beyond!the!scope!of!the!research,!techniques!from!the!literature!were!influential!for!how!I!
approached!the!READ!model!to!theorise!and!develop!it.!!38!The!original!source!for!the!quote!is!given!as:!Françoise!Raynal!and!Alain!Rieunier,!Définir%des%objectifs%pédagogiques,!
1987,!IPNETP*Les!Nouvelles!Éditions!Africaines,!Abidjan.!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 86!
Feeding!into!the!system!are!inputs!and!information!arising!from!the!context,!partnerships!and!art*
making!at!the!site!of!creative!practice.!The!process!of!scoping!and!planning!defines!the!support!
structures!and!strategic!plans!necessary!for!delivering!READ,!including!handling!of!ethical!conduct,!
cultural!competency,!partnership!brokerage,!engagement,!administration,!content!management!and!
so!on.!!The!four!central!READ!strategies!(Reflection,!Evaluation,!Analysis!and!Documentation)!are!
illustrated!as!independent!yet!interrelated!modules,!and!running!across!all!is!an!exchange!of!
methods!and!data/information.!The!READ!strategies!generate!artefacts,!and!once!integrated!to!suit!
the!desired!partner!identified!end*uses,!a!portfolio!of!raw!and!processed!content!results!that!can!be!
used!for!evaluation,!management!and!creative!development/!arts!practice.!Interacting!with!the!
READ!system!are!companion!systems!that!drive!the!community!arts!practice!and!management,!
shown!on!the!left!of!the!diagram!where!the!spheres!overlap.!In!concurrent!operation,!the!creative!
practice,!management!and!READ!spheres!support!and!influence!each!other,!where!a!broader!
exchange!of!information!and!impact!occurs!across!the!site!of!practice.!See!Figure!5.1!below.!
!
+
!
Figure!5.1:!The!READ!model!as!system,!in!relationship!to!community!arts!practice!and!management!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 87!
5.1.3! Research#phases##
In!guiding!the!READ!application!as!a!systems!approach,!a!number!of!phased!activities!were!
conducted.!The!phases,!undertaken!as!evaluation!practice,!fit!within!the!broader!progression!of!the!
DCI!research!(see!Chapter!3!for!full!DCI!methodology).!As!previously!stated,!the!READ!application!
constitutes!four!of!the!central!research!phases!for!the!DCI.!The!READ*related!research!phases!were:!!
1.! Scoping!with!stakeholders!
2.! Planning!and!design!of!the!Balaklava!READ!methodology!
3.! Implementation!(data!collection!and!analysis)!
4.! Production!and!dissemination!of!READ!outputs.!
Figure!5.2!is!an!elaboration!of!Figure!1.2!included!in!the!Introduction!(Chapter!1).!Figure!5.2!provides!
a!brief!explanation!of!each!of!the!READ*related!research!phases!stated!above.!!It!is!important!to!
note!that!while!represented!as!a!linear!sequence,!the!four!research!phases!summarised!in!Figure!5.2!
have!not!conformed!to!a!chronological!or!straightforward!sequence!as!the!diagram!suggests,!and!
were!iterative!processes!delivered!for!both!formative!and!summative!cycles.!!Across!the!READ*
related!research!phases!I!adopted!a!reflexive!approach!that!allowed!me!to!respond!to!emergent!
project!needs,!seize!unforeseen!opportunities!and/or!address!gaps!in!the!evaluation!research.!!
!
! !
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 88!
+
!
!
! !
1. Identify and consult with stakeholders to determine values, needs & opportunities. Get to know the context and partners. What are the strengths of the art practice and partnerships? What is the aim of READ for the project? Identify cultural and ethical considerations. What are the inhibitors and enablers for using READ?
2. Analyse what is unearthed during scoping. Set purpose and sub-questions within READ strategies that balance partner needs. Select methods. Establish indicators where necessary. Decide who facilitates and participates. Work out the when, how and the what. Define the audience and ideal outputs. Develop plans for action, including how to fund and resource activities. Identify what support structures best link READ with arts practice and management.
3. Undertake data collection, analysis and integration for formative, summative or ongoing use. Get permissions. Keep reflexive and opportunistic and refine READ plans as needed. Organise and manage data collected. Support implementation with training and resources. Reduce and check data. Create a flow of information between READ, arts practice and management. Analyse content to establish findings, lessons and conclusions, and process data if needed. Keep lines of feedback and communication with partners open to plan the outputs.
!
4. Integrate the emergent data as needed to create an array of end-products from READ (raw and processed content and outputs) that meet partner needs. What formats and mediums will deliver best results for the type of information/material requested? Disseminate READ outputs to partners and stakeholders. Make use of outputs for continuous improvement, wider learning or legacy outcomes.
READ application
Figure!5.2!:!The!research!phases!for!the!DCI!
!
(Note:!The!four!research!phases!related!to!the!evaluation!practice!for!the!READ!application!are!white)!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 89!
5.2! READ#approach#and#assumptions#
Qualified!in!this!section!are!the!contextual!influences!and!application!assumptions!determined!for!
the!tailored!Balaklava!READ!framework,!such!as!the!internal!positioning!of!the!evaluation!practice!
(via!READ)!and!the!interpretive!stance!adopted!to!reflect!the!needs!and!ethos!of!the!site!of!practice!
(see!Chapter!4!for!full!details).!!
5.2.1! Context#and#parameters##
For!the!Balaklava!READ!application!there!are!three!notable!research!parameters!to!acknowledge.!
First,!while!I!refer!to!the!READ!model!design!and!implementation!process!as!experimental,!involving!
the!trial!and!error!of!methods!to!refine!the!working!model!(underpinned!by!forensic!reflective!
practice),!the!application!does!not!constitute!a!research!experiment!in!the!traditional!scientific!sense!
as!I!(as!researcher!and!practitioner)!did!not!control!the!variables!or!design!the!application!for!that!
purpose.!!
Second,!the!contextual!setting!and!the!Nuclear!Futures!project!selected!have!significantly!impacted!
on!how!the!conceived!READ!model!has!been!interpreted!and!developed.!Boud!and!Walker!(1998,!
p.169)!argue!that!context!is!‘the!single!most!important!influence!on!learning!and!reflection’,!and!this!
is!true!for!documented!community!arts!evaluation!design!(Fitzpatrick!et!al.,!2012;!Keating,!2002).!!
Given!the!importance!of!context!for!the!application!outcomes!presented!in!this!chapter,!it!is!worth!
reiterating!the!characteristic!features!of!the!Balaklava!setting!(refer!to!Chapter!4!for!further!
contextual!background).!Namely,!the!featured!project!was!a!creative!community!partnership!
initiative!positioned!within!a!community!arts!setting,!which!encompassed!broad!creative!and!
collaborative!practices!involving!multiple!partnerships!with!artists!and!communities!(see!Chapter!2).!
The!project*specific!and!geographical!context!for!the!application!was!the!small!regional!South!
Australian!community!of!Balaklava,!on!the!Wakefield!Plains!north!of!Adelaide.!Involved!were!mixed!
constituencies!of!targeted!participants!for!the!multi*arts!objectives,!involving!artists,!local!young!
people,!the!general!public!and!a!network!of!families!linked!with!Australia’s!atomic!test!history.!!
The!third!parameter!to!acknowledge!is!the!influence!of!the!scope!and!emphasis!of!the!READ!
application!for!the!Balaklava!project.!Namely,!the!application!presented!in!this!chapter!focused!on!
the!investigation!of!the!processes,!impacts!and!outcomes!of!the!community!arts!activities!for!
participants,!partners,!program!and!communities.!Figure!5.3!from!An%Evaluation%Guide%for%
Community%Arts!Practitioners,!produced!by!Arts!Victoria%(Keating,!2002),!provides!a!useful!matrix!for!
illustrating!the!areas!of!project!interest!that!came!under!review!of!the!READ!model!development.!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 90!
!
Figure!5.3:!Key!elements!of!Community!Arts!project!to!be!reviewed!!
(Source:!An%Evaluation%Guide%for%Community%Arts!Practitioners!!produced!by!Arts!Victoria%(Keating,!2002,!p.vii))!
!
5.2.2! Internal#position#for#READ#facilitation#
For!the!READ!application,!my!role!as!READ!facilitator!was!internally!positioned!within!the!Nuclear!
Futures!program,!distinct!yet!interrelated!to!my!professional!position!as!program!manager.!This!
meant!that!the!evaluation!practice!for!the!READ!application!was!internally!positioned!within!the!
community!arts!program.!This!insider!positioning!was!a!strategic!decision!considered!as!favourable!
for!facilitating!the!potential!transformative!benefits!of!the!READ!processes.!Internal!design!and!
delivery!meant!I!was!able!to!harness!my!existing!relationships!within!the!Nuclear!Futures!network!to!
support!participation!and!partnership!brokerage,!and!take!advantage!of!my!detailed!knowledge!of!
the!arts!program.!Further,!I!envisaged!that!by!acting!in!the!roles!of!both!program!manager!and!READ!
facilitator!the!efficacy!of!implementation!and!information!sharing!between!READ,!the!creative!
practice!and!program!management!would!be!enhanced.!
5.2.3! Interpretative#stance#
To!compliment!the!internal!positioning!and!the!pluralism!underpinning!the!community!partnerships,!
a!social!constructivist!stance!was!adopted!for!the!Balaklava!application!of!the!READ!approach.!Unlike!
the!positivist!or!objectivist!theoretical!traditions,!an!interpretative!constructivist!stance!recognises!
that!individuals!and!groups!construct!their!perceived!reality!(Glaser!&!Strauss,!2009;!Charmaz,!2000;!
Cresswell,!1994).!Schwandt!(2000,!p.197)!states!‘constructivism!means!that!human!beings!do!not!
An Evaluation Guide for Community Arts PractitionersPrepared by Effective Change for Arts Victoria; Darebin City Council; City of Whittlesea and VicHealth vii
W H O I S T H E G U I D E F O R ?
The guide is for community arts practitioners, rather than specific organisations, as community artsprojects can be located across a variety of organisations.
H O W S H O U L D I T B E U S E D ?
The guide essentially ‘talks’ to a generic community arts project worker and assumes that thisperson will drive the evaluation. However, it can be used in a range of ways. For example, allparticipants in a community arts project could share roles in undertaking an evaluation. Or itcould be used an external student or consultant to undertake an independent evaluation.
C O M M U N I T Y A R T S – N O T J U S T C O M M U N I T Y , N O T J U S T A R T S
This guide recognises the unique place of community arts within community and cultural activity.Case stories abound of community arts projects producing stunning and lasting creativeachievements. There are also numerous examples where extraordinary life changes occurred forparticipants while the creative outcomes were less memorable. Other times, community artsprojects have contributed enormously to their local community, through a range of strategies,such as engaging disenfranchised members of the community, or producing valuable resourcesfor a particular community. Sometimes exceptional community arts projects produceextraordinary results across all areas. Other times, people just feel good by being involved andtaking risks. This guide has been developed to work across all areas – from the creative output,to the creative journey, to the individuals and communities involved. This model of evaluationwill allow you to identify the areas of achievement and success. It works even when, inconventional terms, the project may not appear to be that successful. It will also allow you torecognise extraordinary achievements across this range of factors when they do occur.
The diagram below shows the key elements to be evaluated, using this model of evaluation.
Diagram 1: Key Elements of Community Arts Projects to be evaluated
participantsproject/
organisationcommunity
processhow participants are
involved in theproject
how the project ismanaged
how the community isinvolved in the
project
process
impactwhat happens to, or
for, participantsthrough the project
what happens asresult of the project
what happens to orfor the communitythrough the project
impact
outcomewhat happens to, or
for, participants in thelong term as a result
of the project
what happens in thelong term, as a result
of the project
what happens to, orfor, the community in
the long term as aresult of the project
outcome
participantsproject/
organisationcommunity
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 91!
find!or!discover!knowledge!so!much!as!we!construct!or!make!it’.
In!terms!of!its!relevance!for!evaluation!practice,!the!selected!interpretative!stance!aligns!with!Guba!
and!Lincoln’s!position!that!evaluation!needs!to!extend!beyond!measurement,!description!and!
judgment*!to!develop!methodologies!that!cater!for!the!multiple!realities!and!stakeholder!views!
necessary!to!produce!authentic!and!representative!findings!(Guijt!et!al.,%2011).!The!stance!meant!
that!emphasis!and!power!was!withdrawn!from!the!notion!that!evaluation!is!a!mechanism!for!
producing!objective!summative!judgments,!widening!READ’s!terms!of!reference!to!accommodate!
multiple!knowledge!systems!and!value!paradigms!of!partners,!and!the!pursuit!of!both!evaluative!and!
non*evaluative!outputs/outcomes.!!
It!is!worth!noting!that!for!future!applications,!it!is!envisaged!that!the!READ!model!has!the!flexibility!
to!be!delivered!from!a!range!of!different!stances,!across!positivist,!realist!or!constructivist!
philosophies.!
5.2.4! ParticipatorySoriented#approach#
To!best!align!with!the!community!arts!and!cultural!development!(CACD)!principles!governing!the!site!
of!practice!(which!include!participation,!representation,!democracy,!and!inclusion)!a!participatory!
approach!was!adopted!for!the!READ!application.!As!defined!in!Chapter!2!participatory!evaluation!is!
defined!by!the!level!of!engagement!and!contribution!by!participants!and!partners!in!the!process.!For!
Balaklava,!a!stakeholder*oriented!participatory!approach!was!devised!where!as!facilitator!I!
consulted!and!collaborated!with!stakeholders!and!community!representatives!who!had!a!stake!in!
the!project.!Partners!were!invited!to!be!involved!and/or!contribute!in!the!formulation!of!the!
evaluation!questions,!goals/measures,!methods,!as!well!as!with!the!delivery!and!implementation.!
While!partners!in!the!Balaklava!project!had!varying!capacities!and!interest!to!be!involved,!the!READ!
design!and!implementation!did!draw!on!the!collective!knowledge,!skills,!perspectives,!and!where!
possible,!active!participation!of!partners!during!the!process!(see!Section!5.3!and!5.4.1).!The!methods!
and!results!of!the!invitation!to!participation!in!the!READ!application!are!included!within!the!
documentation!of!the!READ!application!in!this!chapter.!
! !
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 92!
!
5.3! Scoping#
The!scoping!and!preparation!phase!for!the!READ!application!(summarised!in!Figure!5.2)!involved!
preliminary!research,!consultation!and!brokerage!with!stakeholders!and!partners!to!establish!the!
information!and!documentation!needs,!guiding!questions,!evaluation!goals!and!criteria,!and!
opportunities!for!participation!and!learning!exchange.39!Below!are!are!some!of!the!findings!
generated!from!the!scoping!process!I!facilitated!for!the!Balaklava!Community!Arts!Project.!!
(RIPP:!Section!A,!p.6!has!some!examples!demonstrating!aspects!of!the!scoping!research.)!
5.3.1! Stakeholders#identification#and#consultation#
The!first!step!of!the!scoping!process!was!to!identify!relevant!stakeholders!and!groups!for!the!project!
and!for!the!READ!application.!The!stakeholders!for!the!Balaklava!project!I!identified!were:!!
Host!organisation!and!personnel:!Alphaville!and!Nuclear!Futures!staff!and!creative!team.!
Delivery!partners:!Youth!Advisory!Committee!(Wakefield!Regional!Council,!South!Australia),!
freelance!artists/practitioners.!
Community!partners:!Australian!Nuclear!Veterans!Association!(and!Nuclear!Veteran!
communities!and!families),!British!Nuclear!Test!Veterans!Association!(BNTVA).!
Funding!partners:!Australia!Council!for!the!Arts,!Arts!South!Australia.!!
Participants:!community!members!and!stakeholders!participating!in!community!arts!processes!
and!projects;!Balaklava!High!School,!Horizon!Christian!College.!
Audiences!(public):!members!of!the!public!or!communities!of!interest!who!view,!interact!and!
experience!the!creative!works,!showcases,!promotion!and!affiliated!research.!
In*kind!partners/!institutions:!Wakefield!Regional!Council,!State!Library!of!South!Australia,!In!
Place!of!War,!Manchester!University,!Atomic!Photographers!Guild,!University!of!Adelaide.!
Academic!partners:!Queensland!University!of!Technology,!University!of!Western!Sydney,!
University!of!New!South!Wales.!
Venue!partners:!Balaklava!Courthouse!Gallery,!Balaklava!Town!Hall.!
Regional!organisations!and!representatives!of!relevance!(community,!arts!and!media):!
Balaklava!Community!Arts,!local!progress!associations,!youth!and!health!organisations,!
community!leaders,!Carclew!Arts,!local!artists.!
From!the!list!above,!I!determined!those!with!primary!interests!in!the!project!and!those!affected!
directly!by!the!proposed!READ!processes!and!dissemination!to!guide!the!extent!of!consultation!
required.!I!surmised!that!in!particular,!the!READ!activities!and!outputs!would!principally!represent!
the!interests!of!Alphaville/Nuclear!Futures!and!the!program’s!community,!delivery!and!funding!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
39!As!a!directional!guide,!to!keep!the!READ!application!grounded!within!the!DCI!research!progression,!I!have!included!the!
graphic!from!Figure!5.2!at!the!beginning!of!the!three!major!sections!to!signpost!the!related!research!phase.!!
!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 93!
partners.!Beyond!this,!READ’s!formative!and!summative!outputs/!artefacts!were!also!identified!to!be!
of!potential!relevance!to!special!interest!and!niche!audiences,!such!as!CACD!practitioners!and!
evaluators.!It!was!representatives!from!these!primary!stakeholder!groups!that!became!the!key!
informants!for!the!scoping!and!consultation!research.!
The!proposed!READ!approach!was!discussed!with!partners!and!selected!stakeholders,!as!part!of!an!
intensive!period!of!research!and!consultation!at!the!commencement!of!the!project.!For!this!process!I!
collected!information!from,!and!about,!partners!and!stakeholders!using!interviews,!communications,!
workshops,!organisational!documentation,!and!meeting!notes.!During!the!needs!assessment!I!
gathered!input!on!partner!areas!of!interest,!evaluation!and!documentation!needs!(short!and!long!
term)!and!preferred!outcomes!to!establish!the!types!of!questions!the!evaluation!study!would!
address!for!the!focus!of!the!evaluation.!Ahead!of,!and!during!sessions,!communication!tools!were!
used!to!inform!stakeholders!about!the!project!vision!and!other!partners!as!part!of!this!process.!
(RIPP:!Article!8,!p.16!is!an!example!of!a!summary!document!used!for!this!purpose.)!!
5.3.2! Methods#and#analysis###
Semi*structured!and!unstructured!interviews!were!conducted!with!individuals!and!groups!as!part!of!
the!scoping!phase.!The!preliminary!interviews!unearthed!industry!and!stakeholder!interpretations!of!
evaluation!in!the!community!arts!context,!and!specifically!the!evaluation!and!broader!needs!of!
stakeholders!for!the!Nuclear!Futures!READ!application.!In!many!cases,!as!a!prelude!for!the!interview,!
participants!were!asked!to!complete!a!short!profile!questionnaire!(via!email!or!phone)!to!generate!
preliminary!qualitative!and!quantitative!information!about!them!or!their!organisation!which!
informed!the!question!design!of!the!interview.!I!interviewed!thirteen!representatives!from:!
•! Government!Arts!and!funding!bodies!(state!and!federal)!!
•! Peak!arts!organisations!and!cultural!institutions!
•! Community!Arts!practitioners/!creative!artists/!arts!workers/!administrators!
•! Community!partners!and!community!arts!participants!
•! Evaluator!practitioners!
•! Industry!leaders/!experts/!specialists!in!research!area!
•! Academic!researchers.!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 94!
Two!principal!schedules!of!probe!questions!were!designed!to!suit!the!industry!representatives!and!
Nuclear!Futures!partners.!The!methods!for!sampling!and!question!design!used!for!the!interviews!
is!included!in!the!READ%in%Practice!portfolio!(RIPP:!Article!2,!p.8).40!!
A!number!of!meetings!with!partners!were!conducted!between!December!2013!–!April!2014!(with!
discussion!questions!reflecting!the!themes!from!the!interview!schedule)!and!one!workshop!with!ten!
representatives!from!the!creative!team!and!institutional!partner!organisations!held!in!February!2014!
was!undertaken.!Such!sessions!allowed!for!READ!goals!to!be!collaboratively!defined,!core!values!to!
be!shared!and!articulated,!and!brainstorming!for!READ!related!strategies!that!would!maximise!the!
links!to!the!arts!practice!and!program!management.!The!collected!information!was!integrated!into!
both!READ!and!program!management!planning.!(RIPP:!Article!4,!p.11!is!an!example!of!READ!and!
program!management!plans!being!integrated.)!
I!undertook!data!analysis!to!identify!emergent!themes!and!categorised!the!list!of!evaluation!needs,!
examining!the!commonalities!and/or!contrast!of!content.!Predetermined!and!open!categories!
(thematic!coding)!were!used!for!processing!the!large!volume!of!data!generated!via!the!scoping!and!
design!phases!of!the!READ!application,!predetermined!categories!were!used!for!the!stakeholder!
needs!analysis!and!for!developing!an!overarching!evaluation!plan!(with!categories!such!as!evaluation!
questions,!stakeholder!end*uses,!scheduling!needs,!preferred!format),!which!later!informed!the!
READ!theory!development.!The!data!collected!in!the!questionnaire,!interviews,!group!meetings!and!
observation!was!organised!and!synthesised!with!information!sourced!from!secondary!sources!such!
as!program!and!organisational!documentation!and!informal!communications.!
In!order!to!gain!insight!into!the!different!constructs!of!evaluation!operating!in!the!READ!application,!
and!consequently!in!the!collective!process!of!refining!the!READ!methodology,!a!process!of!
comparative!analysis!and!deconstruction!was!employed!via!a!two*tiered!process.!The!first!stage!
involved!examining!the!information!manifest!in!the!responses!to!identify!evaluation!priorities!and!
principles!held!by!the!researcher!and!stakeholders.!The!second!phase!examined!any!evident!
underlying!epistemologies!and!ideologies!(the!latent!content!inherent!in!stakeholder!articulations),!
as!related!evaluation!trends/approaches!identified!in!the!contextual!scan!that!would!be!relevant!the!
theory!development!of!a!more!widely,!transferrable!model.!The!deconstruction!technique!allowed!
responses!to!be!broken!down!at!the!site!of!practice,!and!in!some!cases!revealed!subtle!underlying!
discourses!providing!insight!into!stakeholder!evaluation!agendas!and!priorities!more!broadly.!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
40!Data!collected!in!the!interviews!spanned!both!the!DCI!and!READ!methodologies.!I!have!included!information!about!the!
interviews!in!both!dissertation!sections.!See!also!Section!3.3.1!in!Chapter!3.!
!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 95!
5.3.3! Results#for#the#scoping#phase#
During!the!stakeholder!interviews!I!asked!participants!about!their!definition!of!evaluation!and!its!
purpose/role!for!community!arts.!Definitions!predominantly!described!a!formal!assessment!process!
of!assigning!merit!or!judgment.!For!example,!one!stakeholder!described!evaluation!as!‘a!discrete!
highly!fenced!approach!to!examining!the!successes,!failures,!impacts,!benefits,!etc.!of!a!particular!
project’.!Overall,!the!articulations!of!evaluation!presented!by!the!participants!during!the!scoping!
interviews!represented!diverse!interpretations!and!definitions!of!evaluation.!As!part!of!the!data!
analysis,!I!deconstructed!the!articulations!to!gain!insight!into!the!evaluation!values!underlying!
partner!needs!and!agendas!for!participating!artists,!organisations!and!community!participants.!
The!stakeholder!evaluation!needs!and!expectations!expressed!by!participants!were!invariably!
influenced!by!past!experience!of!evaluation!and/or!the!values!of!the!organisation!stakeholders!
represented.!One!interesting!theme!arising!from!the!data!was!the!trepidation!expressed!by!multiple!
stakeholder!participants!towards!using!community!arts!evaluation!at!all.!One!creative!practitioner!
for!instance!questioned!how!formal!evaluation!could!contribute!meaningfully!towards!professional!
learning!and!future!practice,!while!another!rejected!outright!the!legitimacy!of!evaluation!practice!for!
community!arts!altogether:!‘I!don't!like!evaluations,!I!don't!like!those!standard!evaluations!and!I!
particularly!don't!like!arts!funding!body!evaluations’.!
Included!were!questions!relevant!for!establishing!baselines!for!appraising/!gauging!the!project’s!
impact,!outcomes!and!success.!What!would!success!look!like!to!different!stakeholders?!Interestingly,!
this!area!of!the!consultation!demonstrated!that!many!participants!lacked!clarity!about!how!the!
evaluation!could!best!serve!them!or!their!organisation!and!how!to!articulate!project!indicators!for!
success.!In!other!cases!however,!particularly!for!the!institutional!partners,!some!stakeholders!were!
able!to!clearly!describe!what!evaluation!areas!were!important!for!them!and/or!their!organisation.!
Consider!these!three!examples!from!different!stakeholder!participants.!I!have!anonymised!the!
participants,!and!each!of!the!following!statements!are!from!different!interviewees:!!
#1+Extract+from+interviewee+1+
…and!this!means!concentrate!on:!A*!the!quality!and!the!vibrancy!of!the!work!that!is!produced;!
B*!the!relevance!to!the!community,!and!I!mean!that!in!its!broadest!possible!terms,!and!in!this!
case!relevance!to!the!constituency;!C*!the!developments!in!capacity!of!participants.!!
#2+Extract+from+interviewee+2+
For!me!it’s!the!transformational,!it’s!the!transaction!and!the!transformation!that!comes!from!
that!transaction.!If!people!have!been!through!a!process,!or!witness!an!artwork!and!it!has!left!
them!in!a!different!place,!that!transformative!exchange!is!success.!!!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 96!
#3+Extract+from+interviewee+3+
Evaluation!is!only!relevant!in!terms!of!what!were!the!objectives!and!how!do!you!measure!
against!those!objectives.!!!
!
Another!purpose!of!scoping!was!to!gain!insight!into!the!potential!utility!and!value!READ!processes!
and!outputs!may!have!for!partners.!Through!the!interviews!and!consultation,!I!asked!partners!about!
a!wider!range!of!considerations,!beyond!evaluation*driven!processes!and!outputs,!for!example!
exploring!how!reflective!practice,!academic!analysis!or!creative!documentation!might!be!relevant!for!
their!practice!or!organisation.!Preferred!end*products!and!content!formats!were!discussed,!including!
anticipated!partner!needs!for!confidentiality!or!anonymity!for!content!expected!to!be!generated!
through!READ.!It!was!identified!that!not!all!partners!were!interested!in!traditional!evaluation!
findings!or!reporting!(based!on!the!notion!of!attributing!merit!or!worth!about!the!activities).!For!
example,!the!Balaklava!community!partners!from!the!nuclear!veteran!community,!some!of!the!
creative!team,!and!the!regional!institutional!partners!fell!in!this!category.41!!Partners!were!instead!
interested!in!story*based!outputs!able!to!capture!the!unique!or!powerful!moments!for!participants!
or!that!offered!short!case!examples!of!any!outstanding/!transformative!aspects!of!the!collaboration.!
Collectively,!there!were!a!number!of!needs!and!preferences!arising!from!the!scoping!that!informed!
the!design!of!the!READ!methodology!for!Balaklava,!including!stakeholder!requests!for:!
•! formative!and!summative!outputs,!including!project!reporting,!documentation,!findings!
•! stories!from!the!project,!story*based!examples!
•! evaluation!and!documentation!for!acquittal!
•! use!of!practice*informed!methods!arising!from!the!creative!practice!
•! community!archiving,!broader!learning!and!dissemination!outcomes,!so!information!is!
shared!beyond!site!of!practice!
•! end*products!and!information!that!can!be!used!for!continuous!improvement,!decision*
making,!new!creative!work,!future!funding!applications!and!to!enhance!partnership!
development!and!capacity!building!!
•! creative!documentation!and!web*based!outputs!that!link!with!community!stories.!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
41!More!generally!for!Nuclear!Futures,!another!example!was!the!Yalata!Aboriginal!community!partner,!not!interested!in!
evaluation!reporting!rather!creative!documentation!and!co*ownership!of!raw!data,!of!the!stories!that!arise!out!of!
community.!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 97!
Using!forensic!reflective!practice,!I!analysed!the!site*specific!needs!and!issues!revealed!during!the!
consultation!for!the!Balaklava!Community!Arts!project!to!identify!the!following!list!of!key!challenges!
that!I!would!look!to!address!through!evaluation!practice!via!the!READ!application:!
•! Lack!of!clarity!by!some!stakeholders!about!their!evaluation!needs!and!measures!of!success,!
and!the!integration!of!divergent!stakeholder!evaluation!values!and!goals.!
•! Potential!lack!of!evaluation!engagement!capacity,!as!identified!by!some!institutional!and!
community!partners!due!to!policy,!resource!or!time!constraints!in!relation!to!the!project.!
•! Prevalent!trepidation/!scepticism!by!some!artists!about!the!value!and!meaningfulness!of!
formal!and/or!traditional!evaluation!(especially!if!driven!by!bureaucratic!reporting!agendas).!
•! The!need!to!negotiate!the!funding!agency’s!independent!evaluation!requisite!with!the!in*
house!experimental!approach!under!development!by!Alphaville.!
•! Legal!and!ethical!considerations,!such!as!ensuring!permissions!for!participation!in!
documentation,!questionnaire!and!interviews!for!under!18!years!of!age!participants.!!
•! Time!and!resource!management!of!evaluation!implementation,!partnership!brokerage,!
community!engagement/!participation,!and!the!dissemination!of!evaluation!findings.!
•! The!need!to!reflect!and!embody!the!creative!practice!through!the!evaluation!practice,!to!
diversify!evaluative!outputs!beyond!a!written/reporting!format.!
In!synthesising!the!feedback!from!the!early!consultation!and!analysis,!it!was!evident!that!the!READ!
premise!to!create!a!wide!summative!lens!for!understanding!and!representing!Nuclear!Futures’!
stories!and!impact!was!highly!relevant.!The!register!of!voices!within!the!partnership!collective!called!
for!a!creative!and!engaging!framework!that!extended!beyond!traditional!evaluation!reporting.42!!
The!findings!arising!from!the!scoping!phase!with!stakeholders!critically!informed!my!evaluation!
practice!and!the!collective!design!and!delivery!of!READ!in!action!for!the!Balaklava!Community!Arts!
Project.!The!resulting!READ!methodology!was!tailored!to!suit!the!partner!needs,!concerns!and!
preferences!identified!for!the!creative!community!partnerships,!and!is!presented!next!in!Part!B!of!
the!chapter.!
!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
42!The!terminology!‘register!of!voices’!became!terminology!within!Alphaville!for!the!interviews,!images,!creative!works,!
documents,!press!material!collated!in!the!Nuclear!Futures!archive!and!documentation.!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 98!
Part#B:#READ#methodology#in#action#
5.4! Design#and#implementation#
!
The!information!generated!from!scoping!phase!allowed!me!to!understand,!articulate,!share!and!
negotiate!with!stakeholders!the!emergent!ideas!for!evaluation!practice!within!the!collective.!This!
process!helped!me!to!establish!a!shared!purpose!and!short*list!of!the!collective!evaluation!needs!for!
the!READ!application!for!the!Balaklava!Community!Arts!Project.!Based!on!the!scoping!findings,!I!
devised!a!process!for!further!engaging!stakeholders!to!input!into!the!design!of!a!reflective!and!
reflexive!methodology!for!the!Balaklava!project,!covering!how!we!could!approach!data!collection,!
analysis,!knowledge!generation!and!sharing!within!and!across!each!of!the!READ!strategies.!!In!terms!
of!the!overall!DCI!research!progression,!this!methodological!work!on!the!READ!concept,!included!in!
this!section!(Section!5.4),!constitutes!the!planning,!design,!and!implementation!phases!of!the!READ!
application!within!the!broader!structure!of!my!DCI!research.!
Discussed!first!are!the!support!structures!that!were!designed!to!scaffold!the!four!enquiry!strategies!
of!the!READ!model!–!Reflection,!Evaluation,!Analysis,!and!Documentation.!To!illustrate,!I!have!
included!examples!canvassing!stakeholder!engagement!and!partner!brokerage,!information!and!
management!systems,!and!conduct!for!ethical!practice!(Section!5.4.1).!!!
Next!each!of!the!READ!strategies!are!reviewed,!to!reveal!how!each!was!approached!as!a!
complementary!yet!independent!platform!for!capturing,!monitoring,!understanding!and!assessing!
the!partnerships!and!creative!outputs.!!In!summary,!the!READ!strategies!are!as!follows:!
•! Reflection!was!directed!toward!establishing!a!culture!of!reflective!practice.!
•! Evaluation!was!driven!by!a!set!of!key!evaluation!questions.!
•! Analysis!via!academic!research!aimed!to!generate!broader!learning!and!dissemination!of!
original!knowledge.!
•! Documentation!was!guided!by!creative,!program!and!legacy!goals!set!at!the!site!of!practice!
for!community!story*telling!and!art*making!purposes.!!
These!concurrent!strategies!meant!that!the!emergent!epistemology!and!methodological!design!for!
the!implementation!of!the!READ!approach!supported!more!than!the!traditional!aspects!of!evaluation!
practice!(often!narrow,!merit*driven!and!empirically!based)!while!extending!the!scope!of!practice!to!
explore!and!embrace!constructivist!approaches!for!collective!learning!and!project!review.!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 99!
5.4.1! Support#structures#
Underpinning!the!READ!application,!across!the!four!concurrent!READ!strategies!(Reflection,!
Evaluation,!Analysis!and!Documentation),!were!a!number!of!agreed!principles,!organisational!
systems/!procedures!and!strategic!plans!necessary!for!delivery.!I!refer!to!these!as!READ’s!support!
structures!as!they!constituted!the!scaffolding!for!the!delivery!of!the!READ!strategies!(however!within!
the!list!of!inclusions!are!non*structural!elements!such!as!a!code!of!conduct).!To!provide!insight!into!
the!supports,!I!have!listed!a!selection!below!that!I!initiated!anew!for!the!Balaklava!project!or!
tailored/refined!from!Nuclear!Futures!program!management!systems.!The!selected!examples!are!
grouped!into!three!categories:!
•! Stakeholder!engagement,!participation!and!partnership!brokerage!
•! Information!and!management!systems!
•! Procedures!for!ethical!practice.!
Below!are!examples!of!the!types!of!support!structures!that!we!put!in!place!for!the!application.!
Stakeholder*engagement,*participation*and*partnership*brokerage*
•! A!program!of!education!and!engagement!supported!participation!and!inclusion!in!READ!
activities!via!education!and!training,!induction,!and!professional!development.!For!example,!
mentoring!relationships!were!facilitated!between!established!and!emerging!artists!for!skills!
development!dedicated!to!improving!documentation!and!data!collection!techniques/skills.!
•! Selected!internal!champions!were!asked!and!supported!to!promote!READ!and!to!directly!
participate!in!the!design!and!delivery!of!activities,!to!encourage!community!and!institutional!
partner!involvement.!For!example,!Alphaville’s!Creative!Producer,!several!Nuclear!Futures!
artists,!a!SLSA!representative!and!the!local!Balaklava!coordinator!helped!to!secure!wider!
involvement.!
•! Appointment!of!a!local!project!coordinator!(for!permanent!part*time!work!across!the!project!
including!on!READ!activities!in!Balaklava)!advanced!local!relationships!and!trust!for!data!
collection.!
•! The!READ!communication!strategies!were!linked!with!those!from!project!management!based!
on!the!partner!needs!identified!in!the!scoping!consultation!(such!as!the!timing,!format!and!
content!of!communications!with!partners/stakeholders).!For!example,!when!significant!READ!
findings!were!identified!or!key!outputs!created!for!one!partner,!where!appropriate,!this!
material!was!released!for!feedback!via!internal!dissemination!and!public!sharing!(where!
appropriate)!using!partner!networks,!website,!social!media,!and!email!groups.!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 100!
Information*and*management*systems*
•! Digital!content/data!system!and!procedures!for!managing!data!retention,!storage,!archive!
register,!ownership,!security!and!confidentiality.!For!example,!both!centralised!and!
decentralised!digital!archiving!systems!for!READ!data!were!created!to!ensure!reliable,!high!
quality!and!durable!data!formats!and!processes!for!back*ups,!with!medium!and!long*term!
storage.!For!example,!a!long!term!repository!for!selected!documentation!was!established!
with!the!SLSA!collection!development!team!to!create!public!access!to!some!material,!and!
a!comprehensive!internal!archiving!system!was!managed!catering!for!the!large!quantity!
of!READ!data!generated!which!linked!with!the!Nuclear!Futures’!overall!content!
management!plan.!(RIPP:!Article!5,!p.12!is!the!developed!Digital!content!and!storage!plan.)!
•! Setting!up!and!moderating!online!network!and!forum!applications/platforms!that!supported!
partner!exchange!and!enabled!communications,!document*sharing!and!collective!review!
processes!(such!as!Dropbox,!wiki!page,!email!group!lists,!Facebook,!Skype).!
•! Stakeholder!contact!database!with!necessary!status!and!information!on!staff,!partners,!
volunteers,!and!working!groups!(such!as!the!project’s!advisory!group!of!artists,!nuclear!
veterans,!researchers!and!project!coordinators)!was!established.!
(RIPP:!Article!3,!p.10!includes!extracts!from!databases!and!profile!summaries.)
•! Management!records!for!READ*related!finances,!copyright,!permissions/!contracts,!legal!
matters!and!overall!logistical!aspects!of!READ!operation,!linked!to!the!project’s!management!
record!system.!!
Procedures*for*ethical*practice*
•! Code!of!conduct!was!based!on!Australasian!Evaluation!Society’s!(2010)!Guidelines%for%the%
Ethical%Conduct%of%Evaluations.%For!example,!consent!forms!and!protocols!for!ethical!
research!and!evaluation!conduct!ensured!READ!participants!had!sufficient!explanation!of!the!
purpose,!end*uses!and!audience!for!data!collected!(and!implications!of!the!research),!and!
handling!of!participants!under!18!years!of!age/parental!consent!processes,!and!privacy!and!
confidentiality!needs/preferences.!
•! Permission!protocols!for!decentralised!co*authorship!and!distribution!was!required!for!the!
intended!inclusion!of!the!project’s!register!of!voices!to!ensure!appropriate!acknowledgment!
and!source!identification,!unless!anonymity!was!requested,!for!short!and!long*term!
purposes.!Different!approaches!were!used!for!public!and!partner!participants.!(RIPP:!
Article!7,!p.15!is!an!example!of!a!Release!form!created!for!documentation!activities.)!!
!
!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 101!
5.4.2! Reflection#
Approach*
The!Reflection!strategy!aimed!to!inspire!a!culture!of!reflective!practice!for!artists,!community!
partners!and!participating!stakeholders,!to!make!explicit!practitioner!and!participant!knowledge!
generated!through!the!arts!practice.!Through!reflective!practice,!the!implicit!knowledge!embedded!
in!professional!practice!was!able!to!be!unearthed,!critically!examined,!and!made!applicable!as!new!
knowledge.43!Underpinning!READ’s!reflective!enquiry!is!Schon’s!notion!that!‘our!knowing!is!in!our!
action’!(Schon,!1983,!p.49)!and!that!reflective!practice!is!valuable!in!the!collective!context!(Finlay,!
2008;!Schratz,!1993).44!Schratz!(1993)!describes!how!reflection!in!the!collective!context!has!the!
potential!to!span!three!levels!of!application:!micro!analysis!for!individual!professional!development;!
medium*range!reflection!for!formative!evaluative!and!collective!benefit;!and!macro!analysis!as!a!
summative!instrument!for!organisational!learning!and!sustaining!collaborations!in!the!long!term.!!
For!the!application,!artists,!art!workers!and!community!partners!were!invited!to!provide!reflection!
during!different!stages!or!milestones!of!the!projects!paying!critical!attention!to!the!‘practical!values!
and!theories’!of!the!arts!practice!and!experience!(Bolton,!2010,!p.!xix).!In!this!way,!reflections!
generated!both!formative!and!summative!data.!An!example!of!formative!data!arising!from!READ!is!
demonstrated!in!Article!10,!where!an!artist!reflects!on!a!training!session!and!includes!valuable!
ideas!relevant!to!the!unfolding!creative!development!and!arts!practice!(RIPP:!Article!10,!p.27).!
This!differs!to!the!summative!data!in!Article!11,!from!reflection!content!submitted!at!the!
completion!of!the!public!presentation!of!the!artwork,!which!produced!information!of!relevance!
for!the!arts!organisation!and!for!future!productions!(RIPP:!Article!11,!p.30).!!
The!reflective!practice!sought!to!capture!and!scrutinise!the!progress!of!the!arts!practice,!examining!
activities!and!achievements,!problematic!blocks,!strengths,!failures,!impact!of!practice!and!the!
articulation!of!the!innovative!aspects!of!CACD.!Reflections!were!undertaken!individually!via!
journaling!and!video,!and!as!a!shared!process!through!facilitated!meeting!and!semi*structured!group!
review/debrief!on!practice.!Reflection!was!also!compatible!for!community!partners,!and!able!to!be!
incorporated!into!a!variety!of!the!art*making.!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
43!!See!Chapter!3!for!full!theoretical!discussion!of!reflection,!as!part!of!the!DCI!methodology!chapter;!and!see!Bourner!
(2003)!and!Schon!(1983).!44!There!were!a!number!of!useful!participatory!frameworks!for!collective!reflective!practice!relevant!that!I!consulted!during!
the!design!phase,!such!as!Finlay’s!2008!model!‘5!variants!of!reflexivity’!and!Schratz’s!1993!model!detailing!three!strategies!
for!collective!self*reflection,!coming!from!the!field!of!education!research.!
!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 102!
The!Reflection!strategy!was!also!underpinned!by!Alphaville’s!organisational!values!and!ethos,!
including!the!assertion!that!the!project’s!creative!team!are!the!experts!of!their!own!practice,!and!
that!their!insights!about!the!participants,!the!art*making!and!relationships!would!be!essential!for!
understanding!and!representing!the!project.!Artists!and!artworkers!know!who!showed!up,!how!
often,!and!how!participants!interacted!with!the!CACD!processes!and!other!participants!(Keating,!
2002).!In!terms!of!the!creative!process!itself,!Carter’s!influential!work!on!‘material!thinking’!explains!
the!role!of!reflection!when!creating!art,!an!inherent!process!integral!to!the!art*making!and!learning*!
that!'creative!research!is,!in!itself,!an!act!of!reflection!and!invention'!that!produces!knowledge!
(Carter,!2004,!p.191).!While!critics!and!researchers!can!describe!or!evaluate!the!creative!process!
from!the!outside,!based!on!the!final!outcomes!produced!(the!artwork!work),!it!is!the!artists!and!
community!partners!making!the!creative!work!who!are!able!to!productively!reflect!on!the!creative!
thinking!and!impact!of!their!processes.!!
For!the!creative!team!leading!the!community!arts!residencies!in!Balaklava,!the!Reflection!strategy!of!
READ!tapped!into!the!internal!and!tacit!evaluative!processes!in!operation!during!the!art*making,!to!
generate!insights!that!informed!the!creative!decisions!about!the!arts!practice!and!engagement/!
participatory!processes!with!community!(what!might!work!or!not,!and!why).!Making!such!reflection!
explicit,!using!systematic!processes!and!methods,!became!a!means!to!utilise!the!'intellectual!work!
that!usually!goes!missing!in!translation'!during!creative!development!(Carter,!2004,!p.xiii).!
To!build!a!culture!of!reflection,!I!invited!the!creative!team!members!and!community!partners!to!
participate!in!a!voluntary!program!of!reflective!practice!from!the!project’s!commencement,!to!be!
undertaken!during!or!following!the!art*making.!Where!necessary,!I!provided!information!and!
examples!to!show!how!reflective!practice!could!be!undertaken!and!provided!ongoing!support!(and!
reminders)!to!facilitate!reflection!being!conducted.!Early!in!the!project,!a!shared!understanding!was!
established!amongst!reflection!participants!about!when!and!how!reflective!practice!could!be!
conducted!and!utilised.!
At!the!organisational!level,!a!system!of!interrelated!and!iterative!processes!was!created!to!support!
reflection,!participation!and!information!management!(see!Sections!5.4.1!and!5.4.5).!Of!importance!
to!the!success!of!the!collective!reflective!practice!was!the!early!establishment!of!clear!protocols!for!
how!collected!reflections!would!be!used!and!handled!to!create!a!safe!and!ethical!environment!
based!on!consent.!I!was!able!to!use!my!dual!roles!as!READ!facilitator!and!program!manager!to!
encourage!and!support!collective!reflection,!and!became!the!central!agent!connecting!READ!with!
the!arts!practice!and!management.!The!early!benefits/relevance!of!the!reflective!processes!were!
immediately!evident!to!myself!and!several!core!participants,!and!this!created!shared!momentum.!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 103!
Methods*and*analysis*
Data!collection!via!reflection!began!as!an!organic!process,!that!was!refined!and!systematised!as!the!
READ!application!proceeded.!Multiple!methods+were!used!by!those!participating!in!reflective!
practice,!selected!based!on!their!suitability!for!participants!and!participant!preferences,!and!for!the!
quality!and!depth!of!the!reflection!they!could!deliver.!Reflective!methods!were!applied!to!individual!
and!collective!reflections!as!a!technique!used!for!planning!ahead!of!activities,!as!an!evaluative!tool!
during!creative!residencies,!and!as!a!summative!lens!following!an!event/activity.!Reflections!were!a!
mix!of!pre*planned,!guided!(probe!questions),!spontaneous!and!self*guided!responses!made!by!the!
reflective*practitioners.!Together!the!reflection!methods!generated!13!written!reflections,!many!
hours!of!video!content,!extensive!photomedia!documentation,!online!and!website!content,!and!a!
collection!of!quotes/!extracts!from!speeches,!project!documentation!and!communications.!!!
(RIPP:!Section!B,!p.25!includes!a!photo!overview!and!reflection!data!examples.)!
Through!my!own!reflective!practice,!I!analysed!the!collective!reflection!undertaken!during!the!
project’s!reflective!cycles!to!gain!an!understanding!of!the!suite!of!methods!being!applied!and!
outputs!generated.!I!identified!the!two!main!types!of!data!collection!for!Reflection!as:!
•! Active!reflection!collection:!Self!guided,!semi*structured!and/!or!facilitated!reflection,!
including!written!reflections!(post!practice),!website!blogs!(prior,!during,!or!post!practice)!
video!interviews!and!vox!pops!(during!and!post!practice),!facilitated!collective!sessions!(pre,!
during!and!post!practice),!creative!documentation!(photography!and!video)!and!excerpts!
from!oral!history.!
•! Passive!reflection!collection:!capture!of!informal!reflective!comments!arising!from!arts!
practice!or!project!management!(from!communications,!speeches,!events/workshops,!
meeting!notes,!oral!history,!project!documentation,!field!notes,!observation,!media!and!
social!media.)!
Of!the!methods!which!I!have!categorised!as!active,!the!most!significant!and!productive!were!the!
written!reflection,!interview!and!collective!reflection.!Written!reflections!were!submitted!from!nine!
members!of!the!Nuclear!Futures!creative!team!(artists!and!managers/!administrators),!two!
community!participants,!and!one!collaborating!external!artist.!Reflection!was!invited!at!the!
completion!of!a!residency!or!event,!with!the!insights!generated!being!fed!into!the!subsequent!
program!of!activities!and!concurrent!management!practices.!In!practical!terms,!reflection!invitations!
and!reminders!were!mainly!sent!via!email!or!through!phone!communication.!The!entries!received!
were!predominantly!unstructured/!self!guided!or!semi*structured/guided!written!pieces!(between!1*
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 104!
6!pages!in!length)!with!two!the!product!of!oral!history!interviews!that!were!collected!and!
transcribed!by!community!partners.!Where!appropriate,!and!if!consent!was!given,!the!written!
reflections!were!shared!with!others!in!the!creative!team!and/or!delivery!partners.!
For!the!reflective!interviews,!project!documenters!(Nuclear!Futures!artists)!conducted!video!
interviews!with!all!in!the!creative!team,!as!well!as!community!and!delivery!partner!representatives.45!
The!video!interviews!were!an!effective!method!for!collecting!oral!reflections!from!community!
partners!and!from!key!players!not!participating!in!the!written!reflection.!Not!only!did!it!save!
community!members!and!artists!the!task!of!writing!down!their!reflection,!the!documented!content!
became!immediately!relevant!for!documentary!art*making!and!evaluation!purposes.!Video!
reflections!meant!reflective!practice!was!linked!with!creative!documentation!and!art*making,!and!
was!particularly!engaging!for!creative!team!members!as!a!means!of!reflective!story*telling.!Overall,!
the!video!method!generated!hours!of!reflective!content!from!over!15!stakeholder/partner!
representatives.!Reflective!content!was!also!extracted!from!shorter!vox!pop!videos!undertaken!
with!community!participants!and!audiences.!(RIPP:!Article!12,!p.32!includes!an!excerpt!from!a!
guided!interview!reflection.)!
For!the!reflection!content!that!I!refer!to!as!being!passively!collected,!the!quality!and!quantity!of!data!
captured!from!this!collection!was!significantly!aided!by!my!internal!positioning!as!READ!facilitator!
combined!with!my!role!as!program!manager.!Such!positioning!afforded!me!access!to!a!wide!range!of!
content,!across!most!project!areas!and!from!diverse!sources,!which!resulted!in!the!capture!of!
valuable!embedded!reflective!content!from!the!arts!practice!and!program!management.!(RIPP:!
Articles!13*4,!p.33*34!are!passive!examples!from!a!blog,!an!interview!and!a!public!speech.)!
Results**
Reflective!practice!offered!those!participating!in!the!community!arts!project!a!range!of!methods!that!
encouraged!them!to!stand!outside!the!arts!practice!to!gain!different!perspectives,!in!a!safe!and!
structured!way.!Feedback!from!community!and!artist!participants!indicated!that!reflective!practice!
produced!benefits!at!the!individual!level,!and!promoted!organisational!learning!for!Alphaville!and!
the!Nuclear!Futures!team.46!For!example,!two!artists!stated!the!READ!structure!encouraged!them!to!
reflect*on!practice!in!a!new!way,!and!where!they!otherwise!would!not!have!made!time,!and!that!
they!appreciated!the!opportunity!to!articulate,!analyse!and!share!their!reflections.!For!another,!a!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
45!While!the!reflective!video!interviews!have!been!categorised!as!in!this!section!as!reflection,!they!do!overlap!in!several!
ways!with!interviews!conducted!as!part!of!the!vox!pop!method!outlined!in!Evaluation!section,!considered!separately.!46!Based!on!post*project!consultation!and!interviews!conducted!for!this!research.!
!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 105!
community!participant,!the!invitation!to!conduct!reflective!practice!was!seen!as!a!positive!
reinforcement!that!her!thinking!and!contribution!were!acknowledged!and!valued!in!the!arts!practice!
within!Nuclear!Futures.!
A!significant!proportion!of!the!reflection!content!generated!by!community!and!artist!participants!
was!of!immediate!relevance!for!the!arts!practice,!management!and!partnership!brokerage!for!the!
Balaklava!project.!Through!reflection,!community!participants!brought!unique!perspectives!to!the!
community!arts!practice!and!the!storytelling!of!their!personal!journey!was!rich!and!insightful!(see!
excerpt#1!below).!Feedback,!issues!and!ideas!raised!in!the!reflections!(about!technical,!creative,!
inter*personal,!philosophical!or!production!related!aspects!of!project)!were!able!to!be!incorporated!
into!collective!domain!and!decision*making!as!necessary!(see!excerpt!#2!below).!In!particular,!the!
information!and!perspectives!forwarded!through!collective!reflection!sessions!with!the!creative!
team,!stimulated!creative!and!innovative!shifts!in!practice!on!many!occasions,!which!Haseman!and!
Mafe!(2009)!acknowledge!as!one!of!the!benefits!arising!from!practice*informed!reflective!enquiry.!
For!me,!as!program!manager!and!READ!facilitator,!having!real*time!access!to!the!data!arising!from!
reflection!was!valuable!and!productive,!and!the!reflection!content!was!frequently!relevant!to!the!
other!enquiry!goals!of!READ!for!evaluation,!analysis!and!documentation!(see!excerpt!#3!below).!
Below!are!content!excerpts!derived!from!two!different!written!reflection!methods!and!are!
illustrative!of!the!rich!evaluative!content!arising!from!the!Reflection!strategy.!The!first!two!extracts!
are!from!unstructured,!self*guided!reflections!where!participants!approached!the!reflection!‘without!
necessarily!knowing!where!it!will!lead’!(Boud,!2001,!p.12).!The!third!is!written!in!response!to!a!series!
of!semi*structured!evaluative!questions!that!I!had!circulated!after!one!of!the!public!events!in!
Balaklava,!abbreviated!by!the!reflective!practitioner!to!guide!reflection,!again!producing!relevant!
evaluative!data.!They!are!included!to!demonstrate!the!diversity!of!themes!raised!during!collective!
reflective!practice,!and!the!varying!levels!of!depth!and!consideration!exhibited.!
#1+Excerpt+from+written,+selfHguided+reflection:+Community+participant+
Once!again!I!have!learnt!more,!met!more!people,!and!have!been!able!to!connect!other!
descendants!…!and!find!emotional!support.!It!never!surprises!me!that!when!the!descendants!
get!together!we!can!compare!similar!health!issues!and!find!a!common!thread!among!us.!I!
met!more!women!with!health!issues!as!a!result!of!their!father’s!involvement!with!the!tests...!
It!is!consistently!the!daughters!with!health!issues.!!I!listened!as!the!women!discussed!not!
being!able!to!have!children!and!wondered!how!can!that!just!be!a!coincidence!when!there!
are!only!a!handful!of!us!in!the!room!and!the!stories!are!so!similar?!!
To!know!that!there!are!people!in!the!community,!who!through!their!talent,!compassion!and!
creative!arts!want!to!tell!the!story!of!the!atomic!survivor!communities!and!the!generational!
impact!of!the!British!Atomic!Tests,!helps!to!ease!the!burden!on!the!veterans!and!their!
descendants.%
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 106!
#2+Excerpt+from+written,+selfHguided+reflection:+Artist,+Nuclear+Futures+creative+team+1!
From!the!decision!to!create!the!circular!set!design!to!actually!building!it!took!more!out!of!me!
than!I!imagined!it!would.!I!was!really!excited!that!we!had!the!opportunity!to!create!the!
circular!projection!space.!The!work!involved!did!out!weigh!the!economic!benefits!as!far!as!
time!for!me!but!creatively!and!technically!it!was!extremely!worth!it.%%
#3+Excerpt+from+written+semiHstructured+reflection:+Artist,+Nuclear+Futures+creative+team+2!
What!are!your!impressions!of!how!the!material!worked!with!audiences?:!110%!successful.!
Everyone!loved!it,!including!the!sceptics!and!semi!nuclear*pro.!Some!attended!the!talks!and!
were!immediately!defensiveness,!but!all!eventually!opened!up!and!…!seemed!to!be!taken!
aback.+
Ideas!for!improvement?:!I'm!really!impressed!with!the!running!of!everything,!the!
organisation,!and!the!level!of!enthusiasm!and!cooperation!from!everyone.!Keep!it!up,!and!all!
will!be!well.!
What!did!you!gain!professionally!or!personally?:!Avon!and!the!NF![Nuclear!Futures]!team!
reminded!me!of!the!importance!of!speaking!out,!standing!up,!and!the!importance!of!
creating!art!that!captures!the!story!of!people!who!are!brave!enough!to!do!that.!!!!!!!!!!!!
!
While!some!content!needed!sensitive!handling!and!confidentially,!the!reflection!data!was!
predominantly!shared!where!possible,!especially!between!the!creative!team.!What!evolved!were!
distinct!cycles!of!reflection!linked!to!the!delivery!of!the!creative!residency!program!and!iterative!
feedback!loops!with!program!management!and!arts!practice.!These!reflection!rounds!are!
summarised!in!Figure!5.4.!Aligned!with!Schratz’s!(1993)!three!tiered!notion!of!collective!reflection,!
the!reflective!cycles!identified!for!the!Balaklava!project!produced!varying!levels!of!analysis,!
knowledge!generation!and!utility!(referred!to!in!the!diagram!as!the!primary,!secondary!and!tertiary!
cycles)!producing!real*time!and!longer*term!impacts!at!the!site!of!practice.!Impacts!included:!
improved!decision!making!and!program!management;!enhanced!creative!development;!individual!
and!collective!benefits!for!reflection!participants;!and!development!of!a!learning!culture!
(professional!development!and!organisational)!for!continuous!improvement.!By!the!end!of!the!
application,!Reflection!had!become!an!accepted!and!respected!protocol!of!the!community!arts!
practice!for!the!Nuclear!Futures!program.!
! !
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 107!
+
!
!
Figure!5.4:!Mapping!Reflection!within!READ!for!the!Balaklava!Community!Arts!Project!
!
!
Practitioner, Participant, Partner Reflections Active and passive reflective content is collected from individual or collective self guided, semi-structured, facilitated reflection methods (written, oral, video, documentation). Systems for skills development, support, participation, collection and feedback are in place to create safe, confidential/ permission-based context. Content is copyright of reflective practitioner with levels of permission granted for use. Valuable project relevant information is generated from different stakeholder perspectives.
Collection, Analysis, Reflection Information systems for ethical content handling are established & made transparent. Protocols for record keeping & content analysis are administered by program manager to identify actions and utility for stakeholders, project and broader program. Review/ analysis may be collective as needed. Emergent themes, issues and information are organised, & immediate to long-term actions are identified. Content (raw or adapted) is distributed (as appropriate) for feeding into arts practice or project management.
Sharing, Incorporation and Impact Active response to, & incorporation of, information/ new knowledge into creative practice, production and project management at site of practice and into broader program strategies. Distribution pathways are created, and information is tailored for recipients and purpose. Information may remain internal (confidential), be shared with selected partners/stakeholders, or be included in public outputs. Internal engagement and feedback on the generated findings is encouraged (as appropriate) and iterative refinement of emergent project knowledge is undertaken.
Enriched understanding into technical, creative, socio-dynamic & methodological aspects of the project & arts practice are gained from multiple perspectives. Findings have formative and summative relevance, and generate iterative cycles of reflection by reviewers & managers. Data relevant for evaluation, documentation & analysis are identified & archived contributing to the record of creative practice.
ITERATIVE PROCESSES OF REFLECTION & USE
SYSTEMS & INPUTS
Outputs from incorporated reflective information included new project knowledge, creative content, evaluation data, project documentation/ data; and led to iterative cycles of further primary & secondary reflection to emerging information. Impact at site of practice included benefits such as improved: project & program management decision-making; community arts practice; personal & professional development; participation & engagement; trouble-shooting; continuous improvement; organisational learning; quality of communication outputs; partnership brokerage; resource allocation; & human resource management.
OUTPUTS & IMPACT
Reflective practice generated: formative information for enriching/advancing arts practice and understanding of practice; professional and personal development; opportunity for participant feedback & engagement; opportunity for shared reflection; documentary content & project documentation; relevant evaluation data. The collective and repetitive cycles build a culture of reflection within the arts practice. !
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 108!
5.4.3! Evaluation#
Approach*
The!Evaluation!element!of!the!READ!methodology!employed!a!strategic!mix!of!evaluation!
instruments!to!respond!to!the!key!evaluation!questions!set!for!the!Balaklava!Community!Arts!
Project,!devised!and!implemented!in!consultation!with!the!program’s!contributors!and!partners.!The!
evaluation!questions!driving!the!appraisal!process!focused!on!project!impact,!how!well!the!project!
met!the!stated!objectives,!the!quality!and!depth!of!the!community!arts!practice!and!partnerships,!
and!areas!of!innovation!for!the!CACD!sector.!The!evaluation!activities!aimed!to!produce!
representative!and!defensible!findings!in!line!with!the!host!organisation!and!partner!needs.47!
Underpinning!this!strategy!was!a!conventional!definition!of!evaluation,!based!on!the+‘systematic!
collection!and!analysis!of!information!to!make!judgments,!usually!about!the!effectiveness,!efficiency!
and/or!appropriateness!of!an!activity’!(Australasian!Evaluation!Society,!2010,!p.3).!!
The!process!of!designing!and!delivering!the!Evaluation!strategy!of!the!READ!model!involved!a!
number!of!distinct!steps!that!I!facilitated!and!directed!in!collaboration!with!participating!
stakeholders!representing!Alphaville,!the!creative!team,!and!delivery!and!community!partners.!
Given!the!Balaklava!Project!was!the!first!of!the!Nuclear!Futures!projects!to!be!implemented,!there!
was!no!pre*existing!evaluation!framework!at!the!program!level!to!align!with!or!follow.!As!part!of!my!
evaluation!practice!it!was!first!necessary!to!facilitate!the!design!of!an!overarching!evaluation!plan!for!
the!Nuclear!Futures!program!(not!just!at!the!Balaklava!project!level)!so!that!consistency!across!
projects!was!achieved!for!the!later!summative!program!evaluation!for!Nuclear!Futures.!Below!are!
the!steps!I!followed!for!this!process,!as!part!of!the!DCI!research:!
•! Identify!and!consider!relevant!pre*existing!models,!methods!and!approaches!from!the!
Contextual!Scan!(discussed!in!Chapter!2).!
•! Synthesise!evaluation*related!information!from!the!scoping!research!(stakeholder!needs!
assessment!process,!questions!of!interest)!and!document!analysis!of!partner!agreements.!
•! Define!the!program!logic!and!hierarchy!of!planned!outcomes!for!the!Nuclear!Futures!
program.48!!
•! Formulate!the!key!evaluation!questions!(KEQs).!49!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
47!I!use!the!term!defensible!findings!to!mean!that!the!evaluative!conclusions,!findings!or!claims!arising!from!the!READ!
methodology!need!to!based!on!credible,!prudent!and!ethical!processes!of!corroborating!information,!to!ensure!the!
findings!are!valid!and!able!to!be!explained!and!supported!if!required!by!partners!or!external!parties.!48!The!objectives!and!hierarchy!of!outcomes!for!the!Nuclear!Futures!program!can!be!viewed!in!the!evaluation!plan!(READ%in%
Practice!portfolio)!and!further!information!about!the!project!is!contained!in!Chapter!4:!Site!of!Practice.!49!Key!evaluation!questions!are!a!short!list!of!questions!that!intended!users!of!the!evaluation!want!or!need!addressed!by!
the!evaluation!process!(Kotvojs!&!Hurworth,!2013;!Fitzpatrick!et!al.,!2012).!!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 109!
•! Develop!a!framework!logic!and!methodology!for!the!overarching!evaluation!of!the!Nuclear!
Futures!program.!
•! Produce!the!Nuclear!Futures!evaluation!plan!(discussed!in!next!section).!
•! Tailor!a!project!plan!for!evaluation!suited!to!the!unique!evaluation!goals!of!the!Balaklava!
Community!Arts!Project,!based!on!the!Nuclear!Futures!evaluation!plan.!
•! Deliver/!implement!evaluation!over!the!Balaklava!Project!via!the!READ!framework.!
•! Disseminate!evaluation!findings!from!Balaklava!with!partners,!and!more!broadly!in!the!field.!
•! Refine!and!improve!the!overarching!Nuclear!Futures!evaluation!plan,!informed!by!ongoing!
feedback!and!results!of!the!Balaklava!evaluation!process.!
•! Undertake!ongoing!consultation/communications!between!evaluation!stakeholders,!and!
facilitate!participation,!for!all!of!the!above!milestones.!
For!the!Evaluation!strategy!within!the!READ!model!I!adopted!a!stakeholder!and!participatory*
oriented!approach!that!embraced!the!ethos!and!creativity!of!the!program’s!CACD!practice.!This!
meant!that!for!all!evaluation!activities!that!I!facilitated,!I!explored!how!the!activities!could!be!done!
to!draw!on!the!knowledge,!skills,!perspectives!and!participation!of!interested!partners.!To!support!
this,!I!provided!information!and!training!for!some!participants.!Varying!levels!of!participation!
ensued,!and!personnel,!artists!and!community!partners!were!actively!involved!in!four!ways:!
•! administrating!data!collection!(i.e.!delivery!of!planned!evaluation!instruments,!being!
opportunistic!to!lead!unplanned/!scheduled!data!collection,!using!arts!practice!to!capture!
data)!
•! participating!in!the!evaluation!research!(i.e.!being!interviewed,!doing!reflection,!giving!input)!
•! analysing!the!data!collected!(i.e.!reviewing/!assimilating!data,!debating/!establishing!
meaning)!
•! communicating!and!refining!findings!(i.e.!discussing!and!sharing!patterns,!themes!and!
information!arising!from!evaluation!data!internally!and!externally,!and!reflecting!on!
findings).!
(RIPP:!Article!15,!p.36!has!some!photos!and!short!descriptive!content!that!details!further!aspects!
of!the!approach.)!
Before!presenting!the!project*specific!methods!selected!for!Balaklava,!it!is!necessary!to!frame!
the!evaluation!practice!within!the!overarching!Nuclear!Futures!evaluation!plan.!The!next!section!
explains!some!of!the!methodological!details!of!the!plan,!before!discussion!turns!to!focus!on!the!
Balaklava!evaluation!and!results.!
! *
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 110!
Nuclear*Futures*evaluation*plan*
The!Nuclear!Futures!evaluation!plan!constitutes!one!of!several!significant!research!and!strategic!
planning!activities!undertaken!for!the!READ!application.!The!plan!is!included!in!the!READ%in%
Practice!portfolio!(RIPP:!Article!16,!p.37).!
The!comprehensive!evaluation!plan!included!a!guiding!evaluation!methodology!and!delivery!
schedule!for!the!Nuclear!Futures!Partnership!Initiative!between!2014*2017!for!evaluating!the!
multiple!projects!within!the!initiative,!of!which!Balaklava!Community!Arts!Project!was!the!first.!The!
evaluation!plan!(and!methodology)!sat!within!the!READ!framework!and!set!out!the!program’s!key!
evaluation!questions!(distinct!from!those!questions!driving!the!Reflection,!Analysis,!or!
Documentation!strategies).!While!the!evaluation!plan!was!a!stand*alone!document,!it!was!designed!
to!interrelate!with!the!other!READ!strategies,!and!within!the!suite!of!evaluation!methods!listed!in!
the!plan!were!methods!drawn!from!Reflection,!Analysis!and!Documentation.!
I!produced!the!Nuclear!evaluation!plan!primarily!as!an!in*house!guide!and!communication!tool,!and!
in!several!instances!I!re*formatted!or!abbreviated!the!versions!distributed!to!suit!different!partner!
information!needs.!The!plan!was!intended!to!be!dynamic!and!reflexive%and!has!had!several!rounds!of!
changes!and!refinements!over!the!past!two!years,!influenced!by!emergent!partner!input!and!
program!progress!(such!as!the!lessons!from!the!Balaklava!experience).!This!approach!to!the!
evaluation!plan!as!a!living!document!aligned!with!the!experimental!and!evolving!nature!of!Nuclear!
Futures,!and!the!ongoing!partnership!development!influencing!the!community!arts!practice.!#
As!was!outlined!in!the!section!on!scoping!(Section!5.3),!the!design!process!for!the!evaluation!plan!
involved!me!facilitating!a!process!of!collaborative!planning!and!development,!conducting!interviews!
and!integrating!data!from!the!contextual!scan!and!analysis!of!other!exemplar!program!evaluations.!
This!involved!wider!consultation!beyond!only!those!partners!relevant!to!the!Balaklava!project,!as!it!
sought!to!address!the!evaluation!needs!of!the!larger!Nuclear!Futures!collective!during!the!design!of!
the!evaluation!plan!methodology.!As!Jonassen!(1992)!points!out,!the!constructivist!approach!and!the!
contextual!influences!of!the!site!and!participants!involved!are!strong!determining!factors!in!
methodology!design,!which!was!the!case!for!the!Balaklava!project.!As!outlined!in!Chapter!4,!the!
wider!program!partners!included!diverse!cultural!groups!(Aboriginal,!non*Aboriginal,!and!
international!communities)!making!it!paramount!for!the!developed!READ!model!and!the!evaluation!
plan!to!integrate!a!cultural!lens!in!relation!to!evaluation!for!working!with!multicultural!groups!with!
different!needs.!The!resulting!epistemology!of!evaluation!was!thus!inclusive!of!community*based!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 111!
understandings!of!program!impacts!and!these!include!indigenous!knowledge!and!the!hybrid!forms!
arising!from!collaborations!between!artists,!scientists!and!communities.!
While!it!is!beyond!the!scope!of!the!dissertation!to!comprehensively!discuss!the!evaluation!plan,!
noteworthy!are!the!four!key!evaluation!areas!identified!for!the!Nuclear!Futures!program.!
Specifically,!the!overarching!evaluation!methodology!sought!to:50!
1.! Assess!program!impact!
2.! Measure!program!implementation!against!stated!objectives!and!outcomes!
3.! Provide!a!critical!for!lens!for!community!arts!practice!!
4.! Encourage!skills!development,!capacity!building!and!information!sharing.!!
I!reduced!and!analysed!the!vast!scoping!data!and!discussed!a!short!list!of!key!evaluation!questions!
with!colleagues!to!identify!the!following!key!evaluation!questions!for!the!Nuclear!Futures!program:!
•! To!what!extent!did!the!program!meet!its!stated!objectives?!
•! How!effective!was!the!program!in!terms!of!generating!opportunities!for!individuals!and!
communities!to!actively!participate!in!excellent!artistic!practices?!
•! What!was!the!quality!and!depth!of!new!partnerships!formed!and!developed?!
•! In!what!ways!did!Nuclear!Futures!impact!upon!the!delivery!partners,!participating!groups!
and!the!wider!community?!
•! How!does!the!Nuclear!Futures!increase!understanding!of!the!contribution!of!the!CACD!
sector,!in!the!stated!areas!of!interest?!
Considered!in!the!methodological!design!process!for!the!evaluation!plan!was:!the!selection!of!
methods;!allocation!of!resources,!roles!and!responsibility;!delivery!time!schedule;!stakeholder!
engagement;!communications!plan!for!dissemination;!and!the!integration!of!the!evaluation!practice!
with!other!READ!and!arts!practice!activities.51!While!broad,!the!directives!and!key!evaluation!
questions!included!in!the!Nuclear!Futures!evaluation!plan!informed!the!unique!evaluation!needs!
identified!for!the!Balaklava!Community!Arts!Project.!A!process!of!tailoring!the!overarching!
methodology!set!out!in!the!evaluation!plan!was!undertaken!to!adapt!and!simplify!the!plan!to!meet!
the!needs!of!the!Balaklava!project.!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
50!The!full!details!of!each!of!these!important!aims!for!the!evaluation!is!included!in!the!evaluation!plan!(p.10*11)!and!
provides!an!elaborate!definition!and!discussion!of!each.!51!The!length!of!the!evaluation!plan!is!longer!than!a!standard!Plan.!Abbreviated!versions!have!been!developed!to!suit!
partner!needs!during!consultation.!Alphaville!preferred!a!comprehensive!master!version!be!provided.!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 112!
Evaluation*strategy*for*READ*in*Balaklava*
For!the!Balaklava!project,!the!evaluation!focused!on!how!well!the!proposed!activities!were!delivered!
and!the!value,!quality,!and!impact!of!outcomes!achieved.!As!detailed!in!Chapter!4,!a!central!project!
objective!was!to!create!regional!opportunities!for!arts!engagement!and!participation,!via!the!
production!of!creative!works!of!excellence!in!collaboration!with!community!and!other!partners.!!
In!addition!to!the!broader!program!evaluation!objectives!of!the!principal!funding!partner,!the!
Australia!Council!for!the!Arts,!of!relevance!were!also!project*specific!goals!for!Balaklava,!such!as!the!
committed!outcomes!in!the!project!funding!agreement!with!Arts!SA,!which!included!to:!
•! Involve!communities!in!the!creative!process,!direction!and!management!of!the!project!
•! Develop!and!support!the!skills!development!of!artists!working!in!and!with!the!community!
and!the!development!of!art!form!practice.!
•! Increase!opportunities!for!South!Australians!to!engage!with!the!arts!through!creative!
participation!in!line!with!social!inclusion!principle.!
•! Deliver!innovative,!community!based!arts!programs!with!long!term!outcomes!through!arts!
and!non*arts!partnerships.!!
•! Create,!develop!and/or!present!high!quality!work!that!is!recognised!for!its!artistic!merit!
•! Demonstrate!the!capacity!for!strong!operational!and!financial!management,!effective!
planning,!evaluation!and!use!of!resources.!!!!!!!
(Sourced!from!program!documentation,!courtesy!of!Alphaville).!
Unique!project!evaluation!goals!were!informed!by!the!community!partners!central!to!the!project’s!
development!and!delivery.!The!project!and!evaluation!goals!sourced!from!community!members!
were!not!couched!in!the!bureaucratic!language!of,!or!focus!on,!community!partnerships!but!rather!
connected!with!the!creative!and!social!justice!story*telling!capacity!of!the!project.!For!example,!in!
early!interviews!with!nuclear!veteran!Avon!Hudson,!when!asked!what!would!constitute!project!
success,!he!identified!his!want!for!the!stories!to!go!far!*!for!mainstream!audiences!to!be!engaged;!
for!public!awareness!about!Australia’s!atomic!test!history!to!be!increased;!and!that!public!
recognition!for!impacted!communities!be!promoted.!This!objective!to!make!the!story!go!far!was!
important,!and!was!incorporated!into!the!broader!program!and!Balaklava!project!rationale.!
In!assimilating!the!broader!Nuclear!Futures!evaluation!needs!and!unique!evaluation!goals!for!the!
project,!I!identified!five!priority!domains!for!appraisal!(i.e.!How!well!did!the!project!perform!for…):!
•! creative!community!partnerships,!
•! collaborative!art*making!and!cultural!development,!
•! artistic!vibrancy,!
•! engagement!of!regional!audiences!with!the!arts,!especially!young!people,!
•! making!the!story!go!far.!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 113!
Methods*and*analysis*
Across!March!2014!*!May!2015!participatory!processes!were!used!where!possible!with!core!team!
members!(artists!and!administrative!personnel)!to!employ!quantitative!and!qualitative!methods!
(including!rich!media)!for!the!multi*arts!activities!delivered.!Data!collection!methods!included!semi*
structured!interviews,!reflective!practice,!survey!instruments,!observation,!story*based!techniques,!
and!creative!documentation!such!as!film!and!photo!essays.!In!summary,!the!evaluation!findings!
were!based!on!the!analysis!of!data!captured!from!sources!such!as:!
•! Audience!numbers,!ticket!sales,!participant!registrations,!media!outreach!etc.!
•! Written!and!video!reflections!by!the!creative!team!and!community!partners.!
•! Feedback!forms!and!surveys!completed!by!participants!and!secondary!school!audiences.!
•! General!audience!post*event!feedback!forms!collected!during!the!season.!
•! Vox!pop!video!interviews!with!audiences,!post!event.!
•! Communications!and!informal!feedback!from!local!partner!organisations.!
•! Community!representation!for!management!and!working!groups.!
•! Observation,!reflective!practice!and!academic!analysis!by!the!lead!evaluator.!
•! Creative!documentation!(film,!photography,!video!interviews!etc.).!!
•! Story*based!techniques!(oral!history).!
(RIPP:!Article!17,!p.68!includes!examples!of!some!instruments!and!data!collection!methods.)!
The!methods!selected!were!in!keeping!with!the!envisaged!epistemology!of!the!READ!concept,!that!
seeks!to!foster!a!strengths*based!approach!utilising!the!multimedia!creative!documentation!skills!
and!processes!(such!as!documentary!photography!and!film!making)!embedded!in!Nuclear!Futures’!
art*making.!The!suite!of!creative!documentation!methods!employed!for!the!Evaluation!strategy!
naturally!overlapped!with!the!activities!of!the!Documentation!strategy!(see!Section!5.4.5).!This!is!
a!demonstration!of!the!way!evaluation!spanned!across!the!other!READ!strategies!(and!visa!
versa).!(RIPP:!Article!18,!p.69!provides!links!to!three!video!documentation!examples.)!!!
Information!generated!about!the!project!during!the!data!collection!phase!was!treated,!organised!
and!synthesised!to!generate!meaningful!and!robust!findings!in!response!to!the!KEQs.!My!role!
involved!managing!and!organising!the!data,!and!the!preliminary!task!of!reducing!the!data!collected!
prior!to!consultation!and!shared!processes.!I!also!used!triangulation!techniques!to!verify!data!
generated!from!different!sources.!The!aggregate!analysis!and!shared!review!undertaken!by!myself!
and!other!participants!(at!least!five)!used!processes!such!as!evaluative!reasoning,!impact!rankings,!
performance!rubrics,!and!partnership!mapping.!The!Artistic!Vibrancy!Framework!developed!by!the!
!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 114!
Australia!Council!for!the!Arts!was!the!guiding!criteria!for!claims!and!evaluation!findings!on!the!
artistic!merit!of!the!works!produced.!The!analysis!also!included!consideration!of!the!broader!
implications!arising!from!the!work,!such!as!industry!accolades!and!feedback!from!prospective!future!
venues.!
For!each!of!the!five!domains!that!I!identified!as!priority!areas!for!appraisal,!I!drafted!a!descriptive!
account!of!what!was!achieved!and/or!delivered!through!the!project!and!integrated!the!evaluative!
findings!demonstrating!the!extent!or!quality!of!what!was!achieved!and!impacts!evidenced.!In!the!
domain!narratives!I!substantiated!the!emergent!evaluative!claims!with!specific!reference!to!data!
sources,!demonstrable!examples,!specific!quotes,!statistics!or!other!evidence.!I!shared!these!
evaluation!narratives!and!findings!for!feedback!with!core!creative!team!members!and!community!
partners.!In!doing!so,!the!content!reflected!my!own!interpretation!of!the!evaluation!information!
generated,!and!the!shared!values!and!consent!from!others.!My!intention!was!that!the!evaluation!
narratives!would!assist!in!profiling!the!project!outcomes!and!impact!of!most!interest!to!the!partners,!
and!create!material!relevant!for!the!program’s!summative!evaluation.!For!the!Balaklava!project,!no!
external!review!involving!an!independent!panel!and/or!evaluation!specialist!was!undertaken.!
An!example!of!the!domain!narratives!is!included!in!the!Acquittal!Report!for!Arts!SA!(RIPP:!Article!
36,!p.140).!Below!are!a!two!examples!of!the!evaluative!claims!resulting!from!the!process:!
The!collaborative!workshop*based!cultural!development!approach!employed!for!the!Field%of%View!exhibition!for!example,!was!highly!praised!by!participants,!community!partners!and!Wakefield!
Regional!Council.!Prevalent!from!the!results!of!the!participant!feedback!survey!was!consistent,!
positive!review!of!the!art*making,!training!and!mentoring!processes!facilitated!by!the!artists,!with!
‘skills!development’!and!the!‘opportunity!to!publicly!exhibit’!recorded!as!frequently!stated!
benefits!of!participation.!Short!reflective!interviews!with!participants!revealed!personal!and!
sometimes!transformative!impact!for!participants,!including!increased!confidence!for!art*making,!
advancement!of!individual!practice,!and!personal!rewards!from!having!their!work!celebrated!
within!the!community!context!(for!many!participants!this!was!the!first!time!their!work!had!been!
exhibited!in!a!Gallery).!
As!stand!alone!creative!works![10%Minutes%to%Midnight!showcase],!the!artworks!produced!fit!the!Australia!Council!for!the!Arts’!classification!for!‘great’!art.!As!defined!in!the!Australia!Council!for!
the!Arts!Artistic!Vibrancy!framework,!the!attributes!of!‘great’!art!is!that!it:!‘demonstrates!
integrity!of!process;!embodies!excellence!of!craft!and!skills;!demonstrates!imagination,!
distinctiveness!and!originality;!contributes!to!artistic!practice;!engages!with!the!diversity!and!
complexity!of!contemporary!life;!and!is!relevant!in!a!local,!national!and!global!context’!(taken!
from!the!e*book!introduction!http://www.australiacouncil.gov.au/strategies*and*
frameworks/artistic*vibrancy/).!These!attributes!are!evidenced!in!the!evaluation!findings,!and!
recognition!of!the!high!levels!of!artistic!merit!achieved!was!affirmed!by!audience,!partner!and!
team!member!reflections,!interviews,!feedback!forms!and!vox!pop!reviews.!Community!
sentiment!from!attending!nuclear!veterans!was!consistently!affirming!–!‘You!got!it!right’.!
(Extracts!from!the!Balaklava!Community!Arts!Project!Acquittal!Report!for!SA,!2015,!courtesy!of!
Alphaville.)!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 115!
Results**
The!mixed!evaluation!methods!employed!generated!data!from!qualitative,!quantitative!and!
performative!sources.!!Where!possible!I!encouraged!collective!sensemaking!and!analysis!of!the!data!
collected,!to!ensure!different!perspectives!within!the!community!partnerships!were!included!in!the!
process!of!reviewing!and!synthesising!data.!Some!evaluation!findings!were!useful!immediately,!and!
other!data!got!analysed!through!longer!term!processes.!For!example,!the!results!from!the!school!
student!evaluation!for!the!10%Minutes%to%Midnight%showcase!were!used!in!real*time!by!the!creative!
team!to!adjust!the!contextual!information!provided!for!the!art!and!also!to!finalise!some!creative!
aspects!ahead!of!the!programmed!Fringe!festival!events.!(RIPP:!Articles!19,!p.70)!!
Evaluation!findings!included!statistical!statements!evidenced!on!survey!data!as!well!as!rich,!
qualitative!feedback!by!community!participants,!audiences!and!artists.!The!evaluation!outputs!and!
outcomes!generated!from!this!READ!strategy!were!multi*faceted!and!multiplatform!to!suit!the!key!
evaluation!audiences!and!partners!and!included:!
•! the!Nuclear!Futures!evaluation!plan!and!tailored!project!plan!for!Balaklava!
•! tested!survey!instruments!for!future!projects!(i.e.!audience!feedback!forms)!
•! video!documentary!content!of!artists,!partners,!community!participants,!audiences!
•! statistical!information!based!on!audience!feedback!(RIPP:!Article!20,!p.72!for!example.)!
•! testimonies,!letters!of!support,!case!studies!and!evaluative!findings!of!relevance!for!
future!arts!practice,!funding,!partnership!development!
•! report!documents:!monthly!reporting!for!Australia!Council!for!the!Arts!(included!evaluation!
findings!as!produced),!Acquittal!report!for!Arts!SA,!and!partner!summaries/!presentations!
to!local/regional!institutional!and!community!partners!
•! infographics!summarising!evaluation!findings!
•! skills!development!and!working!knowledge!of!group!processes!for!evaluation!practice.!
The!evaluation!outputs!were!distributed!as!necessary!to!the!partners!and!participants!involved.!
Feedback!and!consultation!following!the!dissemination!of!evaluation!findings!indicated!that!the!
outputs!and!content!met!partner!needs!and!expectations!for!the!evaluation,!and!for!several!
partners!evaluation!outputs!exceeded!expectations.!There!was!also!evidence!that!the!evaluation!
findings!were!of!immediate!use.!For!example,!a!number!of!compelling!themes!and!insights!emerged!
from!the!evaluation!that!were!the!basis!of!formative!discussions!by!the!creative!team!for!
subsequent!Nuclear!Futures!arts!practice!and!projects,!and!influenced!the!approach!and!
development!of!the!10%Minutes%to%Midnight!showcase!delivered!at!The!Block,!at!the!Queensland!
University!of!Technology,!later!in!2015.!!!
!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 116!
It!is!important!to!note!that!the!evaluation!outputs!distributed!to!partners!in!2015!are!not!the!end!of!
the!READ!cycle!for!this!project.!Data!has!and!will!continue!to!emerge!about!the!project!beyond!the!
date!of!the!partner!closure!meetings!for!the!Balaklava!project.!Subsequent!to!the!dissemination!of!
the!evaluation!outputs!(like!the!Arts!SA!Acquittal!Report)!some!of!the!longer!term!and/or!broader!
implications!of!the!Balaklava!work!have!been!revealed,!and!these!have!a!bearing!on!how!the!project!
should!be!evaluated.!A!few!examples!are:!photomedia!artist!Jessie!Boylan!won!a!major!photography!
prize!for!artwork!generated!at!Balaklava,!the!projection!installation!10%Minutes%to%Midnight!was!
award!as!a!finalist!in!an!international!moving!image!competition,!and!audience!feedback!of!the!
artworks!continue!to!be!generated!at!following!exhibitions.!This!rich!data!will!continue!to!be!
collected!by!Alphaville!and!fed!into!the!summative!evaluation!for!the!Nuclear!Futures!program,!
however!it!does!highlight!how!timing!of!final!READ!activities!will!have!a!significant!influence!on!the!
range!of!data!able!to!be!included.!
5.4.4! Analysis#
Approach*
The!aim!of!the!READ!Analysis!strategy!is!to!promote!scholarship,!research!and!dissemination!of!
original!knowledge!arising!from!community!arts!in!areas!of!significance!about!the!practice!and/or!
emergent!creative!works.!This!READ!strategy!is!a!research*oriented!and!analytical!enquiry!approach!
for!learning,!driven!by!a!community!of!researchers,!to!provide!a!complimentary!critical!lens!for!
Reflection,!Evaluation!and!Documentation.!Analysis!focuses!on!the!wider!implications!arising!from!
CACD,!using!systematic!and!multi*step!processes!of!investigation!(documentation,!description,!
analysis,!discovery,!interpretation,!or!theorisation)!to!advance!knowledge!in!this!sector!of!
community!arts.!
For!Nuclear!Futures,!Analysis!involved!a!cohort!of!academics!working!with!artists!and!community!
members!to!develop!scholarly!publications,!reports!and!presentations.!Alphaville!refers!to!the!
collaborators!as!a!‘circle!of!advisers,!academics,!creative!artists!and!other!researchers’.52!As!well!as!
generating!original!knowledge,!the!approach!seeks!to!promote!broader!learning!via!the!wider!
dissemination!of!action!and!practice*led!research.!It!is!grounded!in!the!premise!that!facilitating!
collaboration!between!academics,!researchers!and!arts!practitioners!would!enrich!the!endeavours!
of!each!other!(National!Ideas!Summit!&!Australia!Council,!1990).!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
52!Sourced!from!Nuclear!Futures!documentation,!courtesy!of!Alphaville.!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 117!
Driving!the!Analysis!strategy!was!Alphaville’s!interest!in!unlocking!the!academies+for!collaboration!
between!creative!artists!and!researchers!for!CACD.!The!notion!of!‘unlocking!the!academies’!first!
gained!traction!in!the!early!1990s!following!the!release!of!an!Australia!Council!discussion!paper!
entitled!Unlocking%the%Academies,!prepared!as!part!of!the!follow*on!from!the!1990!National!Ideas!
Summit!(National!Ideas!Summit!&!Australia!Council,!1990).!In!its!early!iteration,!attention!was!on!
reducing!the!cultural!and!professional!gaps!between!academics,!journalists!and!media!publishers,!
however!the!need!to!engage!and!collaborate!with!the!academies!is!also!relevant!for!other!sectors!
(Curthoys,!1991).!It!was!argued!that!too!often!academic!knowledge!is!not!accessible!for!the!wider!
public!or!non*specialist!audiences!due!to!a!lack!of!a!common!language,!preventing!knowledge!
exchange!and!learning!between!industry!practitioner*researchers!and!academics!(National!Ideas!
Summit!&!Australia!Council,!1990).!Based!on!observation!during!my!professional!experiences,!this!
remains!true!for!the!CACD!sector.!With!this!goal!in!mind,!Alphaville!was!well!placed!to!explore!how!
the!CACD!sector!might!progress!toward!‘unlocking!the!academies’!as!a!large!number!of!individuals!
connected!with!the!Nuclear!Futures!network!had!academic!roles,!interests!or!previous!research!
experience.!!
Methods*and*analysis*
During!the!developmental!stage!of!Nuclear!Futures,!Creative!Producer!Paul!Brown!approached!
potential!academic!partners!to!research!the!unfolding!community!arts!activities!of!the!creative!
community!partnerships!initiative.!Academics!from!the!following!institutions,!representing!multiple!
disciplines,!partnered!with!the!Nuclear!Futures!program:!
•! Queensland!University!of!Technology!(QUT)53!
•! University!of!Western!Sydney!(UWS)!
•! University!of!New!South!Wales!(UNSW)!
•! Murdoch!University!
•! Hiroshima!Peace!Institute,!Hiroshima!University!
•! Manchester!University!
•! Winchester!University.!
For!the!Balaklava!project,!the!partnerships!with!QUT,!UNSW!and!UWS!have!been!most!relevant.!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
53!QUT!has!been!involved!via!my!role!as!a!DCI!researcher!and!while!independent!to!the!Analysis!approach!described!in!this!
section,!is!still!contributing!to!the!Analysis!strategy!and!outcomes.!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 118!
The!following!text!demonstrates!the!way!that!Alphaville!framed!the!academic!scholarship!in!relation!
to!the!community!arts!practice:!
The!Nuclear!Futures!project!brings!together!streams!of!theory!and!practice,!from!theatre!to!
activism,!to!wider!social!movements!and!politics,!to!memory,!loss,!survival,!history!of!
imperialism!and!displacement,!etc.!It!is!about!utopian!and!dystopian!futures,!capturing!
important!historical!developments!in!Australia,!United!Kingdom,!Japan!the!United!States!and!
Pacific!nations.!The!stories!of!atomic!survivors!are!too!often!neglected!and!ignored,!glossed!
over!and!obscured,!even!purposefully!recast!or!excluded.!!
Multiple,!dynamic!and!diverse!arts!practices!have!driven!the!project!so!far,!alongside!
scholarship!from!institutions!in!several!countries.!It!is!now!time!for!those!of!us!seeking!to!
make!sense!of!the!research!side!to!take!stock!and!ask:!what!are!the!theoretical!and!wider!
academic!implications!of!this!work?!(Courtesy!of!Alphaville!program!documentation)!
A!research!plan!for!the!Analysis!strategy!for!the!READ!model!was!devised!in!2014!by!Alphaville!and!
partnering!academics!from!University!of!Western!Sydney,!Professor!James!Arvanitakis!and!Dr!Ingrid!
Mathews.!A!staged!program!of!research!and!publication!was!designed.!The!first,!a!descriptive!phase,!
aimed!to!shed!light!on!the!arising!testimonials!and!emergent!register!of!voices,!with!delivery!of!a!
small*scale!seminar!and!publication!of!documentation.!The!second!phase,!scheduled!for!2016,!
planned!for!further!conceptual!work!and!dissemination!of!analysis!via!wider!conferences!and!
forums,!with!the!objective!to!publish!and!disseminate!a!second!compilation!of!material!generated!
on!Nuclear!Futures!and!related!topics!on!‘art!and!the!bomb’.!At!the!time!of!writing,!only!the!first!of!
these!two!phases!had!been!undertaken.!(RIPP:!Article!22,!p.!75!details!the!Research!Plan.)!
Alphaville!and!partners!were!interested!in!multi*disciplinary!research!areas!encompassing!CACD!as!
well!as!other!non*arts!disciplines!such!as!Science!and!Technology!Studies!(STS)!studies!and!Political!
History.!In!February!2014,!the!core!Nuclear!Futures!creative!team!and!participating!scholars!from!
UNSW,!UWS,!QUT!and!Murdoch!University!attended!a!visioning!workshop!to!canvas!potential!
research!topics.!The!ideas!generated!by!this!collective!process!identified!the!following!prospective!
themes:!Memory!and!narrative;!Arts*Science!interactions;!The!history!of!military!and!civil!nuclear!
programs;!Imperialism,!the!Cold!War,!and!other!key!historical!dimensions;!Wastes!and!
contaminated!sites;!Health!and!radiation!impacts;!The!role!of!the!arts!(and!of!community*based!
arts);!Art,!protest,!and!disruption;!Gift!theory!and!reciprocity;!Justice!and!the!arts;!Resilience!and!
survival;!Eco*criticism;!and!Future!Studies.!These!themes!were!driven!by!the!research!interests!of!
the!network!of!engaged!academics!and!the!creative!team,!and!as!is!evident,!were!not!related!to!the!
key!evaluation!questions!or!limited!to!areas!covered!by!the!program!objectives.!
!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 119!
Results**
To!date,!the!Analysis!strand!of!READ!has!involved!academic!partners!and!collaborators!working!with!
creative!artists!(some!of!whom!are!also!academics)!and!communities!across!a!range!of!disciplines.!
The!first!phase!of!Analysis!has!resulted!in!the!following!scholarly!research!and!presentations:!
•! Conference!Presentation:!Arts!and!the!deep!Nuclear!Future:!a!prospectus!by!P.!Brown,!!
A.!Hudson,!E.!Barkley,!J.!Arvanitakis!at!the!Eco*Arts!Conference,!Wollongong!12*13!May,!
2013.!(RIPP:!Article!23,!p.77!includes!Abstract!and!Article!21,!p.74!shows!image.)!
•! Conference!Presentation:!Harnessing!evaluation!through!integration:!A!participatory!
model!for!Reflection,!Evaluation,!Analysis!and!Documentation!(the!‘READ’!model)!in!
Community*based!Arts!by!E.!Barkley!at!the!Australasian!Evaluation!Society!Conference,!
Darwin!10*12!September,!2014.!(RIPP:!Article!24,!p.78!includes!Abstract!and!slides.)!
•! Conference!Presentations:!Nuclear!Futures!by!P.!Brown!and!testimony!by!M.!Goodwin!at!
British!Nuclear!Test!Veteran!Association!(BNTVA)!Conference,!Manchester,!23*26!May!2014.!
•! Academic!Seminar:!Creative!Arts!and!Nuclear!Futures,!The!Glasshouse,!QUT!organised!by!
Alphaville!6*7!August!2015!(two*day!presentation!program!involving!20!participants).!
(RIPP:!Article!25,!p.96!includes!presentation!slides!for!my!presentation!on!READ.)!
!
Of!the!above,!of!particular!relevance!was!the!QUT!Seminar.!The!seminar!was!a!chance!to!‘take!stock!
and!ask:!what!are!the!theoretical,!academic!and!wider!social!implications!of!this!work?’!(taken!
from!Seminar!program/Alphaville!documentation).!Across!the!two!days!17!researchers!and!
practitioners!were!involved!in!the!program!of!presentations,!panel!discussions,!screenings,!
podcasts!and!live!streaming.!(RIPP:!Article!21,!p.74!for!images,!and!Article!34,!p.122!for!program.)!!
In!addition,!there!have!also!been!several!academic!events!and!forums!at!which!I!have!disseminated!
research!with!industry!peers!and!networks!about!Nuclear!Futures!and!the!READ!approach,!including:!
•! Evaluation!workshop!to!CSIRO!social!research!teams,!Brisbane,!December!2013.!
•! Sharing!of!research!at!the!masterclass!for!postgraduate!students!prior!to!the!Spectres%of%Evaluation%conference!in!Footscray,!February!2014.!
•! Dissemination!of!materials!to!Nuclear!Futures!stakeholders/!partners!(partner!summaries,!
online!publication,!sharing!of!DCI!research!outputs).!
Based!on!the!results!so!far!from!the!academic!led!Analysis,!it!is!anticipated!that!the!Analysis!outputs!
when!integrated!with!the!evidence!of!program!impact!generated!from!the!other!READ!strategies,!
will!greatly!contribute!to!an!increased!understanding!of!the!contribution!of!the!program,!and!also!
more!generally!of!the!CACD!sector.!This!is!however,!an!area!of!READ!that!is!still!being!developed!and!
necessarily!extends!beyond!the!DCI!research.!For!example,!we!are!still!exploring!how!best!to!
engineer!feedback!loops!over!the!medium!and!long*term!between!the!academic!research!produced!
and!the!communities!involved;!the!preferred!means!of!distribution;!and!issues!relating!to!co*
authorship!between!academic!researchers!and!participants!involved!in!the!art*making!and!research.!! !
!
!
!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 120!
5.4.5! Documentation#
Approach*
Documentation!aimed!to!enhance!knowledge!and!understanding!through!the!organised!collection!of!
records!and!artefacts!arising!from!community!arts!practice!and!sourced!from!private!archive!
collections!of!community!participants.!For!the!READ!application,!Documentation!served!both!as!a!
creation!and!organisation!strategy.!This!was!consistent!with!prescriptions!advocated!by,!among!
others,!UNESCO!and!ICCROM!(2015).!Art*based,!administrative!and!archival!methods!were!used!to!
chronicle!the!story!of!the!community!arts,!and!embedded!community!and!artist!stories,!for!
immediate!and!long*term!purposes.!This!included!the!collection!of!printed!and!online!
documentation,!digital!and!photo!media,!oral!history,!archival!and!education!resources,!and!raw!and!
produced!creative!documentary*art!content.!The!documentation!captured!key!project!turning!
points,!emergent!themes/tensions,!stories!of!broad!public!relevance,!creative!works,!partnership!
exchanges!and!participant!experiences.!Creative!processes!such!as!curation,!production,!editing!and!
archiving!offered!an!interpretive!lens!for!understanding!the!artistic,!social,!economic,!political!and!
environmental!implications!of!the!work.!Multi*platform!and!dynamic!representations!of!the!arts!
practice!and!people!involved!resulted.!!(RIPP:!Article!26,!p.100!presents!a!photo!overview).!
Influential!to!the!approach!was!Alphaville’s!intent!to!create!opportunities!and!partnerships!that!
would!support!public!accessibility!to!the!documentation!and!long*term!storage!beyond!the!three*
year!Nuclear!Futures!program.!For!Balaklava,!one!example!of!this!is!the!partnership!developed!with!
State!Library!of!South!Australia!(SLSA)!where!SLSA!staff!mentored!community!members,!artists!and!
arts!workers!in!the!archiving!of!Nuclear!Veteran!related!materials!unearthed!during!creative!
research.!The!Library!initiated!a!new!collection!on!South!Australia’s!atomic!history,!so!that!archival!
materials!of!state!significance!and!oral!histories!generated!could!be!donated!to!the!institution!for!
long*term!public!access.!To!date,!two!nuclear!veteran!families!involved!in!the!Balaklava!project!have!
contributed!to!this!collection!and!it!is!anticipated!further!materials!will!be!donated!in!later!phases.!
Like!the!other!READ!strategies,!the!design!and!implementation!of!the!Documentation!strategy!was!
facilitated!as!a!collaborative!process,!with!Alphaville,!community!partners!and!Nuclear!Futures’!
artists!with!filmmakers!and!multi*media!artists.!Driving!the!strategy,!were!the!stakeholder!
needs/interests!identified!in!the!scoping!phase!as!well!as!goal*free!creative!development!using!
documentation!art!processes!led!by!artists.!The!resultant!approach!was!a!strength*based!process!for!
recording,!organising!and!reviewing!data!to!produce!multi*platform!documentation!and!archives.!
The!methods!used!are!discussed!below.!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 121!
Methods*
For!community!arts,!it!is!both!the!processes!and!objects!created!that!convey!significant!messages!
about!the!arts!practice,!partnerships!and!cultural!development!outcomes!(Goldbard,!2006).!In!the!
first!instance,!process!documentation!was!conducted!to!chronicle!and!support!the!community!arts!
process!itself!and!this!involved!internal!documentation!(by!artists,!administrators!and!community!
members)!during!the!project!for!the!purpose!of:!capturing!stories!from!the!implementation!
experience;!as!art*making!in!and!of!itself;!and!for!continuous!improvement.!This!linked!with!the!
ongoing!project!monitoring!and!represents!the!‘how’!of!implementation!(Stauffer,!2015).!Secondly,!
attention!was!given!to!the!emergent!outputs,!artefacts!(such!as!the!creative!artworks!generated)!
and!the!outcomes!of!the!CACD!processes.!Focus!was!on!the!‘what’!of!the!art*making.!Lastly,!
reflective!methods,!inherent!in!the!creative!and!organisational!processes!used!during!the!processing!
phase!for!the!documentation!collected!(such!as!art*making,!publication,!and!production),!were!a!
means!for!exploring!the!‘why’!and!‘what!next’!aspects!of!the!community!arts!being!undertaken.!
Methods!included:!photography;!video!and!film;!audio!recording;!reporting;!publication;!script!
writing;!field!notes;!transcripts;!blogging;!infographics/!visual!summaries;!data!base!and!record!
keeping;!communications;!interviews;!media!and!documentation!scans;!oral!history;!archiving!and!
collection!development.!For!Nuclear!Futures’!Balaklava!Community!Arts!Project!I!surmised!that!
there!were!four!principal!types!of!documentation!data!sets!resulting:54!
•! Creative!documentation:!Art*based!and!storytelling!content!informed!by!documentary!arts!
traditions!that!culminated!in!vast!quantities!of!photomedia!and!video!content,!a!Nuclear!
Futures!promotional!video,!photo!essays,!a!photo*story!book,!video!interviews,!website!
material/blogs!and!planned!applications!for!film.!Creative!documentation!was!carried!out!by!
visiting!and!local!artists,!including!photomedia!artists!and!the!project’s!playwright,!and!
community!participants.!Managerial!and!administrative!staff,!including!myself!and!the!local!
coordinator!for!the!Balaklava!project,!also!collected!and!produced!creative!materials.!
Such!art*based!methods!used!visual!and!audio!stimuli!to!‘understand!and!communicate!
about!the!world’!(Patton,!1981,!p.!217).!(RIPP:!Article!27,!p.101!has!examples.)+
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
54!While!delineated!in!the!above!documentation!strands,!content!from!each!was!often!related!to!each!other!and!the!larger!
suite!of!READ!outputs,!and!organised/!edited/!archived!in!multiple!ways!(UNESCO!and!ICCROM,!2015).!This!is!also!true!for!
the!application!of!some!documentation!methods!for!the!other!READ!strategies.!
!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 122!
•! Administrative!and!managerial!documentation.!This!involved!the!ongoing!process!of!
scanning!and!extracting!materials!from!Alphaville’s!program!development!and!production!
management!(including!operation!systems).!Sourced!were!predominantly!written!content!
such!as!guidelines,!delivery!summaries,!reports,!promotional!records!(including!branding),!
grant!submissions,!partner!agreements,!registration!records!(participants,!audiences),!media!
records,!communications,!meeting!notes!and!so!on.!My!dual!role!as!program!manager!
offered!an!excellent!vantage!point!to!access!and!intercept!relevant!content.!!
(RIPP:!Article!28,!p.106!includes!4!examples!and!Article!30,!p.111!has!a!media!report.)!
•! Nuclear!Futures!archive!of!creative!works!and!artefacts.!This!included!the!collection,!
storage,!and!digital!register!of!artworks!produced.!Data!was!sourced!from:!creative!team!
biographies!(artists,!community!partners,!participants,!art*workers);!art!form!specifications!
(interpretative!information);!artist!statements;!background!information;!descriptions!or!
representation!of!artworks/!processes;!publication!or!exhibition!records;!community!
information!(geographic,!demographic,!historical);!video!and!photography!of!arts!practice;!
thematic!coding;!and!cataloguing!of!the!content!files!and!copies!for!durable,!safe!
storage.!(RIPP:!Article!29,!p.109!includes!documentation!about!creative!works/artists.)!
•! Community!archives!and!resources!(external).!Inspired!by!the!intent!to!make!accessible!
significant!findings!unearthed!or!generated!during!the!project,!including!historical!and!
education!materials,!this!type!of!documentation!refers!to!the!oral!history!and!collection!
development!of!information/!artefacts!of!public!value.!Archiving!aimed!to!preserve!
documentation!held!by!those!who!have!witnessed!atomic!tests!and!those!who!have!
campaigned!for!recognition!and!reconciliation.!These!were!resources!distinct!from!the!
creative!works,!and!official!records!in!the!public!domain,!and!included!material!held!in!
communities!that!provided!unique!personal!perspectives!in!the!form!of!annotated!reports,!
medical!case!notes,!personal!maps,!photographs,!artefacts!and!documents.!Community!
archiving!was!a!collaborative!activity!by!the!creative!team!(including!myself),!community!
members!and!in!partnership!with!institutional!partners.!
•! Academic!research!and!documentation.!This!included!research!papers,!presentation!
documentation!and!materials!generated!through!linked!academic!scholarship!(see!Section!
5.4.4)!by!academic,!community!and!practitioner!researchers!on!the!CACD!activities!and!
themes,!exploring!the!wider!implications!of!the!community!arts!work!undertaken.!
As!summarised!above,!the!data!collection!and!analysis!methods!involved!creative!development!and!
research!(for!art’s!sake)!as!well!as!the!recording!of!the!CACD!processes,!and!cataloguing!and!archive!
development.!Analysis!of!documentation!data!was!done!via!a!shared!processes!of!curation!that!used!
!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 123!
administrative,!reflective!and!creative!methods!(such!as!editing,!publication!and!review!processes)!
to!produce!online!content,!blogs,!a!photo!book,!video!bites!and!case!studies.!The!interpretative!lens!
of!the!documentation!analysis!is!akin!to!Paul!Carter’s!(2010)!notion!of!‘reflective!archiving’*!where!
souvenirs!are!produced!from!projects!based!on!reflection!and!from!going!over!documentation!in!
order!to!re*evaluate,!to!discover!new!meanings!within!or!between!projects!and!to!reveal!lengthy!
exposure!of!the!arts!practice.!By!reviewing!past!documents,!photos,!and!artefacts!in!reflective!and!
creative!ways,!meaning!and!patterns!are!identified!(Carter,!2010).!Similarly,!Boud!describes!these!
documentary!processes!as!‘a!lens!for!learning’!(Boud,!2001,!p.9)!able!to!generate!grounded!analysis!
and!interpretative!knowledge.!(RIPP:!Article!27,!p.101*5!has!photos!related!to!the!archiving).!!
Results*
The!products!of!the!Documentation!strategy!from!the!Balaklava!project!included!raw!(unprocessed)!
content!and!produced!(processed)!outputs,!such!as!historical,!creative,!educational,!political,!
technical!and!administrative!data,!artefacts!and!resources.!Specifically!the!Documentation!outputs!
were:!video!and!audio!recordings;!extensive!photography!collections!and!artefacts;!data!and!log!
files;!administrative!data!base!contents;!an!online!cultural!repository!for!intergenerational!learning!
at!the!SLSA;!an!oral!history!collection,!testimony!and!artefacts!for!social!justice!campaigns;!original!
creative!works;!education!resources;!promotional!and!media!records;!and!bulk!raw!creative!content!
valued!as!a!future!resource!for!participating!communities!and!artists!to!own!and!use!in!beyond!the!
Balaklava!project.!Materials!were!disseminated!via!stakeholder!networks,!online!and!social!media!
channels,!traditional/mainstream!media!and!SLSA’s!public!archive!collection.!
The!involvement!of!photographers!and!film!makers!in!the!Balaklava!creative!team!meant!that!the!
use!of!video!and!photography!was!an!immediate!and!familiar!practice!to!be!used!for!understanding,!
exploring,!analysing!and!communicating!emergent!themes!arising!from!the!creative!arts!practice.!
Utilising!the!readily!accessible!expertise!from!the!creative!team,!combined!with!external!factors!
(such!as!the!affordability!and!access!to!good!quality!audio!and!video!equipment,!software!and!
online!media!platforms!even!from!the!regional!location!of!Balaklava),!enhanced!the!scope!and!
sophistication!of!the!documentation!collected!and!produced.!Creative!documentation!in!particular,!
helped!relate!READ!as!an!evaluation!framework!with!the!art*making!and!creative!development!
activities!of!the!project,!which!was!engaging!for!artists!and!community!members.!The!Nuclear!
Futures’!Balaklava!artists!recognised!the!rich!details!able!to!be!captured!through!documentary!art!
practice!and!because!the!ownership!of!material!remained!with!them,!co*owned!in!some!cases!with!
partners!and!Alphaville,!documentation!creative!content!had!the!potential!to!be!used!for!future!art*
making!outside!of!the!project!(in!accordance!to!with!the!authorship!and!consent!protocols).!!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 124!
The!end*products!were!of!short!and!long*term!relevance!for!the!project!and!partners.!Short!term!
needs!were!met!through!the!Documentation!deliverables,!with!both!documentary!(verbatim)!and!
story*based!representations!of!the!project’s!participants,!activities!and!impacts.!Documentation!as!a!
storytelling!device!brought!the!community!experiences!to!life,!representing!the!many!views!and!
voices!involved!in!the!project,!and!in!many!cases!provided!valued!gifts!for!community!partners.!
Community!partners!retained!access!and/or!copies!to!all!raw!footage!and!the!produced!outputs,!
such!as!the!published!photo*story!book!gifted!to!Avon!Hudson!and!family!as!a!project!keep*sake!
(RIPP:!Article!33,!p.121!has!photographs!of!the!book!to!demonstrate).!!
The!Documentation!strategy!resulted!in!the!emergent!language,!and!vibrancies!of!the!community!
arts!practice!to!begin!to!be!reflected!and!integrated!through!‘the!discursive,!the!pictorial,!the!
persuasive,!and!the!instrumental’,!and!challenged!the!privilege!traditionally!given!to!written!
language!in!the!research!and!evaluation!context!(Doloughan,!2002,!p.62).!Creative!documentation!in!
particular!was!relevant!to!the!community!partnerships!context!and!accommodated!the!different!
cultural!preferences!and!needs!for!some!stakeholders!as!identified!in!scoping.!Utilising!arts*based!
methods!was!valuable!as!a!means!to!represent!multiple!stakeholder!perspectives.!In!other!studies,!
researchers!argue!such!methods!‘evokes!different!ways!of!knowing!and!understanding!the!values!of!
the!program’!(Simons!&!McCormack,!2007,!p.!292;!Patton,!1981).!Art*based!methods!also!worked!
well!for!documenting!the!community!presentation!of!the!community!arts,!particularly!for!preserving!
ephemeral!aspects!of!the!showcasing!such!as!the!speakers!corner!for!the!10%Minutes%to%Midnight%
showcase!and!aspects/encounters!not!able!to!be!articulated!in!words!or!measured!by!metrics.!!
Lastly,!another!area!of!interest!arising!from!the!application!was!in!respect!to!the!embedded!long!
term!and!legacy!objectives!to!make!the!story!go!far.!The!documentary!art*making!and!storytelling!
methods!from!the!site!of!practice!produced!a!multi*platform!collection!of!nationally!significant!
documentation!as!part!of!the!creative!development!of!the!arts!practice,!and!the!READ!processes,!
including!rich!verbatim!narratives!and!documentary!textual!materials!that!were!engaging!for!READ!
collaborators!and!audiences.!In!the!medium!term!across!the!ongoing!arts!practice!of!Alphaville,!the!
creative!documentation!such!as!video!interviews,!enable!the!preservation!of!ongoing!creative!
conversations!within!the!broader!Nuclear!Futures!program!which!will!be!of!use!across!the!program’s!
three!years,!and!beyond.!Further,!it!is!envisaged!that!the!long*term!storage!and!public!access!to!
atomic!survivor!community!materials!donated!to!the!SLSA!collection!will!be!a!legacy!resource!for!
future!generations.!In!doing!so,!the!relevance!and!accessibility!of!project!artefacts!has!been!
extended!beyond!primary!participants!and!audiences!to!the!wider!public,!researchers,!historians!
and!artists!now!and!in!the!future.!(RIPP:!Article!6,!p.14!shows!an!early!plan.)!
!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 125!
!
5.5! READ#outputs,#sharing#and#use#
Integration*
While!the!READ!strategies!have!been!presented!here!as!discrete!methodologies,!in!reality!many!
methods!and!data!collection!processes!were!relevant!for!multiple!strategies.!As!is!outlined!through!
this!chapter,!mixed!data!was!generated!across!the!four!READ!strategies!of!Reflection,!Evaluation,!
Analysis!and!Documentation.!Overall!the!sheer!volume!of!data!meant!as!an!evaluator!and!manager,!
I!considered!this!outcome!as!a!data*utopia.!Processes!to!integrate!and!synthesise!data!were!used!in!
real*time!across!the!four!READ!strategies!(denoted!by!the!grey!arrow!in!previous!Figure!5.1),!and!at!
the!processing/!production!stage!when!findings!and!products!were!generated!to!meet!the!partner*
identified!end*uses.!The!integration!meant!reviewing!different!types!of!data!(quantitative!and!
qualitative)!together,!to!produce!findings!or!published!outputs!of!the!READ!process.!The!following!
discussion!spanning!the!integration,!utilisation!and!dissemination!correlates!to!the!fourth!research!
phase!undertaken!for!the!READ!application!(READ!outputs,!sharing!and!use).!
Spontaneous!and!planned!sessions!were!held!to!review!the!READ!data!for!both!formative!and!
summative!rounds!of!integration.!I!used!facilitation!techniques!to!encouraged!participation!in!the!
review!and!integration!of!the!READ!material,!for!example!coordinated!shared!collective!review!
sessions!(involving!presentation!and!discussion!of!READ!data!at!group!meetings!with!artists,!
administrators!and!community!partner!representatives)!and!individual!consultations.!The!
integration!process!was!semi*structured,!and!was!intended!to!facilitate!triangulation!of!the!data,!
where!data!from!different!sources!were!reviewed!to!verify!validity,!themes!and!consistency!or!
tensions!in!the!findings.!Integration!was!about!synthesising!data!to!create!sound!and!credible!
findings!of!use!and!meaning!to!partners.!In!the!process!of!integration,!I!reduced!the!data!into!
manageable!chunks!so!that!themes!and!findings!became!clear.!I!used!visual!approaches!including!
tables,!graphs!and!graphics!for!quantitative!responses.!!
! !
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 126!
The!following!analysis!techniques!were!used:!
•! Thematic!analysis:!examining!information!for!synthesis!potential!and!identifying!points!of!
divergence/!contrast;!includes!textual!analysis!and!comparative!analysis,!and!feedback!of!
analysis!via!peer!review.!
•! Reflective!analysis:!process!of!reviewing,!interrogating,!re*evaluating!and!making!tacit!
knowledge!explicit.!
•! Evaluative!reasoning:!framed!for!KEQs!and!using!rubrics/!performance!criteria.!
•! Quantitative!analysis:!generating!statistics!and/or!numerical*based!findings!from!project!
information.!
•! Creative!and!practice*led!processes:!selection,!editing!and!storytelling!through!
documentation,!creative!development!etc.!
During!these!processes!there!were!a!number!of!considerations!that!I!took!into!account!in!the!
planning,!as!READ!facilitator.!These!included:!determining!who!needed!to!be!involved!in!the!
collation!process!to!ensure!adequate!representation;!confidentiality!issues!during!analysis;�defining!
the!audience!for!READ!outputs!(to!suit!permissions!and!preferences);!and!ongoing!monitoring!of!the!
impact!on!organisations!and!individuals!involved!in!the!process.!!
READ*output*and*outcomes*
For!the!Balaklava!Community!Arts!Project,!a!collection!of!digital!and!hard*copy!outputs!resulted!
from!using!the!READ!methodology,!including:!
•! Raw!data:!databases!of!information,!statistics,!film!content,!photos,!audio!recordings,!oral!
history,!archival!resources!and!so!on.!
•! Management!resources:!an!evaluation!plan!for!Nuclear!Futures!program,!templates/pro!
formats!for!evaluation!instruments!and!reflective!practice,!evaluation!findings,!residency!
report,!media!reports,!partner!summaries!and!communications,!financial!reports!and!
visual!summaries!such!as!infographics.!(RIPP:!Article!35,!p.126!has!residency!report.)!
•! Evaluation!and!project!reports:!such!as!the!Acquittal!Report!delivered!to!Arts!SA!and!
modified!reports!for!various!partners.!(RIPP:!Article!36,!p.140!has!Acquittal!Report.)!
•! Academic!Research:!research,!conference!presentations,!Research!Plan!for!publications.!
•! Creative!content/!artwork:!significant!photomedia!repository,!website!archive,!photo!story!
book,!promotional!materials.!!
•! Spin!offs/pending/future!outputs:!creative!works!(such!as!documentary!film!and!webisodes),!
program/promotion/marketing!assets.!(RIPP:!Article!31,!p.119!has!promotional!video.)!
The!access!and!shared!copyright!(authorship)!vary!across!partners!and!outputs.!Demonstrations!
of!the!outputs!are!included!in!the!READ%in%Practice%portfolio!(RIPP:!Section!F,!p.118).!
!
!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 127!
In!the!process!of!the!READ!collaboration!there!were!also!various!outcomes.!For!Alphaville!(the!host!
organisation)!READ!resulted!in!the!generation!of!extremely!valuable!data,!for!understanding!and!
promoting!its!current!work!and!for!scoping!for!future!artwork!based!on!the!content!created.!Nuclear!
Futures!personnel!(artists,!art!workers,!administrators)!and!the!local!freelance!artist!participating!
directly!in!designing!and!implementing!READ!received!training!and!skills!development!(of!immediate!
and!future!benefit!for!their!practice).!Other!benefits!were!the!demonstrable!cultural!outcomes!
evidenced!for!the!Alphaville!organisation!from!the!reflective!practice,!such!as!the!development!of!a!
culture!of!learning!at!the!project,!program!and!organisational!level,!which!I!evaluated!to!be!an!
indication!of!capacity!building!for!the!community!arts!organisation!as!a!whole.!There!were!also!
many!material!benefits!of!the!READ!process!for!contributors!and!Alphaville,!such!as!templates,!plans!
and!resources!readily!applicable!for!other!Nuclear!Futures!READ!applications.!Relevant!for!funding!
partners,!Australia!Council!for!the!Arts!and!Arts!SA,!READ!produced!information!for!inclusion!in!
regular!reporting!(included!evaluation!findings!as!they!emerged)!and!acquittals.!
For!the!community!partners,!one!of!the!original!interests!was!to!‘make!the!story!go!far’!(Alphaville,!
2015).!For!this!objective,!READ!had!myriad!outcomes,!in!addition!to!the!community!arts!outcomes,!
and!these!will!continue!into!the!future.!For!the!Hudson!family!in!particular,!guidance!and!formalised!
arrangements!resulted!in!their!establishing!a!long*term!relationship!with!SLSA!in!relation!to!the!use!
of!their!significant!private!archive.!Also!large!volumes!of!documentation!(such!as!photos,!videos,!
transcripts!and!so!on)!created!valuable!resources!that!were!gifted!back!to!community!participants!
through!the!READ!process.!In!other!instances,!such!as!for!community!participants!involved!in!the!
Field%of%View!photography!exhibition,!the!digital!documentation!meant!their!works!were!
published!and!disseminated!to!wider!audiences!via!an!online!gallery!(RIPP:!Article!32,!p.120).!
Interestingly!through!READ!processes,!particularly!from!the!Academic!strategy,!the!mixed!cohort!
arising!from!the!creative!community!partnership!initiative!(involving!community!members,!artists,!
researchers,!art!workers!and!activists)!formed!a!community!of!practice.!The!benefits!of!communities!
of!practice,!within!which!knowledge!construction!is!made!explicit,!have!been!enunciated!by!several!
analysts!(Cox,!2005;!Hildreth!&!Kimble,!2004;!Wenger,!McDermott,!&!Snyder,!2002;!Lave!&!Wenger,!
1991).!For!Nuclear!Futures,!the!community!of!practice!evolved!around!the!common!interest!in!the!
domain!of!‘art!and!the!bomb’,!created!with!the!goal!of!making!explicit!the!knowledge!generated!
through!creative!practice,!and!publishing!original!knowledge!as!research.!This!was!clearly!
demonstrated!when!contributors!from!Nuclear!Futures,!and!beyond,!came!together!at!the!two*
day!academic!seminar!at!QUT!in!August!2015!(RIPP:!Article!34,!p.!122).!Further!discussion!on!the!
outputs!and!outcomes!of!READ!are!included!in!the!reflective!practice!chapter!next,!in!Chapter!6.!!
!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 128!
Utilisation*and*dissemination*
READ!data!and!outputs!were!used!as!a!lens!to!help!artists,!administrators!(including!myself)!and!
community!participants!to!understand!and!review!the!community!arts,!which!Schon!would!refer!to!
as,!in*!and!on*!action!(see!Schon,!1983,!and!Chapter!2!for!elaboration!of!this!concept).!Scriven!(2012)!
describes!evaluation!in!terms!of!its!purpose!and!timing,!and!how!these!factors!influence!the!scope!
for!evaluation!utilisation.!For!READ,!the!evaluation!data!produced!was!generated!and!used!
throughout!the!project!for!the!following!purposes:!formative!(during!the!early!stages!of!
implementation);!developmental!(ongoing!or!for!uncertain!and/or!innovation*oriented!contexts);!
and!summative!(once!activity!is!completed).!Thus,!the!utilisation!of!the!data!collected!was!of!
immediate,!short*term,!intermediate!and!long*term!benefit.!!Figure!5.5!summarises!the!timeline!of!
project!and!READ!activities!(generalised)!and!the!formative!and!summative!READ!outputs!that!were!
generated!from!this!ongoing!utilisation!(refer!to!‘Key!for!Diagram’!box,!Figure!5.5,!for!explanation).!
As!illustrated!in!Figure!5.5,!the!arrows!illustrate!cycles!of!utilisation!arising!from!one!round!of!READ!
activity!and!fed!directly!into!the!planning!or!delivery!of!the!project.!For!example,!data!from!
reflective!practice!and!the!evaluation!instruments!used!during!the!March!‘Pop*up’!residency!was!
assimilated!and!disseminated!immediately!for!partner!updates,!program!management!purposes,!
and!to!inform!creative!development!and!continuous!improvement!for!the!subsequent!residencies.!
The!internal!cycles!of!utilising!READ!generate!data!also!extended!beyond!the!Balaklava!project,!to!be!
fed!into!the!broader!program!management!of!the!Nuclear!Futures!program.!Most!significantly!was!
the!value!created!by!the!READ!outputs!and!findings!for!the!planning!of!subsequent!Nuclear!Futures!
exhibitions/projects,!such!as!the!July*August!10%Minutes%to%Midnight%showcase!season!at!The!Block,%
QUT!and!the!later!Screengrab7!exhibition!of!10%Minutes%to%Midnight!at!the!Pinnacles!Gallery,!
Townsville!in!December!2015!–!February!2016.!
In!addition!to!the!cycles!of!utilisation!internally!for!the!program,!were!cycles!of!reporting!back!to!
communities!and!partners!(not!illustrated!in!Figure!5.5).!Presentation!and!reporting!formats!were!
designed!to!meet!stakeholder!needs!and!interests.!In!addition,!READ!generated!materials!were!also!
disseminated!to!industry,!academic!and!peer!forums!via!seminars,!presentations,!mainstream!
media,!online!platforms,!and!community!networks.!As!was!indicated!from!the!scoping!consultation,!
partner!interests!in!the!utilisation!of!READ!data!and!products!was!divergent.!While!the!examples!of!
utilisation!and!dissemination!resulting!from!the!READ!application!are!too!numerous!to!cover!in!full,!
below!are!a!number!of!examples.!!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 129!
Figure!5.5:!!READ!timeline!of!activities!and!outputs!for!Balaklava!Community!Arts!Project!!
Community Photography Exhibition
Pre-project Planning and partnership development
Public showcase/events
2013 Dec
2014 Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
2015 Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
READ planning (Balaklava project)
Exhibition
Residency: Photography workshops
Residency: public events, creative
Fieldwork: RAD Tour
Residency: planning & partnerships
Reflection, Evaluation, Documentation
Academic Seminar, QUT
Nuclear Futures showcase/events, QUT
Reflection, Evaluation, Analysis Documentation
READ instruments developed
READ model devised 2012 2013
Reflection, Evaluation, Documentation
Reflection, Documentation
Reflection, Documentation
Reflection, Evaluation, Documentation Curation of community exhibition
Residency: Creative development workshop Reflection, Documentation
Reflection, Documentation, Analysis
Creative development Reflection, Analysis, Documentation
Reflection, Documentation, Analysis
Reflection, Evaluation, Analysis, Documentation
Reflection, Evaluation, Analysis, Documentation Residency: Project closure meetings
READ instruments developed for showcase
Summative READ integration and outputs
Formative READ Outputs Draft Evaluation Plan Evaluation instruments Reflections Creative documentation Oral history READ data Partner updates Monthly reports Media/ blogs/ website Presentations (industry/academic) Research Evaluation Findings AES conference presentation
Reflection, Documentation, Analysis
Reflection, Documentation
Future!utilisation!!
Cycles of utilisation
Planning workshop (delivery partners)
Formative READ integration (ongoing)
Summative READ Outputs Refined Evaluation Plan Project Acquittal Partner reports/ summaries Evaluation Findings Creative documentation Creative works Hardcopy photo-story book Research Media/ blogs/ websites Presentations (industry/academic)
Fieldwork: BNTVA, UK
Key for Diagram Abbreviations: READ (denotes the READ model and activities, integrating reflection, evaluation, analysis and documentation); RAD Tour (Radioactive Exposure Tour, organised by Friends of the Earth); BNTVA (British Nuclear Test Veteran Association); QUT (Queensland University of Technology). Explanation: On the left is the project timeline. Marked along the timeline are the community arts activities (black) and READ activities (orange). The data generated from READ activities usually had immediate uses, as well as a cycle of utilisation that fed into the next creative residency or activity (see arrows). During the project data also had long-term cycles of utilisation (not indicated in diagram). On the right side, are examples of the outputs produced in the formative and summative phases of the project’s READ activities.
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 130!
For!some!artists!and!community!partners,!the!material!was!used!for!social!justice!and!political!
campaigning,!for!others,!READ!materials!added!to!their!personal!archives.!For!example,!artist!Jessie!
Boylan!(of!the!Atomic!Photographers!Guild)!will!continue!to!exhibit!and!further!develop!the!creative!
documentation!in!line!with!her!anti*nuclear!activism,!and!present!the!material!in!the!future!with!
other!peace*related!and!political!organisations.!Likewise,!community!member!Avon!Hudson!has!
used!the!documentation!about!the!creative!works!as!a!story*telling!devise,!a!tool!for!communicating!
on!the!broader!issues!of!atomic!testing!in!his!networking!and!advocacy!work.!For!the!local!
government!partner,!Wakefield!Regional!Council,!evaluation!and!documentation!were!used!to!fulfil!
reporting!and!strategy!development!purposes.!Information!was!reported!back!to!the!Council!and!
staff,!and!filed!as!a!record!of!local!CACD!activities.!Venue!partners!in!Balaklava!utilised!
documentation!and!information!for!venue!promotions!and!as!a!means!to!gain!insight!into!how!well!
the!venues!worked!as!a!Fringe!Festival!venues!(in!terms!of!accessibility,!capacity,!functionality,!
audience!experience),!given!they!were!previously!unregistered!as!Fringe!venues.!For!funding!bodies,!
the!READ!outputs!were!used!for!accountability!purposes!and!for!final!project!acquittal.!
Alphaville!and!Nuclear!Futures!personnel!used!the!READ!outputs!for!continuous!improvement!and!
knowledge!generation!to!advance!their!community!arts!practice.!As!program!manager,!I!have!
utilised!the!data!and!outputs!of!the!READ!process!in!wide*ranging!and!ongoing!ways:!as!reference!
material,!for!communications!and!promotion,!and!for!managerial!decisions.!One!specific!example!
was!the!high!utility!value!of!the!READ!content!in!the!planning!and!curation!of!a!subsequent!
showcase!of!the!artwork!at!The!Block,!QUT.!In!this!case,!the!lessons!from!the!Balaklava!project!
(produced!through!READ)!strongly!informed!the!creative!team’s!approach!to!the!development!of!a!
second!projection!installation!Ngurini%(Searching),!as!well!as!underscored!decisions!on!the!
presentation,!production,!management,!and!resourcing.!Another!demonstration!of!utility!is!that!
Alphaville,!and!several!artists!independently,!have!used!documentation!and!evaluation!data!in!new!
funding!applications!for!spin!off!projects,!and!in!the!promotion!of!their!arts!practices!online.!!
Lastly,!READ!material!was!disseminated!in!the!public!domain!as!long*term!artefacts!for!broader!
learning!and!future!utilisation.!The!first!example!is!the!SLSA’s!collection!development!to!ensure!that!
nationally!significant!material!unearthed/generated!through!the!project!remains!publicly!accessible!
for!future!use!by!researchers!and!artists.!A!second!example!is!demonstrated!by!the!planned!book!
publication!based!on!the!outcomes!of!the!community!of!practice!(established!through!the!Analysis!
strategy!of!READ),!another!means!of!creating!longevity!and!relevance!beyond!the!timeframe!and!
bounds!of!the!original!arts!practice,!and!the!evaluation!practice.!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 131!
5.6! Conclusion#
Characterised!by!a!diverse!program!of!creative!arts,!involving!multiple!community!and!institutional!
partnerships,!the!Balaklava!Community!Arts!Project!provided!a!dynamic!and!fascinating!test!ground!
to!explore!the!functionality!and!nuances!of!the!READ!approach!in!action,!within!the!community!arts!
context.!To!examine!the!application,!I!adopted!a!systems!approach!to!document!and!profile!the!six!
core!elements!of!the!READ!concept:!the!supporting!structures;!Reflection;!Evaluation;!Analysis;!
Documentation;!and!integration,!utilisation!and!dissemination.!My!role!as!researcher!was!to!lead!
the!tailored!design!and!implementation!of!the!READ!methodology!for!the!Balaklava!project!as!a!
collaborative!process!with!participating!partners!and!stakeholders.!The!methodology!developed!and!
applied,!as!described!in!this!chapter,!reflects!the!story!of!the!Nuclear!Futures!(Alphaville)!site!of!
practice!and!a!diverse!cohort!of!contributors!and!participants.!Arising,!is!an!extensive!body!of!work!
that!has!formulated!and!substantiated!many!of!the!technical!and!philosophical!aspects!of!the!READ!
approach.!
As!demonstrated!through!the!Balaklava!application,!the!READ!methodology!employed!a!series!of!
known!learning!and!appraisal!strategies!relevant!to!community!arts!and!management,!and!these!
utilised!an!innovative!suite!of!qualitative,!quantitative!and!performative!methods.!READ’s!integrated!
framework!supported!wide*ranging!evaluative!action!by!artists!and!community!participants,!and!
harnessed!the!strengths!and!resources!of!the!art*making,!management!structures!and!
partners/organisations!embedded!in!the!community!arts!practice.!Participants!and!artists!engaged!in!
reflective!journaling,!video!diaries,!oral!history,!interviews,!art*making,!creative!documentation,!
surveys,!media,!vox!pops,!questionnaires,!a!community!of!practice!and!research.!A!number!of!
original!and!constructive!processes!for!partnership!development,!creative!development,!engagement!
and!organisational!learning!were!actioned,!and!a!diverse!portfolio!of!summative!materials!(both!
evaluative!and!non*evaluative)!resulted,!tailored!to!suit!the!multiple!needs!of!the!partnerships.!!
The!theoretical!and!practical!demonstration!of!the!READ!approach!provided!in!this!chapter!lays!the!
foundation!for!the!critical!analysis!of!the!READ!model!of!practice!as!it!emerged!and!was!refined,!and!
this!is!the!central!concern!of!Chapters!6!and!7.!As!detailed!in!the!following!chapters,!I!employed!
forensic!reflective!practice!as!the!principal!research!strategy!to!review!READ’s!performance!as!a!
framework!for!learning!and!appraisal.!The!aim!of!the!reflective!analysis!was!to!draw!out!the!tacit!
learnings!arising!from!the!evaluation!practice!and!to!inform!the!further!development!and!refinement!
of!the!READ!model,!in!order!to!establish!a!consolidated!READ!product!for!future!use.!The!findings!of!
this!critical!review!of!the!activities!outlined!in!this!chapter,!are!included!in!Chapter!6.!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 132!
!
! !
Blank!page!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 133!
6! Findings##
Chapter!5!detailed!the!collective!design!and!implementation!for!the!industry!trial!of!the!integrated!
READ!evaluation!approach!for!the!Balaklava!Community!Arts!Project!in!South!Australia,!a!process!
which!I!led!as!researcher!for!the!Doctor!of!Creative!Industries!(DCI).!Developing!an!operational!
model!of!practice!for!the!READ!concept!involved!an!evaluation!needs!assessment!process,!ongoing!
consultation!with!stakeholders,!and!the!integration!of!data!collected!via!the!methodological!design!
of!READ’s!four!key!learning!and!appraisal!strategies!–!Reflection,!Evaluation,!Analysis!and!
Documentation!(READ).!Also!documented!in!Chapter!5!were!the!wide!range!of!dissemination!and!
utilisation!outcomes!linked!to!the!outputs!generated!from!the!READ!methodology!devised.!At!the!
site!of!practice,!the!international!creative!communities!partnership!initiative!‘Nuclear!Futures’!
(funded!by!the!Australia!Council!for!the!Arts!and!Arts!South!Australia),!the!integrated!READ!learning!
and!appraisal!processes!generated!many!immediate!and!medium*term!benefits.!
Concurrent!with!delivering!the!READ!application,!I!conducted!an!ongoing!process!of!forensic!
reflective!practice,!focused!on!my!evaluation!practice,!as!well!as!on!the!development!and!
performance!of!the!READ!model.!This!enquiry!was!driven!by!two!overarching!research!questions:!
a)! How!does!the!READ!model!contribute!to!program!learning!and!appraisal!for!creative!
community!partnerships!within!community!arts,!based!on!the!Nuclear!Futures!experience?!!
b)! What!innovation!does!READ!offer!evaluation!practice!and!program!evaluation?!
Chapter!6!establishes!the!research!findings!based!on!the!critical!enquiry!undertaken!during!the!DCI,!
via!forensic!reflective!practice,!and!is!the!basis!for!the!summative!conclusions!in!Chapter!7.!
This!chapter!is!written!from!my!perspective!as!evaluation!practitioner,!and!offers!insight!into!the!
collective!experience!at!the!site!of!practice!evidenced!while!trialling!the!READ!approach.!I!reflect!on!
the!READ!application!and!the!complex!interactions!between!the!stakeholders!and!the!READ!system,!
and!the!outcomes!generated!through!the!creative!and!participatory!processes!of!the!evaluation!
practice.!My!reflective!analysis!covers!the!major!themes!and!findings!that!arose,!which!I!then!used!
to!advance!the!READ!model!development.!Meta*evaluation!techniques!were!employed!to!analyse!
the!performance!of!the!READ!model!in!action,!and!to!identify!the!strengths,!constraints!and!
opportunities!of!the!READ!approach.!In!particular,!I!outline!the!relevance!of!positioning!the!READ!
facilitation!internally,!explore!the!versatility!of!the!model!as!a!modular!or!integrated!system,!and!
examine!several!of!the!READ!methods!that!demonstrated!strong!results,!such!as!collective!reflection!
and!creative!documentation.!The!chapter!culminates!in!the!presentation!of!the!developed!READ!
product,!which!includes!an!infographic!and!a!list!of!key!considerations!for!future!evaluation!practice.!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 134!
6.1.1! Analysis#of#the#READ#application#
This!chapter!is!divided!into!two!parts.!First!is!the!critical!analysis!of!the!collaborative!design!and!
implementation!of!the!READ!methodology!for!the!Balaklava!Community!Arts!Project!(Section!6.2).!It!
details!the!challenges!and!successes!of!the!evaluation!practice!and!sheds!light!on!the!complexities!
associated!with!actioning!the!READ!approach.!The!analysis!identifies!what!aspects!of!READ!
performed!well!in!the!Balaklava!context,!details!examples!of!necessary!modifications!for!improving!
the!system’s!function,!and!highlights!some!of!the!unforeseen!and!surprising!consequences!of!the!
application.!The!purpose!of!the!analysis!is!to!produce!findings!to!inform!the!refinement!of!the!READ!
approach!so!that!a!general!model!of!practice!can!be!developed.!In!making!explicit!the!strengths!and!
weakness!of!the!model!(based!on!the!Balaklava!experience),!the!forensic!reflective!practice!aimed!to!
identify!aspects!of!the!model!that!need!further!development!beyond!this!research,!as!well!as!any!
areas!of!innovation!that!may!be!of!relevance!for!broader!evaluation!practice!in!community!arts.!!
The!themes!examined!are!the:!modular!and!integrated!approach!of!READ;!trialled!social!
constructivist!stance;!internal!facilitation;!emergent!evaluation!practice;!performance!(as!assessed!
via!meta*evaluation);!and!opportunities!for!enriching!future!evaluation!practice.!
During!my!reflective!analysis!across!these!themes,!I!approached!reflection!on!practice!through!the!
lens!of!my!own!internally!derived!goals!for!betterment!of!my!professional!practice!and!the!
continuous!improvement!of!evaluation!practice!at!the!Nuclear!Futures!site.!This!strategy!generated!
reflective!analysis!of!my!journey!as!the!READ!facilitator,!with!a!frame!of!reference!responsive!to!the!
information!and!insights!being!generated!through!the!collective!experience,!in!accord!with!my!own!
judgment!and!evaluation!experience.!As!outlined!in!Chapter!3,!data!for!my!meta*analysis!was!
collected!through!written!journaling,!video!diaries,!stakeholder!interviews!and!consultation,!
observation!and!use!of!structured!evaluative!tools!from!the!literature.!A!number!of!predominant!
themes!and!findings!emerged!and!these!are!covered!in!Section!6.2.!!
The!second!part!of!the!chapter,!Section!6.3,!consolidates!the!lessons!from!practice!and!presents!the!
substantiated!READ!product!that!has!been!designed!during!the!research,!that!offers!an!evaluation!
tool!for!future!practice!through!the!Nuclear!Futures!program.!The!infographic!in!Section!6.3.3!is!the!
outcome!of!this!product!development!and!is!supported!by!a!profile!summary!that!identifies!the!
characteristics,!assumptions,!constraints!and!variables!of!the!approach.!Section!6.3!articulates!the!
model!of!practice!developed!for!the!READ!approach!that!will!be!relevant!to!my!own!evaluation!
practice!in!the!future,!as!well!as!the!evaluation!practice!of!the!Nuclear!Futures!program!in!2016*7.!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 135!
Before!presenting!the!findings!from!the!critical!analysis,!the!following!section!provides!an!overview!
of!the!reflective!questions!that!emerged,!and!that!guided!the!forensic!reflective!practice.!!
6.1.2! Questions#arising#from#forensic#reflective#practice#
The!Doctor!of!Creative!Industries!(DCI)!research!strategy!of!forensic!reflective!practice!invites!the!
opportunity!to!question!everything.!Reflective!practice!also!has!particular!relevance!as!an!
evaluator’s!tool,!and!for!critical!reflection!and!meta*evaluation!for!continuous!improvement!
(Davidson,!2005).!For!the!purpose!of!the!research!reflective!practice!became!the!lens!for!examining!
how!well!the!READ!model,!developed!during!the!research,!performed!as!a!learning!and!appraisal!
framework.!In!addition,!for!the!purpose!of!professional!development,!the!reflective!strategy!was!a!
means!to!scrutinise!and!advance!my!own!evaluation!practice.!!
Questions!about!why!elements!of!the!evaluation!practice!worked,!or!did!not!work,!generated!rich!
and!insightful!data.!Questions!also!focused!on!what!could!have!been!done!differently,!and!the!
consideration!of!how!my!answers!might!be!viewed!from!others’!perspectives.!To!my!surprise,!the!
forensic!reflective!practice!conducted!across!the!research!generated!a!large!volume!of!questions.!
The!process!of!questioning!was!as!valuable!as!the!process!of!generating!my!responses!and!analysis.!
Many!of!the!questions!arising!were!investigated!through!applied!reflection!on!the!DCI!process,!and!
others!served!as!technical!or!epistemological!signposts,!which!marked!important!conundrums!
beyond!the!scope!of!the!enquiry,!yet!were!relevant!for!future!investigation!and!contemplation.!
While!the!reflective!questions!produced!from!the!structured!and!reflexive!sessions!were!generated!
to!direct!analysis!towards!the!research!problem,!the!collection!of!questions!that!emerged!
constitutes!a!valuable!research!output!that!is!of!relevance!for!my!future!evaluation!practice.!!
Below,!in!Box!6.1,!is!a!summary!of!selected!reflective!questions!arising!during!the!research,!
generated!from!the!written!exercises!and!video!diary!sessions,!which!include!a!mix!of!strategic!and!
spontaneous,!open!and!closed,!and!evaluative!and!descriptive!(non*evaluative)!questions.!As!a!
collection,!the!questions!arising!from!the!forensic!reflective!practice!demonstrate!the!depth!and!
breadth!of!considerations!associated!with!the!pursuit!of!best!practice,!and!highlight!the!complexity!
involved!in!examining!evaluation!practice!to!check!validity,!utility,!ethical!conduct,!credibility!and!
cost!effectiveness!during!community!arts!program!appraisal.!Their!relevance!extends!beyond!the!
appraisal!of!the!READ!application,!and!provides!insight!into!the!plethora!of!issues!facing!
practitioners!engaged!in!continuous!improvement!of!evaluation!practice.!!
!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 136!
Box!6.1:!Reflective!questions!arising!from!the!forensic!reflective!practice!sessions!for!the!DCI!
!
Questions about READ’s performance as a framework for appraisal of arts-based activities with multiple creative community partnerships:
• What worked well? What didn’t? • What were the challenges for implementation? How were potential limitations or inhibitors overcome? • What does a SWOT analysis on the application reveal? • Any failures? Why? • What outcomes are evident during and post application? • How well did READ perform as a guiding framework for program development and/or learning? • What were the results of the participatory elements? • Where those implementing READ familiar enough with the context? • Was there adequate evaluation expertise in evaluation and the subject of the evaluation? • How well were any conflicts of interest dealt with? • What does the experience of stakeholders say about READ? • Were outcomes and outputs satisfactory and useful for partner needs/expectations? How well? • Were findings and communications timely for decision-making processes? • Were the costs and inputs incurred by stakeholders and partners reasonable in relation to benefits? • What was the opportunity costs of the personnel devoted to READ application? • Were the conclusions arising from READ valid and justified? Clearly communicated? • Does it suit the context and partnership challenges? • How does READ perform against Scriven’s Key Evaluation Checklist for meta-evaluation? (Scriven,
2003) • Where did READ deliver most value? Were there omissions? If so, were they serious enough to
compromise validity or accuracy of findings? (Davidson, 2005) • How well was the READ application conducted in terms of ethical and professional standard
measures? (Australasian Evaluation Society, 2010) • Was the design and implementation of the tailored READ model culturally appropriate? On what
grounds? • What improvements/refinements of the model and support structures are necessary? • Are findings and lessons transferable and relevant for future practice? • What aspects of READ were particularly relevant to the needs for the creative community
partnerships of the Nuclear Futures Partnership Initiative experience? • What practical lessons arise for the use of READ at the Nuclear Futures site of practice? • How well does the model generalise to be of relevance to other practitioners? • What are the assumptions, variables and constraints of the application and site of practice? • In what ways and to what extent did the READ model advance evaluation practice? • What outstanding areas of innovation for community arts management and evaluation are
identifiable? What attributes were unique, effective, creative and/or outstanding? • What is the scope and significance of impact during the application? • What outcomes, impacts or dynamics were surprising or unexpected? • What can be improved for future READ applications? • How does the READ application at the site of practice relate to the community arts evaluation? • What are the original aspects of the READ model? • What does the READ product arising from the research look like? • What areas of READ need further research and development? What are areas of future research or
direction?
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 137!
6.2! Findings#and#discussion#
6.2.1! Modular#and#integrated#approach#
Through!the!development!of!the!READ!product,!I!examined!the!function!and!evaluative!capacity!of!
each!READ!strategy!to!build!a!theoretical!understanding!of!the!model!and!critically!review!READ’s!
performance!as!a!framework!for!appraisal.!For!the!purpose!of!analysis,!I!defined!appraisal!to!
encompass!the!broad!definition!of!evaluation!formulated!at!the!site!of!practice!(which!included!
merit*driven!processes!as!well!as!descriptive!accounts!without!judgement),!as!well!as!those!more!
unassuming!processes!of!sensemaking!and!reflective!assessment!that!were!inherent!in!the!creative!
arts!practice.!I!identified!that!the!focus!of!appraisal!at!the!site!of!practice!related!to:!the!creative!
processes;!artworks!produced;!community!engagement;!project!progress;!and!the!social!dynamics!
and!quality!of!the!partnerships.!
As!articulated!in!Chapter!5,!each!READ!strategy!(Reflection,!Analysis,!Evaluation,!Documentation)!
was!driven!by!different!impulses!and!approaches!to!learning!and!appraisal,!and!drew!on!different!
method!sets.!To!gain!a!holistic!picture!of!the!tailored!READ!framework!for!the!Balaklava!project,!I!
was!interested!in!understanding!what!appraisal!activities/methods!operated!across!all!of!the!four!
strategies,!as!well!as!those!undertaken!specifically!outside!the!Evaluation!strategy!(which!already!
resembled!conventional!evaluation).!My!intent!was!to!analyse!what!the!other!three!READ!strategies!
(and!the!integrated!system!as!a!whole)!contributed!to!the!appraisal!outcomes,!beyond!the!
Evaluation!strategy!alone.!In!one!reflective!practice!session!for!example,!I!considered:!What!is!the!
difference!between!READ!as!an!integrated!approach!and!a!conventional!evaluation*only!lens?!
The!systems!approach!was!useful!for!determining!and!exploring!the!activities!of,!and!relationships!
between,!the!strategies!(see!systems!approach!in!Section!5.1.2).!In!summary,!the!four!READ!
strategies!were:!reflection!for!establishing!a!culture!of!reflective!practice;!evaluation!driven!by!
agreed!key!evaluation!questions!in!order!to!make!evaluative!conclusions;!analysis!via!academic!
research!to!generate!broader!learning!and!dissemination!of!original!knowledge;!and!documentation!
to!deliver!creative,!program!and!legacy!artefacts!to!help!sensemaking!within!the!project!and!the!
broader!field!of!community!arts.!The!study!in!Chapter!5!showed!that!across!all!four!strategies,!a!
diverse!mix!of!appraisal!methods!were!employed,!including!conventional!evaluation!instruments,!
qualitative!methods!(such!as!interviews!and!group!feedback!sessions),!as!well!as!creative!and!
reflective!methods.!This!suite!of!methods!was!used!for!reviewing!the!art*making!(creative!
development,!content,!production!and!community!arts!and!cultural!development!(CACD)!practice),!
partnership!development,!and!the!artistic!vibrancy!of!the!creative!work!being!generated.!It!was!
evident!that!the!evaluative!processes!from!each!strategy!overlapped,!and!I!was!surprised!at!how!far*
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 138!
ranging!the!evaluative!activities!were!across!the!READ!system,!and!also!how!many!were!directly!
drawn!from!the!creative!practice!and!culture!of!reflective!practice.!!
The!Analysis!and!Documentation!strategies!produced!valuable!descriptive!content,!which!
predominantly!did!not!focus!on!evaluative!conclusions.!Rather,!these!strategies!illuminated!several!
of!the!broader!implications!of!the!community!arts!practice!in!its!wider!social!context.!Even!for!
outputs!produced!that!did!not!include!overt!evaluative!conclusions,!it!was!evident!in!many!instances!
that!the!processes!involved!in!generating!those!outputs!contained!embedded!evaluative!processes,!
such!as!the!framing!of!storytelling,!and!the!choices!made!during!any!form!of!visual!representation!or!
video!editing.!The!implication!was!that!READ!produced!a!mix!of!evaluative!and!non*evaluative!
outputs!that!were!all!considered!the!results!of!evaluation!practice!through!the!READ!approach!at!
the!site!of!practice!(see!Section!5.5!for!a!discussion!on!utility!and!READ%in%Practice%portfolio!
(RIPP):!Section!F,!p.117!for!examples!of!the!mix!of!READ!outputs).!
I!established!that!while!reflection,!analysis!and!documentation!are!certainly!all!well!known!and!are!
already!used!in!conventional!evaluation!as!methods,!the!difference!was!that!within!the!READ!
framework!they!operated!as!strategies!that!served!their!own!set!of!strategic!objectives.!Each!of!the!
READ!strategies!had!independent!yet!interrelated!plans!for!knowledge!generation,!related!to!the!
mix!of!evaluative!and!non*evaluative!aims!of!the!READ!system!as!a!whole.!Sometimes!the!methods!
used!for!fulfilling!each!strategy!overlapped.!One!significant!finding!was!how!the!benefits!from!one!
strategy!had!the!potential!to!enable!or!enhance!the!success!of!another.!For!example,!the!sustained!
culture!of!reflective!practice!meant!a!continuous!supply!of!data!relevant!for!Evaluation,!Analysis!and!
Documentation.!
I!concluded!that!when!combined,!the!four!strategies!generated!appraisal!and!learning!outcomes!
that!would!not!have!been!achieved!by!the!Evaluation!strategy!alone,!and!secondly!that!the!other!
three!READ!strategies!enhanced!the!effectiveness!of!the!Evaluation!strategy!in!addressing!the!
project’s!key!evaluation!questions.!Examples!of!the!outcomes!that!would!not!have!been!created!by!
the!Evaluation!strategy!alone!included!the!goal*free!creative!documentation,!the!community!of!
practice!(of!researchers!and!community!partners)!established,!community!archiving!and!public!
collection!development,!and!the!wider!dissemination!pathways!that!were!developed!for!the!
dissemination!of!findings!to!industry!and!academic!audiences.!The!results!from!the!READ!application!
demonstrated!that!bringing!together!data!generated!from!the!four!READ!strategies!enabled!a!wide!
summative!lens!to!be!created!for!recording,!reviewing!and!representing!community!arts!activities.!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 139!
In!addition,!there!were!also!conceptual!benefits!arising!from!the!integrated!approach!of!the!four!
READ!strategies.!At!the!site!of!practice,!I!observed!that!the!READ!approach!shifted!the!focus!off!
‘evaluation’!and!onto!a!broader,!experimental!endeavour!of!sensemaking!about!the!arts!practice,!
and!this!was!engaging!for!stakeholders!(including!those!who!had!made!known!their!trepidation!and!
scepticism!towards!evaluation).!In!particular,!the!engagement!with!artists!was!secured!through!the!
creative!documentation!and!reflection!practices,!and!for!the!community!partners,!their!interest!in!
READ!was!ignited!by!the!legacy!and!broader!learning!opportunities!it!promised.!Once!engaged!and!
familiar!with!the!READ!approach,!I!observed!that!participation!then!spilled!over!into!Evaluation!and!
Analysis!activities,!and!that!there!was!renewed!receptiveness!to!the!more!standard!types!of!
evaluation!methods!within!the!READ!set.!Also,!once!trust!and!interest!had!been!established,!
participating!stakeholders!were!exposed!to!new!methods!(previously!unfamiliar!to!them),!and!many!
participated!in!data!collection!and!analysis!activities!that!were!beyond!their!original!sphere!of!
interest!or!which!may!have!lacked!credibility!initially!(for!them).!
One!example!is!as!follows.!In!a!planning!meeting!to!design!the!secondary!school!engagement!
sessions!(which!were!to!occur!after!students!had!viewed!the!10%Minutes%to%Midnight!projection!
installation!as!part!of!the!education!program!for!schools!(see!Chapter!4)),!I!asked!the!creative!team!
if!they!could!facilitate!the!student!evaluation!as!part!of!the!‘meet!the!artist’!session.!This!would!
involve!facilitating!the!process!of!students!filling!out!the!evaluative!feedback!forms!(included!in!
RIPP:!Article!19,!p.70).!The!artists!had!previously!expressed!interest!in!attaining!feedback!from!
students,!however!the!forms!were!a!much!more!structured!method!than!they!had!envisaged.!While!
this!was!agreed!to,!I!observed!some!reluctance!to!engage!with!the!delivery!of!the!standard!
evaluation!instruments.!I!got!the!impression!that!the!forms!were!perceived!as!a!bureaucratised!
means!of!engaging!with!the!audience.!!!
We!trialled!the!creative!team!delivering!the!evaluation!instrument!in!the!first!student!class!session,!
and!as!the!sessions!progressed,!the!artists!became!increasingly!engaged!in!the!evaluation!data!that!
was!being!generated!and!the!richness!of!the!resulting!discussions!with!students!(because!students!
had!already!formulated!ideas!via!the!form!to!input!into!the!collective!discussion).!The!data!collected!
became!immediately!relevant,!both!as!feedback!on!the!creative!work!and!as!a!guide!for!the!iterative!
improvement!of!how!the!sessions!with!students!worked!best.!By!the!third!student!class!through!the!
installation,!the!artists!were!gathering!together,!reviewing/scanning!the!collected!forms!and!
discussing!outstanding!comments!or!questions!at!the!end!of!each!session.!Over!the!two!days,!the!
end*of*session!review!of!the!forms!by!artists!continued!and!built!momentum.!The!creative!team!
delivering!the!forms!also!started!to!experiment!with!how!the!forms!were!used!in!order!to!maximise!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 140!
student!engagement!and!input.!Increased!time!and!attention!was!dedicated!to!the!use!of!the!
evaluation!instrument,!and!this!evolved!to!be!coupled!with!reflection!in!pairs!to!further!draw!out!
student!ideas.!The!evaluation!activity!went!from!being!a!potential!imposition!to!a!valued!tool!for!
engagement!and!provided!data!on!the!creative!content!of!the!showcase!ahead!of!the!public!season,!
which!led!the!artists!to!make!refinements.!This!is!a!prime!example!of!real*time!utility!resulting!from!
the!participatory!approach!that!involved!stakeholders!in!the!design!and!implementation!of!READ!
processes,!and!shows!the!interactivity!between!evaluation!practice!(through!READ),!creative!
practice!and!program!management.!!!
The!reflective!analysis!also!highlighted!several!lessons!about!approaching!evaluation!as!an!
integrated!practice!and!revealed!that!each!READ!strategy!showed!evidence!of!operating!well!as!
stand*alone!elements!(modules).!Analysing!the!four!READ!strategies!as!distinct!operational!
components!of!the!model!of!practice!tested!the!notion!that!the!READ!strategies!have!the!potential!
to!function!independently!(as!modular!elements!in!the!system).!Part!of!this!analysis!was!dedicated!
to!examining!how!well!the!READ!strategies!functioned!for!their!own!unique!purpose.!!
From!the!outcomes!described!in!Chapter!5,!it!was!determined!that!each!READ!strategy!did!well!in!
achieving!this.!Take,!for!example,!the!Reflection!strategy,!aiming!to!generate!a!culture!of!reflective!
practice!for!learning!and!improvement!outcomes.!The!devised!approach!for!this!strategy!included!
primary,!secondary!and!tertiary!processes!of!reflection!(explained!in!Figure!5.4),!and!produced!
developmental!outcomes!for!individuals!and!the!organisation,!such!as!professional!development,!
skills!development!and!organisational!learning.!Another!example!is!the!Analysis!strategy,!and!how!it!
achieved!the!objective!of!generating!original!knowledge!through!academic!enquiry,!which!has!been!
demonstrated!through!the!research!outputs!of!the!community!of!practice!so!far.!I!concluded!that!
each!READ!strategy!represented!an!established!and!viable!learning!strategy!that!could!be!used!
independently!or!combined.!This!finding!indicates!the!potential!flexibility!of!the!model!for!delivering!
a!spectrum!of!appraisal!and!learning!needs!arising!from!diverse!contexts!and!partnerships.!
The!learning!for!my!own!evaluation!practice!as!facilitator!was!the!insight!the!study!of!the!READ!
application!offered!into!the!many!ways!that!we!had!gone!about!learning!and!appraisal!of!the!
activities!and!arts!practice.!Many!of!these!evaluative!activities!were!already!inherent!in!community!
arts!processes,!yet!they!were!tacit.!This!process!enabled!the!practical!wisdom!arising!from!the!
community!arts!to!be!made!explicit!through!the!deliberate!focus!and!methods!of!the!READ!
methodology.!I!also!observed!that!the!acronym!of!‘READ’!in!itself!offered!value!as!a!collective!
compass!for!our!shared!evaluation!practice.!The!easy!to!remember!acronym!was!useful!for!quickly!
assessing!a!situation!in!terms!of!its!potential!for!data!collection!by!scanning!the!four!READ!focus!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 141!
areas!to!assess!relevance!and!opportunity!during!an!activity.!While!professional!evaluators!have!a!
level!of!coherency!about!how!to!undertake!evaluation!practice!(based!on!observation!from!previous!
experience),!this!is!not!the!case!for!many!creative!practitioners,!artists!and!technical!professionals!
who!may!benefit!from!having!a!conceptual!guide!such!as!the!READ!acronym!and!framework.!!
From!the!analysis,!I!determined!the!READ!approach!had!instrumental!value,!developmental!value!
and!conceptual!value!for!the!stakeholders.!The!following!summarises!the!emergent!findings!on!this!
theme!of!constructing!the!READ!model!to!have!both!modular!and!integrated!possibilities:!
•! Formal!evaluation!processes/methods!were!contextualised!and!strengthened!from!being!
presented!and!managed!as!part!of!the!broader!integrated!READ!framework.!
•! Information!from!different!program!areas,!art*making!processes!and!knowledge!bases!were!
connected!and!integrated!through!the!one!system,!and!also!linked!to!program!management.!
•! The!READ!approach!functioned!effectively!as!an!integrated!system!and!each!modular!
approach!employing!individual!strategies!was!effective!as!a!self*contained!subsystem.!
•! The!READ!strategies!draw!on!established!methods!that!are!not!new!to!evaluators!and!
creative!practitioners.!However,!the!innovation!of!the!system!lies!in!their!integration.!!
•! The!methodology!provides!a!means!to!make!explicit!some!of!the!often!under*valued!or!
disregarded!tacit!systems!of!knowledge!generation!inherent!in!community!arts!practice!and!
program!management,!providing!a!framework!inclusive!of!traditional!evaluation!
instruments!and!experimental,!arts*based!or!partner*specific!methods.!!
•! The!relevance!of!evaluation!instruments!and!traditional!methods!within!the!integrated!
framework!for!identifying,!deciphering!and!valuing!program!impact!was!affirmed!and!
offered!a!range!of!benefits!to!community!participants,!project!staff!(including!creative!
team),!volunteers,!program!partners!and!mentors.!
•! As!a!summative!lens,!READ!explored!the!efficacy!of!the!project!in!terms!of!meeting!its!stated!
objectives,!as!well!as!the!artistic,!theoretical,!academic!or!the!wider!social!implications!of!
community!arts!practice!(as!delivered!by!the!multi*strategy!approach).!!
•! There!were!benefits!from!the!extended!scope!of!evaluation!practice!via!READ.!
•! The!integrated!approach!delivered!a!wide*range!of!outputs!that!mixed!evaluative!and!non*
evaluative!products!that!were!relevant!for!gaining!an!understanding!of!community!arts!
practices.!As!a!collection,!these!were!beyond!what!would!have!been!delivered!with!an!
evaluation*only!approach.!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 142!
6.2.2! Social#constructivist#stance#
Identifying*and*understanding*conceptualisations*of*evaluation*
Given!this!was!the!first!industry!application!of!READ,!and!that!the!methodology!and!epistemology!
were!being!developed!along!the!way,!the!level!of!scoping,!preparation!and!design!consultation!
required!a!large!amount!of!concerted!time!and!resources.!As!is!outlined!in!Section!5.3.3,!the!results!
from!the!scoping!research!I!conducted!revealed!that!stakeholders!used!many!different!constructs!of!
evaluation,!and!this!meant!that!stakeholders!had!mixed!world!views!and!interpretations!of!
evaluation!and!its!value.!Also!identified!was!a!lack!of!clarity!for!some!stakeholders!about!their!
evaluation!needs!and!definition!of!success.!!
The!constructs!of!evaluation!presented!by!stakeholders!were!influenced!by!their!context,!past!
experiences!of!evaluation,!and/or!the!values!of!the!organisation!stakeholders!represented.!One!
interesting!theme!arising!from!the!data!was!trepidation!expressed!by!multiple!stakeholders!towards!
evaluation!of!community!arts.!One!creative!practitioner,!for!instance,!questioned!how!formal!
evaluation!could!contribute!meaningfully!towards!professional!learning!and!future!practice,!while!
another!rejected!outright!the!legitimacy!of!evaluation!practice!for!community!arts!altogether:!‘I!
don't!like!evaluations,!I!don't!like!those!standard!evaluations!and!I!particularly!don't!like!arts!funding!
body!evaluations’!(see!Section!5.3.3).!Analysing!the!emergent!theme!of!evaluation!trepidation!
featured!heavily!in!my!forensic!reflective!practice!during!the!scoping!phase!and!was!of!personal!
interest,!as!it!resonated!with!my!observations!and!experience!from!previous!management!and!
evaluation!work.!!
I!interpreted!the!evident!mistrust!and/or!evaluation!fatigue!of!some!stakeholders!as!a!potential!
barrier!to!the!effective!delivery!of!the!READ!application,!and!wanted!to!examine!this!finding!in!
detail.!I!recognised!that!these!views!would!also!have!a!bearing!on!partnership!brokerage!and!
engagement.!While!the!trepidation!theme!was!not!representative!of!all!stakeholder!responses,!I!
considered!it!important!for!me!to!develop!a!sophisticated!understanding!of!this!dimension,!in!order!
to!progress!the!READ!model!development!and!application.!
My!interpretation!of!the!data!from!interviews!and!consultation!was!that!for!some!stakeholders,!the!
pursuit!of!merit*based!and!formal!evaluation!were!perceived!as!unwanted!acts!of!institutionalisation!
or!instrumentalisation!of!their!creative!practice!to!fit!a!social!benefits/community!development!lens,!
or!that!the!process!was!a!waste!of!time/effort!where!no!apparent!end*utility!or!meaningfulness!
eventuated.!One!tension!identified!was!that!orthodox!and/or!formalised!evaluation!models!and!
instruments!were!perceived!to!undervalue!or!dismiss!the!relevance!of!the!reflective!and!evaluative!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 143!
processes!inherent!in!creative!arts!practice.!
The!evaluation!trepidation!sub*theme!also!resonated!with!my!own!professional!experience!and!
observation.!In!order!to!identify!and!assess!my!espoused!values,!forensic!reflective!practice!was!
employed!to!examine!my!conceptualisations!of!evaluation!and!my!possible!biases.!Reflective!analysis!
revealed!and!clarified!that!my!trepidation!of!evaluation!was!of!strictly!positivist!approaches,!and!I!
also!identified!some!scepticism!(based!on!my!observation!and!professional!experience)!that!too!
often!evaluation!outputs!(traditionally!word*heavy!reports)!are!under*utilised,!not!adequately!
disseminated!or!worst,!sit!unread!or!redundant!on!a!funding!agency’s!shelf.!I!also!reflected!on!
previous!discussions!with!industry!peers!and!colleagues,!which!identified!an!observed!tendency!
within!the!community!arts!context!for!practitioners!and!organisations!to!talk!up!the!impact!and!
outcomes!of!activities!(or!present!positive*lens!only!findings),!in!case!favourable!evaluation!and!
acquittal!reporting!relates!to!future!funding.!This!trend!within!the!field!reduces!the!meaning!that!
some!practitioners!place!on!evaluations!and!discourages!learning!about!the!failures!and!responses!
that!may!have!been!part!of!a!project.!This!means!that!a!representative!picture!of!creative!practice!
and!its!consequences!is!difficult!to!establish.!
As!covered!in!Chapter!2,!trepidation!of!community!arts!evaluation!was!evident!in!the!literature!and!
industry!forums!I!cited!for!the!contextual!scan!(Belfiore,!2014;!Jackson,!2009;!Jennings!&!Baldwin,!
2010;!Newman,!et!al.,!2003;!Donovan,!2012;!Selwood,!2002).!Most!striking!was!the!quote!included!in!
Chapter!2!by!Goldbard!(2008)!arguing!the!detrimental!effect!evaluation!and!the!instrumentalisation!
of!the!arts!has!had!on!the!field.!As!part!of!her!argument,!Goldbard!asserts!that!damage!is!done!when!
terminology!and!assumptions!are!borrowed!from!other!fields!and!misappropriated!for!use!in!
community!arts.!Such!an!argument!supports!Alphaville’s!initial!impulse!to!draw!on!the!methods!and!
philosophy!of!community!arts!(see!Goldbard,!2015,!2008).!
The!reflective!analysis!was!driven!by!questions!such!as:!how!can!we!make!READ!inclusive!and!how!
can!such!divergent!views!of!evaluation!co*exist,!or!be!represented,!explored!and!responded!to!
collectively!during!the!READ!application?!The!implication!of!acknowledging!the!diversity!of!values!
about!evaluation!was!that!across!the!READ!application,!there!were!many!open!and!frank!discussions!
amongst!participating!stakeholders!that!questioned!and!challenged!what!READ!should!deliver!and!
what!the!boundaries!of!the!evaluation!practice!could!and!should!be.!For!my!own!evaluation!practice,!
I!gained!skills!in!identifying!and!articulating!different!stakeholder!constructs!of!evaluation!(as!READ!
facilitator!and!manager)!and!negotiated!these!with!multiple!partners,!which!offered!deeper!insight!
into!the!role!that!partnership!brokerage!plays!in!facilitating!a!shared!dialogue!and!establishing!the!
terms!of!reference!for!evaluation!practice.!!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 144!
Negotiating*evaluation*constructs*
One!way!in!which!I!addressed!the!findings!of!the!scoping!phase!with!stakeholders!was!by!
negotiating!and!establishing!a!wide!frame!of!reference!for!our!internal!evaluation!practice,!during!
our!application!of!the!multi*strategy!READ!approach.!As!explained!above,!this!involved!open!
dialogue!with!all!partners.!Of!interest,!was!one!negotiation!process!with!the!Australia!Council!for!the!
Arts,!as!funding!body,!which!had!significant!bearing!on!the!READ!application!that!eventuated.!An!
issue!had!arisen!from!Alphaville’s!proposal!to!conduct!the!READ!application!as!an!internal!process,!
which!was!contrary!to!the!specification!for!independent!evaluation,!as!stated!within!the!Australia!
Council!funding!contract.!READ!also!included!activities!beyond!the!conventional!scope!of!evaluation.!
Thus,!the!prospective!READ!application!required!discussion!and!negotiation.!The!outcome!of!this!
process!was!important,!in!light!of!another!of!the!contract!specifications,!that!required!10!per!cent!of!
the!secured!funding!value!(a!significant!amount!of!resources)!to!be!dedicated!to!evaluation.!!
Despite!many!of!the!READ!practices!not!being!strictly!evaluative!in!outcome,!it!was!agreed!that!wide!
representations!and!accounts!of!the!program!would!provide!insight!into!the!wider!social!
implications!of!the!arts!practice!and!for!an!experimental!program!such!as!Nuclear!Futures!this!was!
desirable.!The!scoping!process!with!stakeholders!resulted!in!license!being!granted!to!myself!and!
Alphaville!to!experiment!with!a!broad!mix!of!learning!and!appraisal!strategies!(beyond!‘evaluation’).!!
While!the!READ!application!represented!a!departure!from!the!positivist!foundation!of!program!
evaluation,!it!was!by!no!means!a!rejection!of!evaluation!as!a!systematic!practice!for!learning!and!
appraisal.55!As!one!of!the!four!core!READ!strategies,!evaluation!commands!a!central!position!in!the!
READ!practice.!At!the!same!time!however,!READ’s!integration!responds!to!the!limitations!of!current!
practice!(see!problem!statement!in!Chapter!1)!and!reasons!that!resources!allocated!to!‘evaluation’!
in!the!community!arts!context!might!be!best!spent!across!a!spectrum!of!critical!review,!learning!and!
appraisal!strategies!(using!qualitative!and!quantitative!methods).!In!deviating!from!the!dominant!
empirical!epistemologies!of!evaluation!practice!towards!more!constructivist!approaches!–!where!no!
objective!or!unified!judgment!of!program!worth!may!result!from!the!‘evaluation’!process!–!advocacy!
and!negotiation!on!a!project*to*project!basis!is!needed!to!secure!the!agreement!of!stakeholders.!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
55!It!is!worth!reiterating!that!general!discussion!about!‘evaluation’!is!fraught!given!the!term!encompasses!such!a!broad!
range!of!definitions,!underpinned!by!divergent!value!paradigms,!and!that!there!are!myriad!approaches!to!how!it!is!
undertaken.!In!the!literature,!the!diversity!for!what!constitutes!evaluation!is!much!more!clearly!acknowledged!than!is!
evident!in!contemporary!practice!where!a!somewhat!limited!scope!for!practice!is!demonstrated,!conforming!to!the!
definition!of!evaluation!focused!on!research!for!the!purpose!of!making!judgments,!usually!about!the!effectiveness,!
efficiency!and/or!appropriateness!of!an!activity!(such!as!those!definitions!by!Australasian!Evaluation!Society,!2010;!Scriven,!
2012;!and!Davidson,!2005).!!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 145!
For!Nuclear!Futures,!the!conditions!were!favourable!for!such!negotiation,!and!renegotiation,!of!how!
widely!the!evaluation!resources!could!legitimately!be!allocated.!I!attribute!this!outcome!to!the!
experimental!context!of!the!Creative!Communities!Partnership!Initiative!(CCPI)!funding!scheme!
(which!was!established!to!encourage!recipient!organisations!to!be!reflexive!as!a!means!of!promoting!
innovation)!and!the!fact!that!the!partners!acknowledged!the!differing!knowledge!paradigms,!
languages,!ideologies!and!cultural!systems!at!the!table!(which!included!Aboriginal!and!non*
Aboriginal!organisations!and!community!partners,!academic!institutions,!international!communities,!
government,!non*for*profit!and!community!organisations).!
Social*construction*
The!READ!approach!for!Nuclear!Futures!also!challenged!the!‘independent’!requisite!of!the!Australia!
Council.!As!explained!in!Chapter!5,!it!was!determined!that!a!constructivist!stance!for!the!READ!
application!in!Balaklava,!and!for!Nuclear!Futures!generally,!would!be!favourable!for!accommodating!
the!differing!partner!values!identified.!The!interpretative!constructivist!stance!recognises!that!
individuals!and!groups!construct!their!perceived!reality!(Glaser!&!Strauss,!2009;!Charmaz,!2000;!
Cresswell,!1994),!and!was!seen!as!a!credible!stance!by!the!stakeholders!because!it!had!the!capacity!
to!deal!with!a!register!of!voices!and!knowledge!systems.!The!implication!was!that!the!evaluation!
practice!was!not!conducted!to!conform!to!notions!of!seeking!objective!summative!judgments.!
The!data!reviewed!for!the!analysis!evidenced!that!the!constructivist!approach!for!the!READ!
application!proved!effective!for!accommodating!and!being!inclusive!of!the!divergent!world!views!
and!evaluation!needs!arising!from!the!community!partnerships.!Stakeholder!input!from!informal!
feedback,!interviews!and!reflective!discussions!on!the!evaluation!processes!and!outcomes!(and!the!
overall!experience!of!READ)!indicated!consistently!that!READ!deliverables!in!the!domains!of!
appraisal!and!engagement!exceeded!expectations!and!requirements.56!!
While!the!constructivist!approach!was!effectively!applied!in!this!context,!it!is!conceivable!that!future!
READ!applications!could!have!success!using!alternative!interpretative!stances!(if!deemed!
appropriate!for!the!context!and!partners)!because!of!the!methodological!design!flexibility!READ!
offers.!For!example,!in!any!of!the!READ!strategies,!a!different!interpretative!stance!would!lead!to!a!
different!methodological!orientation!and!method!selection,!as!long!as!the!chosen!stance!
accommodated!the!needs!and!values!at!the!site!of!practice.!This!claim!is!speculative!and!is!a!
potential!area!for!future!research.!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
56!Data!from!observation!and!communications!corroborated!this!finding.!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 146!
6.2.3! Internal#facilitation#
Of!paramount!significance!for!the!outcomes!of!the!READ!application!for!the!Balaklava!project!was!
the!internal!position!of!the!READ!delivery,!where!design!and!implementation!of!the!tailored!READ!
framework!was!undertaken!by!stakeholders!situated!within!the!project.!Likewise,!the!leadership!
governing!the!READ!application!(my!role!as!facilitator)!was!embedded!within!the!project!and!arts!
practice,!and!closely!linked!with!my!role!as!Alphaville’s!program!manager!for!Nuclear!Futures.!
For!the!Balaklava!project,!it!was!evidenced!that!the!internal!position!of!the!READ!facilitation!was!
effective!and!generated!particular!benefits!as!a!result!of!this!positioning.!In!my!role!as!facilitator,!my!
existing!knowledge!of!the!politics,!organisational!culture,!project!structure,!personnel,!partners!and!
community!participants!(including!my!experience!of!working!with!some!community!partners!
previously)!gained!from!my!manager!role!created!immediate!and!sustained!advantages!for!
progressing!the!READ!delivery.!This!was!also!true!for!other!participating!stakeholders!to!varying!
degrees,!who!were!able!to!bring!their!local!knowledge!or!familiarity!with!the!project!into!the!
appropriate!tailoring!of!the!design!and!implementation!of!READ!for!the!Balaklava!project.!
Beyond!the!general!benefits!attributed!to!the!internal!position!of!the!READ!delivery,!the!analysis!
showed!that!positive!outcomes!were!enhanced!by!my!dual!role!as!READ!facilitator/program!
manager.!For!example,!relationship!building!and!establishing!trust!with!participants!was!already!at!
an!advanced!level!from!the!commencement!of!the!READ!application,!and!in!this!dual!role!I!had!
ready!access!to!project!resources!and!communication!channels!with!participants.!The!dual!
facilitator/manager!role!also!meant!I!was!able!to!feed!raw!and!processed!READ!data!into!the!
decision*making!cycles!of!the!creative!process,!program!management!and!strategically!link!delivery,!
activities,!resource!and!outcomes!between!the!arts!practice!and!READ.!One!outcome!was!increased!
reflexivity!of!program!planning!and!management,!and!real*time!impact!of!evaluation!data!collected.!
A!specific!demonstration!of!how!my!capacity!as!READ!facilitator!was!aided!by!the!enriched!
understanding!of!the!political!and!social!dimensions!relevant!for!the!evaluation!conduct!–!a!direct!
result!of!the!internal!position!–!was!in!regard!to!known!sensitivities!and!partnership!brokerage!
needs!for!this!specific!project!and!community!of!interest.!For!the!Balaklava!project,!there!were!
unique!personal!and!political!sensitivities!related!to!the!subject!matter!of!atomic!testing!for!nuclear*
affected!communities!and!general!audiences!being!exposed!to!this!content!for!the!first!time.!The!
testimonies!and!information!at!the!centre!of!the!community!arts!for!this!project!can!be!distressing!
and!uncomfortable!for!participants,!artists!and!audiences.!For!community!participants!especially,!
having!familiar!insiders!(myself!and!the!Nuclear!Futures!team!delivering!READ)!maintained!a!level!of!
continuity!and!confidence!across!the!art*making,!management!and!review!processes.!Inviting!and!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 147!
collecting!READ!input!also!became!a!normalised!and!integrated!process!across!project!operations!
(and!evolving!sub*culture),!in!simple,!efficient!and!opportunistic!ways!with!our!community!partners.!!
Figure!6.1!is!a!diagram!sourced!from!my!reflective!practice!documentation!(originally!hand!drawn)!
where!I!had!graphically!explored!the!implications!of!the!internal!position!in!relationship!to!the!
manager!role.!The!top!diagram!in!Figure!6.1!represents!the!scenario!when!the!two!roles!are!
distinct/separate,!and!the!reflective!extract!explores!the!implications!of!this!approach.!The!lower!
diagram!represents!the!dual!position!of!evaluator/manager!(where!evaluator!terminology!is!
equivalent!to!READ!facilitator)!and!discusses!the!observed!efficiencies!achieved,!while!also!warning!
that!validity!checks!are!needed.!The!reflection!was!based!on!observation!from!the!Balaklava!project!
as!well!as!on!my!previous!experience!of!both!scenarios.!It!is!included!as!example!of!data!collected!in!
my!primary!reflective!sessions!during!the!forensic!reflective!practice.!
!
!
!
Figure!6.1:!Extracts!from!reflective!practice:!exploring!the!internal!positioning!of!READ!facilitation!
!
Original notes: a significant relationship exists between the two roles of manager and lead evaluator if conducted by different individuals, each is important to the other. Open communication and shared information systems are needed to make optimal. Program manager is an important champion for the evaluator, the eyes of the project. Evaluator has the tools and expertise for capture and assessment, and is a program management resource for reviewing operations, via an evaluative- analytical lens. Timeliness is important for evaluator to feed in information, as evaluation data impacting creative practice or program management may need real-time action. ‘Objectiveness’ vs opportunity for use?
**Tim
e*access*communicatio
n**!
**validity
!checks!im
portant**!
Original notes: utility and relevance of both processes may be maximised by manager as internal evaluator, if issue of conflicts of interest, bias and validity is addressed. There are so many times, ongoing, that I am making management decisions based on READ data. Time was saved in terms of setting up the lines of communication and accounting between roles in preparation phase. Promotes data collection via program documentation and incidental feedback from informal sources arising from management communications. If validity can be managed, the exchange between roles is rich and rewarding.
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 148!
In!the!READ!application,!one!of!the!overriding!benefits!of!the!internal!position!and!dual!role!was!the!
real*time!access!created!and!resultant!utility!of!the!information!being!generated!throughout!the!
READ!and!project!cycles.!Internal!management!also!meant!I!was!able!to!respond!to!gaps!in!data!
collection!and!instigate!opportunities!to!collect!further!data!where!necessary.!It!is!important!to!
acknowledge!however!that!there!are!limitations!and!compromises!for!the!internal!position!of!the!
evaluation!practice,!in!regard!to!implications!for!validity!and!credibility!(I!address!these!in!Section!
6.2.5).!More!broadly,!the!benefits!of!the!internal!approach!may!have!been!undermined!if!tensions!or!
conflicts!had!arisen!between!partners,!which!was!not!the!case!for!the!Balaklava!partners,!or!if!the!
facilitation!of!the!READ!was!conducted!via!unethical!means,!or!if!bias!privileged!some!partners!over!
others!in!terms!of!participation,!inclusion!of!input,!representation!and!access!to!information.!!
For!the!Balaklava!experience!and!READ!application,!I!established!that!positive!outcomes!resulted!
from!the!internal!facilitation,!and!these!were!well!supported!from!the!data!collected.!Overall,!the!
assessment!led!me!to!conclude!that!a!number!of!the!utility,!engagement!and!empowerment!
outcomes!were!evident!for!the!Balaklava!project!as!a!result!of!the!internal!position!and!dual!role.!
While!context*specific,!I!interpret!this!to!be!a!relevant!finding!for!the!wider!community!arts!context,!
as!the!example!shows!that!creative!practitioners!and!community!partners!at!the!heart!of!the!arts!
were!practice!are!well*positioned!to!undertake!the!evaluative!practice!(assuming!an!adequate!level!
of!evaluation!experience!exists!or!can!be!accessed),!and!their!proximity!for!collecting!data,!and!the!
relevance!of!their!collective!insight!for!making!sense!of!the!data,!has!the!potential!to!enhance!the!
overall!outcomes!of!evaluation!practice.!The!finding!also!raises!a!question!about!the!value!and!
appropriateness!of!the!routine!prescription!by!funding!bodies!for!independent!evaluation!for!the!
community!arts!context.57!Assuming!that!issues!such!as!validity!and!credibility!can!be!adequately!
managed!and!ensured!through!methodological!means,!the!internal!position!of!evaluation!offers!
potential!partnership*oriented!benefits!for!this!context.!
! !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
57!While!independent!evaluation!does!not!exclude!internal!facilitation,!it!does!mean!that!significant!resources!need!to!be!
invested!in!verification!of!the!internal!process!by!an!external!specialist!(at!the!very!least).!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 149!
6.2.4! Emergent#evaluation#practice#
Theoretical*orientation*
The!outcome!of!the!internally!driven!and!constructivist!approach!were!defining!theoretical!
underpinnings!of!the!evaluation!practice!that!developed!for!the!Balaklava!project.!To!gain!insight!
into!the!emergent!evaluation!approach!for!the!Balaklava!application,!and!to!analyse!the!theoretical!
underpinnings!of!the!evaluation!practice,!I!examined!the!approach!in!relation!to!contemporary!
theories!within!the!evaluation!field.!Via!a!structured!reflective!exercise,!I!examined!where!the!READ!
application!for!Balaklava!would!be!positioned!within!the!field.!The!findings!from!this!exercise!focus!
only!on!the!approach!of!the!Balaklava!tailored!methodology,!not!for!the!general!READ!model.!!
First,!I!established!that!along!the!spectrum!of!participatory*oriented!approaches!described!in!
Chapter!2,!the!evaluation!approach!for!the!Balaklava!project!reflected!a!stakeholder*oriented!and!
utilitarian!approach,!rather!than!an!empowerment!example!(although!capacity!building!and!
empowerment!outcomes!did!result).!This!was!based!on!the!premise!that!while!the!activities!were!
participatory!and!interested!stakeholders!did!participate,!overall!the!hands*on!learning!and!capacity!
building!dimensions!of!evaluation!practice!were!embraced!to!varying!levels!by!stakeholders!(less!
than!what!I!had!initially!envisaged).!In!many!cases,!input!was!facilitated!through!consultation!rather!
than!active/direct!participation,!which!meant!the!process!remained!centralised!through!Alphaville,!
and!me!as!facilitator/lead!evaluator.!What!was!demonstrated!by!the!approach!I!adopted!was!a!
prioritisation!of!stakeholder!relationships,!brokerage!and!shared!learning.!This!dimension!of!the!
emergent!evaluation!practice!aligned!with!some!of!feminist!evaluation!theory,!such!as!by!Belenky,!
Clinchy,!Goldberger,!and!Tarule!(1986),!in!Women’s%Ways%of%Knowing,!where!meaningful!
connections!with!program!stakeholders!are!developed!in!order!to!better!understand!the!reality!of!
their!program!experiences,!as!well!as!the!impacts!of!evaluation!on!stakeholders,!delivery!partners!
and!community!participants!(Sielbeck*Bowen,!2002).!
Of!importance!for!defining!the!philosophy!of!practice!of!the!READ!approach!for!the!Balaklava!project!
was!the!constructivist!stance!outlined!in!Section!6.2.2.!In!attempting!to!understand!how!the!
application!related!to!the!field!of!evaluation,!I!referred!to!the!evaluation!tree!by!Christie!and!Alkin,!a!
diagrammatic!schema!that!maps!the!‘roots’!and!‘branches’!of!evaluation!theory!(Christie!&!Alkin,!
2008).!The!evaluation!tree!proposed!by!Christie!and!Alkin!is!included!as!Figure!6.2.!!
!
!
!
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 150!
!
!
Figure!6.2:!Evaluation!theory!tree!by!Christie!and!Alkin!(version!3)!!
(Source:!Christie!&!Alkin,!2008,!p.133)!
!
In!the!illustration,!Christie!and!Alkin!have!plotted!seminal!evaluation!theorists!along!three!respective!
branches!according!to!the!primary!emphasis!of!the!models!included.!While!all!evaluation!theorists!
are!interested!in!the!process!of!valuing,!the!methods!used!and!their!utility,!approaches!do!differ!in!
terms!of!the!particular!emphasis!placed!on!one!or!another!of!these!dimensions.!The!diagram!also!
defines!the!two!main!theoretical!roots!of!contemporary!evaluation.!The!first!is!social!accountability!
and!fiscal!control,!and!the!second!is!social!inquiry.!
My!analysis!during!this!reflective!practice!exercise!revealed!that!the!emergent!READ!approach!for!
Balaklava!represented!a!theoretical!orientation!from!the!values!branch!of!Christie!and!Alkin's!(2008)!
theory!tree,!aligned!with!theorists!such!as!Jennifer!Greene,!who!advocates!a!value*engaged!
approach!for!framing!evaluation,!with!an!emphasis!on!stakeholder!values!(Greene,!2005).!While!the!
values!branch!originated!in!response!to!theorists!such!as!Scriven,!who!argue!that!the!process!of!
attributing!value!is!an!essential!component!of!an!evaluator’s!work,!this!branch!also!includes!
constructivist!approaches.!Christie!and!Alkin!(2008,!p.134)!states:!‘this!branch!also!includes!the!work!
of!those!interested!in!social!justice!in!evaluation,!as!well!as!those!who!espouse!the!philosophy!of!
subjectivity,!that!is,!the!claim!that!there!is!no!one!objective!reality.!Here,!theorists!are!also!
concerned!with!valuing,!but!the!emphasis!is!on!the!extent!to!which!values!shape!the!evaluation’.!!
Use branch revisions
In its most recent incarnation, the Use branch underwent onlyminor revision and still closely resembles the Tree presented inChapter 26 of the Evaluation Roots book. The slight modificationsinclude repositioning Joseph Wholey’s work (2004) to reflect anemphasis towards the Methods branch. Also, we deleted MalcolmProvus from the branch because much of his work has hadrelatively little influence on other perspectives on the branch.
Methods branch revisions
The Methods branch received more substantial changes. RalphTyler was repositioned to a sub-branch to reflect that while histheoretic point of view was in fact heavily methods related(objectives based evaluation), he was not a theoretical predecessorof those further up on the branch. His original positioning wasintended to reflect his influence on the field of educationalevaluation (which was very significant), but upon furtherreflection we concluded that his overall influence on the methodsbranch specifically was less than his original position implied.Thus, we placed him on a small sub-branch near the base of thebranch.
Donald Campbell continues to be the heart and soul of thisbranch both in the way that he directly influenced other theoristswith his work on experiments, quasi-experiments, and validity, aswell as the way in which his views provided a basis for counterpoint, particularly with respect to Lee Cronbach’s views.
We have also deleted Edward Suchman from the branch. Werecognize the important influence of Suchman’s writing onevaluation—particularly in positioning Campbell’s work promi-nently in the evaluation discourse. Nonetheless, despite hishistorical significance, it did not seem appropriate to continueto include him on the branch because he himself did not offer thefield a specific evaluation model. Rather, he promoted Campbell’swork as the most effective approach for conducting evaluationstudies to measure program impact. Peter Rossi continues to be
shown on the main portion of the branch as an influence on bothCarol Weiss and, more particularly, on Huey Chen.
A significant change on this branch was moving Tom Cook whopreviously had been positioned in a sub-branch flowing fromCampbell. He is now positioned on the main branch, and his sub-branch was deleted. A new sub-branch was created for Bob Boruch.Boruch’s sub-branch comes directly out of Campbell and Cook. Thisplacement reflects the steadfast influence of Campbell’s rando-mized control trial (RCT) work on Boruch. The branch is offset on asub-branch to emphasize his departure from both Cook andCampbell in his lack of regard for quasi-experiments. This positionalso offsets Boruch from the others on the methods branch, all ofwhom have embraced quasi-experiments as an acceptable methodfor studying causality, although everyone on the branch wouldagree that the experiment is ideal if context and conditions permit.
Another major change on this branch is the addition of GaryHenry, Mel Mark and George Julnes. In the Roots book we said:
(T)heories included were able to be classified onto a singlebranch of the tree. . . Whether some theories were not included dueto their comprehensiveness or our conceptual inability is unclear.A particular example comes to mind: the work of Mark, Henry, andJulnes (2000). These authors view social betterment as theultimate objective of evaluation and present a point of viewgrounded in what they refer to as a ‘common sense realistphilosophy.’. . . The very diversified nature of this perspective,while a great strength in presenting an understanding ofevaluation, precludes its inclusion on the tree (pp. 58–59).
We admit that in this instance our conceptual analysis wasincomplete. Our views in determining exclusion were basedheavily on the Mark et al. (2000) book. However, in reflectingfurther on the writings of these authors, we were further struck bythe views presented in the ‘‘realist evaluation’’ monograph in NewDirections for Evaluation (Henry, Julnes, & Mark, 1998). Based upona more in-depth reading of this text, we have placed Henry, Mark,and Julnes on the methods section branch, leaning toward thepost-positivist valuing branch. The leaning highlights Henry andMark’s collaboration with George Julnes to create ‘‘a new theory
Fig. 2.
C.A. Christie, M.C. Alkin / Studies in Educational Evaluation 34 (2008) 131–135 133
AN!INTEGRATED!APPROACH!TO!EVALUATION! ELLISE!BARKLEY!
! 151!
Specifically,!it!can!be!argued!that!the!Balaklava*tailored!approach!sits!in!the!valuing!branch!as!a!
post*positivist!and!constructivist!paradigm!framework,!towards!the!right!fork!where!the!valuing!
theorists!are!located.!While!the!findings!from!reflective!practice!also!identified!the!READ!application!
had!a!secondary!relationship!to!the!‘use’!branch!of!the!tree,!in!that!the!application!was!concerned!
with!utility,!legacy!and!the!end*use!purposing!of!evaluation!practice!for!the!benefit!of!stakeholders,!
overall,!it!was!the!social!enquiry!orientation!of!the!values!branch!that!best!matched!the!practice.!
The*model*of*practice*arising*from*the*READ*methodology*
An!alternative!review!technique!I!used!to!understand!and!analyse!the!evaluation!practice!was!to!
consider!the!different!domains!of!activity!resulting!from!the!READ!methodology!in!a!generalised!
way.!This!section!presents!the!simplified!summary!table!I!created!to!capture!the!transferable!
elements!of!the!model!of!practice!(see!Table!6.1).!While!it!is!based!on!the!activities!and!observations!
of!the!Balaklava!trial!(Chapter!5),!I!attempted!to!step!back,!and!create!a!snapshot!of!the!sources!and!
flow!of!information,!and!link!these!to!the!outcomes!and!relationships!across!the!four!READ!
strategies.!The!purpose!of!generating!the!summary!table!was!to!present!a!consolidated!record!of!
the!READ!model!and!evaluation!practice!arising!from!the!Balaklava!trial,!that!could!be!easily!
communicated!back!to!partners.!Combined!with!the!analysis!of!the!theoretical!orientation!of!the!
READ!application!in!Balaklava!(discussed!above),!the!tabular!summary!provides!insight!into!the!
emergent!evaluation!model!that!evolved!from!the!original!READ!concept.!It!was!also!a!formative!
process!for!the!READ!product!development!presented!later!in!the!Chapter!in!Section!6.3.!
(Please!note:!in!Table!6.1,!I!have!used!categorisations!for!data!collection,!analysis!techniques,!and!
outputs.!However,!in!practice,!several!of!these!processes!and/or!methods!overlapped,!and!the!
system!as!a!whole!was!iterative!and!not!sequential!as!is!implied!by!the!table.)
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 152#
#Table#6.1:#Summary#of#READ#model#processes#and#outputs#
STRATEGIES APPROACH DATA SOURCES ANALYSIS & INTEGRATION
OUTPUTS
REFLECTION Individual & collective reflective practice makes tacit knowledge explicit
•! Voluntary, inclusive, safe •! Culture of reflective practice •! Individual and collective,
regular and reflexive •! Self-guided, semi-structured
or facilitated •! Suit methods to participants
Journals Video diaries Interviews Oral history Digital stories Field notes Documentation
Blogs, websites Communications Critical buddies Observation News media Social media Art methods
Lead by partner needs, desired outputs and level of participation Mix of evaluative, reflective, creative and critical analysis processes
Examples include: Thematic analysis Textual analysis Comparative analysis Peer review Reflective analysis Evaluative reasoning Rubrics/performance criteria Statistical/quantitative analysis Creative methods Practice-led methods Story-based methods (Most Significant Change technique)
Formative and summative outputs generated across READ implementation
Mix of evaluative and non-evaluative content
Examples include:
Raw data Management resources Evaluation reports Publications Research Creative content Infographics Artworks Promotional assets Archives Project stories Pending/future outputs (outputs not yet completed or known) (Access and shared: copyright may vary across partners)
•! Participatory and stakeholder engagement approaches
•! Formative and summative •! Unique project goals •! Skills development •! Key evaluation questions •! Access to all READ data
Scoping Questionnaires Feedback forms Interviews Vox pops Reflections Statistics
Documentation Research Field notes Observation Communications Media Focus groups
ANALYSIS Scholarship & research enquiry to generate original knowledge for broader learning
•! Network of engaged researchers and partners: Community of practice
•! Priority on publication and dissemination
•! Practice-led research •! Promotes broader learning
Scholarship Forums Seminars Media Research Critical reviews
Practice-led methods Commentary Presentations Contextual scans Publications
DOCUMENTATION Collection of written, visual and audio records, artefacts, & stories
•! Multi-platform •! Mixed sources •! Links with creative practice •! Legacy and archival elements •! Digital content archiving for
long-term storage •! Rich ephemeral data
Photography Film/video Publications Artworks Reports Websites Social media Webisodes
•! Databases/ repositories Sound recordings Communications Archival artefacts Writing Digital stories
Real-time and short-term outputs to support project decision-making, improvement (creative practice and organisational), accountability, partner updates/communications
Medium to long-term outputs to support accountability, continual improvement, broader learning, legacy
EVALUATION Key evaluation questions addressed using evaluative instruments and reasoning
#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 153#
6.2.5! Meta)evaluation0
While#findings#from#the#reflective#practice#using#the#metaKevaluation#and#SWOT#tools#(see#Chapter#
3)#were#applied#across#many#of#the#themes#presented#in#this#chapter,#this#section#provides#a#focused#
metaKevaluation#discussion#aligned#to#Davidson’s#criteria#(2005)#of#validity,#utility,#conduct,#
credibility#and#cost.#Informed#by#the#findings,#I#was#able#to#identify#and#assess#the#constraints#and#
opportunities#of#the#evaluation#practice,#for#continuous#improvement#for#future#applications.#
Validity(of(READ(findings((
The#internal#position#of#the#READ#delivery#and#my#dual#role#of#READ#facilitator/manager#were#
identified#as#potential#conflicts#of#interest#that#could#have#implications#for#the#validity#of#the#READ#
findings.#It#could#be#argued#that#READ#results#may#be#subject#to#my#personal#and#professional#bias#
given#that#‘the#project#manager#is#seen#as#ultimately#responsible#for#the#project’s#success#in#
achieving#objectives’#(Davidson#and#Rowe,#2009,#p.573).#To#ensure#validity#of#the#findings#measures#
were#incorporated#into#the#READ#methodology#so#that#evaluation#practice#would#be#rigorous#and#
credible,#such#as#using#triangulation#techniques#to#corroborate#data#and#conclusions.#
During#the#design#and#implementation#of#the#tailored#READ#framework,#quality#and#rigour#were#
addressed#by#ensuring#the#authenticity#of#the#data#collected#and#the#trustworthiness#of#the#data#
analysis#processes#used.#Within#each#of#the#READ#strategies,#established#research#and#evaluation#
methods#were#selected#based#on#their#appropriateness#and#reliability.#To#ensure#adequate#expertise#
existed#amongst#the#collective,#participating#stakeholders#received#skills#training#where#necessary#by#
a#member#of#the#team#with#adequate#expertise#in#that#method.#For#example,#artists#involved#in#
documentation#were#responsible#for#conducting#the#semiKguided#reflections,#vox#pop#recordings#and#
interviews#and#this#relied#on#them#employing#question#design#and#delivery#techniques#that#would#
ensure#reliable#and#authentic#responses#from#participants.#Before#documentation#of#key#events,#the#
documentation#approach#and#questions#were#critically#reviewed#with#another#team#member#and#
refined.#For#several#artists,#there#was#already#a#high#level#of#expertise#in#documentation#and#
interview#methods,#given#their#experience#in#investigative#journalism#and#documentary#artKmaking.#
This#example#also#demonstrated#the#way#in#which#the#READ#methodology#was#designed#to#draw#on#
the#existing#strengths#of#the#participating#stakeholders,#while#delivering#appropriate#methods#for#the#
data#collection#needs#of#the#project.#
Given#that#the#READ#application#largely#relied#on#interpretative#processes,#collective#review#of#data#
during#analysis#with#other#participants#was#undertaken#so#conclusions#were#not#solely#my#own.#
Different#analysis#techniques#were#experimented#with,#including#evaluative#reasoning#based#on#a#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 154#
performance#rubric#to#guide#criteria#for#claims#and#findings#on#the#artistic#merit#of#the#Balaklava#
creative#works#(see#Chapter#5,#Section#5.4.3).#In#complement#to#the#collective#reflective#practice#
being#undertaken,#one#measure#I#implemented#to#regulate#my#own#evaluation#practice#was#to#utilise#
the#forensic#reflective#practice#as#an#evaluator’s#tool,#and#this#involved#video#reflections#and#keeping#
a#log#of#practice#where#reflection#within#and#on#practice#was#used#to#explore#issues#arising,#identify#
conflicts#of#interests#and#examine#assumptions#during#the#READ#application.#I#also#coordinated#the#
triangulation#of#data#involving#multiple#participant#reviews,#to#verify#the#validity#of#data#collected#
from#different#sources.##
For#the#scale#of#the#Balaklava#project,#no#external#review#of#the#READ#process#was#deemed#
necessary.#This#was#a#combination#of#the#scale#of#the#project#and#the#identified#evaluation#outcomes#
needed#by#partners#for#the#Balaklava#project.#However,#for#the#summative#program#evaluation,#the#
strategy#to#engage#an#evaluation#specialist#external#to#the#project#to#review#the#methodology#and#
findings#is#being#considered#as#a#means#to#overcome#the#vulnerability#that#the#internal#positioning#of#
the#READ#delivery#places#on#validity.#On#reflection,#the#evaluative#conclusions#were#relatively#
straightforward#to#establish#for#the#Balaklava#project,#as#the#multiple#perspectives#represented#in#
the#data#collected#led#to#convergent#conclusions.#A#different#level#of#rigour#and#process#to#ensure#
validity#of#findings#would#have#been#needed#if#divergent#conclusions#had#arisen,#or#if#data#required#
synthesis#from#a#more#complex#range#of#methods#covering#diverse#program#areas#and#goals#(such#as#
will#be#the#case#in#the#summative#evaluation#needs#of#the#Nuclear#Futures#program#in#2017).#
For#example,#one#area#for#improvement#I#identified#from#the#metaKevaluation#was#the#further#
development#of#techniques#for#data#analysis,#to#make#the#collective#processes#of#review#more#robust#
in#future#during#the#Nuclear#Futures#summative#evaluation.#For#the#endKofKprogram#evaluation,#the#
level#of#sophistication#for#these#processes#will#need#to#increase#compared#with#what#was#
implemented#at#the#Balaklava#project#level.#The#summative#evaluation#may#also#need#to#revisit#the#
findings#established#at#the#end#of#individual#projects,#to#take#into#account#any#later#successes#or#
impacts#generated#from#the#community#arts#(such#as#subsequent#awards,#reviews#and#exhibitions).#I#
anticipate#that#reKevaluating#the#arts#practice#across#a#longer#time#period#will#be#important#for#
contributing#to#a#richer#assessment#of#the#merit#and#vibrancy#attributed#to#artwork#generated#in#
Balaklava#beyond#time#constraints#of#this#research.##
In#approaching#future#applications#of#the#READ#model,#there#are#a#number#of#‘next#steps’#that#I#
identified#in#the#reflective#practice#that#will#be#of#relevance.#For#example,#there#may#need#to#be#an#
increased#use#of#rubrics#and#establishment#of#thresholds#for#practice#and#outcomes#to#identify#
minimum#standards.#Indices#may#also#need#to#be#developed#that#speak#to#impact#categories#across#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 155#
the#project#domains#within#the#program#evaluation,#to#enable#cohesive#and#summative#collective#
analysis.#This#may#also#involve#investigating#and#assessing#the#applicability#of#the#latest#international#
standards#or#cultural#indicators#that#may#be#developed#in#the#future,#and#examining#the#role#of#
technology#in#future#applications,#as#well#as#the#latest#techniques#for#presenting#impact#case#studies.#
It#is#worth#noting#that#analysis#of#the#validity#of#READ#findings#was#not#straightforward,#and#required#
different#credentials#be#used#given#the#mix#of#learning#and#appraisal#methods#employed#across#the#
four#READ#strategies.#While#it#is#beyond#the#scope#of#the#research#to#cover#this#point#in#detail,#it#is#
acknowledged#that#the#READ#methodology#was#a#combination#of#evaluative#(the#process#of#valuing),#
researchKoriented#(the#process#of#enquiry)#and#creative#(the#process#of#artKmaking)#methods,#and#
there#is#a#difference#between#what#will#count#as#valid#for#each#of#these#methods.#For#example,#the#
parts#of#the#READ#methodology#dedicated#to#determining#the#quality,#value#or#significance#of#
community#arts#practice#demand#quite#specific#methodologies#to#ensure#validity#(that#extend#
beyond#social#enquiry#methods#to#include#evaluative#reasoning),#as#per#definitions#of#evaluation#by#
Davidson#(2005)#and#Scriven#(2011).#In#the#READ#application,#drawing#valid#evaluative#conclusions#
relied#on:#ensuring#credible#data#was#collected;#inclusion#of#the#multiple#perspectives#of#partners;#
the#utilisation#of#multiple#data#sources;#and#applying#criteria#for#merit#that#reflected#partner#values.##
Utility(
As#detailed#in#Chapter#5,#the#READ#methodology#generated#written#and#video#reflections,#creative#
documentation,#raw#content,#reports,#research#outputs,#artworks,#management#resources,#archives#
and#evaluation#findings#(see#Table#6.1).#These#READ#outputs#(both#formative#and#summative)#
delivered#shortK#and#longKterm#value#to#partners,#and#led#to#improvements#in#arts#practice,#program#
management#and#partnership#brokerage,#plus#broader#dissemination#of#original#knowledge#arising#
from#practice#via#academic#and#industry#forums.#The#outputs#have#already#been#used#in#diverse#
ways,#beyond#the#project#scope,#such#as#in#the#case#of#several#original#documentary#videos#and#
artworks#that#evolved#from#the#data#collection,#which#have#since#been#distributed#to#wider#
audiences.#
While#some#theorists#such#as#Scriven#(1991)#argue#the#limitations#of#using#endKutility#as#a#measure#
for#metaKevaluation#(because#the#actual#use#of#findings#or#outputs#is#beyond#the#control#of#the#
evaluation#team),#the#analysis#of#utility#outcomes#offered#valuable#insight#into#where#the#READ#
deliverables#were#shared#(in#terms#of#which#outputs#were#shared#with#whom#and#what#organisation,#
and#why,#and#how#outputs/findings#were#used,#if#at#all).#The#findings#of#this#analysis#evidenced#that#
the#mix#of#learning#and#evaluation#outputs#delivered#by#the#tailored#framework#were#sufficient#for#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 156#
meeting#the#identified#needs#of#the#stakeholders,#and#exceeded#expectations#and#requirements.#I#
attribute#the#effectiveness#of#the#READ#approach#in#this#domain#to#be#the#result#of#the#tailored#
methodology#designed#and#also#the#open#and#ongoing#communication#with#stakeholders#about#their#
endKuse#preferences#throughout#the#duration#of#the#project.#SielbeckKBowen#et#al.#(2002)#argue#that#
the#construction#and#generation#of#evaluation#outputs#is#an#interactive#process#that#occurs#within#
relationships,#and#from#my#perspective#as#READ#facilitator,#this#rang#true#for#the#partnerKoriented#
community#arts#initiative#in#Balaklava.##
One#of#the#surprising#results#arising#was#the#diverse#and#longKterm#utility#value#that#was#delivered#
through#creative#documentation.#The#creative#documentation#outputs#included#raw#data#and#
creative#content,#photographs,#video,#produced#assets#(such#as#online#material,#promotional#video,#
photoKstory#book)#and#creative#works.#Using#multiKart#methods,#the#creative#documentation#was#
effectively#used#for#engagement,#data#collection,#analysis,#communications#of#findings,#promotions,#
artKmaking#and#for#spinKoff#purposes#(beyond#the#Balaklava#project).#This#is#discussed#further#in#
Section#6.2.6.#
At#the#organisational#level,#one#outstanding#example#was#the#demonstrated#utility#of#the#READ#
generated#findings#by#Alphaville#and#the#Nuclear#Futures#team,#and#this#use#of#outputs#significantly#
impacted#creative#practice.#READ#data,#findings#and#outputs#played#a#seminal#role#in#decisionK
making#and#creative#development#for#future#showcases#in#a#number#of#ways.#One#outcome#specific#
to#the#work#generated#during#the#Balaklava#project#was#that#minor#modifications#were#made#to#the#
projection#installation#10#Minutes#to#Midnight#in#response#to#audience#feedback.#The#findings#also#
influenced#the#development#of#new#artworks#created#for#future#showcases,#such#as#the#creative#
approach#to#the#companion#projection#installation#Ngurini#(Searching)#that#was#developed#for#a#later#
exhibition.#For#Alphaville#as#host#organisation,#READ#findings#(including#the#resources#generated)#
were#used#for#organisational#learning,#partnership#development,#program#management,#funding#
proposals#and#strategic#planning.#Examples#of#utility#outcomes#for#community#partners#and#other#
stakeholders#are#included#in#Chapter#5.#
Conduct(
The#conduct#for#the#evaluation#practice#aimed#to#be#appropriate#and#inclusive,#to#ensure#relevance#
for#all#partners#and#so#that#ownership#of#the#process#was#shared#amongst#interested,#participating#
partners.#For#all#activities,#consideration#was#given#to#the#ethical#and#legal#aspects#of#practice,#and#all#
participating#stakeholders#adhered#to#ethical#conduct#guidelines#for#evaluation#practice#that#was#
overseen#by#myself#as#facilitator#(see#Australasian#Evaluation#Society,#2010,#for#code#of#conduct).#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 157#
Implementation#of#ethical#conduct#was#supported#by#industry#and#research#protocols,#such#as#
those#established#for#the#data#collection#and#documentation#with#persons#under#18#years#of#age#
(RIPP:#Article#7,#p.15#is#an#example#of#a#permission#form#that#required#parental#consent).##
For#some#cohorts#participating#in#the#READ#application,#their#activities#required#adherence#to#
specific#disciplineKbased#codes#of#conduct,#such#as#in#the#case#of#the#community#of#practice#
established#through#the#Analysis#strategy,#where#researchers#conformed#to#academic#policies#for#the#
ethical#conduct#of#research.#The#organisational#culture#of#Alphaville#also#guided#conduct#aligned#with#
principles#for#community#arts#and#cultural#development.#Overall#this#was#a#relatively#easy#process#to#
manage#as#facilitator,#as#the#creative#team#personnel#and#the#majority#of#partners#were#accustomed#
to#working#with#codes#of#conduct#and#adherence#to#these#was#understood#to#be#a#professional#duty.##
In#the#leadership#role#as#READ#facilitator,#my#conduct#was#important#as#a#I#was#a#role#model#for#
ethical#behaviour#and#responsible#for#facilitating#engagement#with#our#multiple#partners#in#a#
culturally#sensitive#and#just#manner.#Clements#comments:#
In#reality,#the#evaluator#is#awkwardly#placed#between#funder,#manager,#animateur#and#participant#as#well#as#methodologically#ensnared#by#conflicting#demands.#This#requires#a#keen#political#awareness#and#reflexivity#in#order#to#prioritise,#encourage#and#facilitate#transparency#as#well#as#help#enfranchise#participants#by#supporting#their#cultural#and#educational#rights,#thereby#enabling#greater#democracy.#(Clements,#2007,#p.334)#
As#facilitator,#one#of#the#potential#issues#I#identified#was#the#power#imbalance#between#myself#and#
other#stakeholders,#given#my#authority#in#governing#the#process#as#facilitator.#This#was#true#as#
facilitator#and#as#researcher,#as#I#had#final#authorship#of#the#account#of#the#fieldwork,#for#the#final#
design#decisions#for#the#application,#and#was#responsible#for#how#the#evaluation#practice#and#final#
READ#product#were#represented.#Fonow#and#Cook#(1991,#p.9)#point#out#that#the#‘emphasis#on#
collaboration#between#researcher#and#researched#masks#the#real#power#of#the#researcher,#who#has#
much#greater#control#over#the#research#process#and#product’.#Reflective#practice#as#an#evaluator’s#
tool#was#used#to#explore#my#process#of#integrating#partner#input,#to#assess#the#authenticity#and#
priority#of#partners’#input,#and#to#gain#insight#into#the#levels#of#agency#I#was#assuming#at#different#
stages#of#the#READ#application#(in#the#field#and#in#writing#up#the#research).##
Credibility((
In#assessing#the#credibility#of#the#READ#application,#I#examined#the#three#elements#stated#by#
Davidson#(2005,#p.211)#to#be#most#important#for#ensuring#credibility:#‘(a)#familiarity#with#the#
context;#(b)#independence,#impartiality,#and/or#lack#of#conflict#of#interest;#and#(c)#expertise#in#
evaluation#and#in#the#subject#matter#under#investigation’.#For#the#Nuclear#Futures#CCPI#context,#I#
interpreted#credibility#to#mean#gaining#acceptance#by#and#maintaining#relevance#to#community#
#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 158#
partners#during#the#exchanges#and#outputs#produced#by#the#READ#approach.#Further,#we#aimed#for#
the#approach#and#its#outcomes#to#be#deemed#acceptable#and#sound#by#peers#and#other#community#
arts#practitioners.#I#determined#that#there#were#a#number#of#factors#that#supported#the#credibility#of#
the#READ#methodology#and#implementation.#These#included:#
•! The#participatory#approach#was#inclusive#of#various#cultural#and#knowledge#paradigms#to#
reflect#partner#values,#capacity#and#needs#(important#for#the#community#partnerships#
context).#
•! READ#processes#and#findings#were#based#on#the#different#and#combined#expertise#of#a#range#
of#individuals#across#professional#and#community#roles#participating#at#the#site#of#practice.#
•! The#guiding#principles#and#priorities#for#both#internal#and#broader#learning#during#the#READ#
application#were#partnerKoriented,#based#on#stakeholder#input,#inclusive#of#all#partners.#
•! Stakeholder#expertise,#input#and#participation#was#overtly#valued,#as#well#as#critically#
reviewed.#
There#was#also#evidence#that#the#READ#application#being#undertaken#as#academic#research#added#
credibility#for#the#trial#and#ultimately#the#findings,#from#the#perspective#of#several#stakeholders#
(based#on#feedback#and#interviews).#While#the#READ#research#was#not#funded#by#the#Australia#
Council#for#the#Arts,#the#close#association#with#Nuclear#Futures#partners#and#Alphaville#through#the#
program,#also#provided#credibility#for#the#READ#application#for#the#Balaklava#project.#Further,#my#
known#previous#experience#in#evaluation#and#program#management,#combined#with#my#familiarity#
with#the#project#context#and#subject#matter,#afforded#credibility#for#me#as#READ#facilitator.#The#
known#expertise#of#Alphaville#personnel#(who#were#handsKon)#also#added#significant#credibility.#
Cost(and(time(
The#metaKevaluation#criteria#of#cost#and#time#were#the#most#difficult#to#analyse,#as#the#READ#
facilitation#had#been#undertaken#as#academic#research#and#did#not#reflect#the#resourcing#or#terms#of#
reference#of#a#standard#industry#application.#For#example,#the#time#demands#for#conducting#the#
READ#application#as#academic#research#were#much#greater#than#if#the#application#had#been#
administered#by#Alphaville#as#an#industryKbased#evaluation#practice.#The#academic#conventions#also#
influenced#my#management#style#and#the#level#of#focus#directed#to#the#theoretical#aspects#of#
practice.#While#this#enhanced#the#evaluation#practice,#this#was#beyond#the#scope#of#my#usual#
practitioner#approach.#In#order#to#maximise#innovation#and#encourage#experimentation,#there#was#a#
deliberate#flexibility#for#the#scope#and#scale#of#the#READ#activities,#which#had#time#and#cost#
implications,#and#these#managerial#aspects#of#the#READ#application#were#not#addressed,#as#they#
would#have#been#outside#an#academic#context.#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 159#
The#process#of#developing#and#testing#the#tailored#READ#framework#beyond#the#conceptual#
representation#of#READ#formulated#for#Alphaville’s#funding#applications#took#concerted#effort.#The#
Balaklava#project#was#the#first#of#the#Nuclear#Futures#projects#to#come#under#the#READ#lens,#so#it#was#
not#a#straightforward#delivery#of#an#established#approach.#As#this#dissertation#documents,#my#
research#has#involved#devising#both#a#tailored#methodological#framework#for#the#Balaklava#project#
and#a#more#generic#model#of#practice,#and#these#each#involved#extensive#consultation#and#iterative#
development.#Such#processes#were#often#concurrent#with#the#implementation#itself.#Lastly,#following#
the#Balaklava#project,#it#is#evident#from#my#experience#of#facilitating#the#READ#approach#in#other#
Nuclear#Futures#projects#that#the#evaluation#approach#at#this#site#of#practice#is#so#much#more#
advanced#as#a#result#of#the#knowledge#gained#and#systems#created#through#the#Balaklava#application.#
This#will#reduce#the#time#and#effort#of#applying#the#READ#model#of#practice#for#subsequent#projects.#
While#no#conclusion#was#possible#on#the#considerations#of#cost#and#time,#some#general#observations#
were#made:#
•! The#main#financial#outlay#for#the#application#was#for#the#salaries#of#the#creative#team,#
involved#in#the#delivery#of#technical#aspects#of#documentation#(covered#by#host#organisation#
Alphaville).#This#paid#work#overlapped#significantly#with#their#creative#residency#activities#
and#was#often#absorbed#within#the#creative#development#artist#fee#and#project#budget.#My#
role#was#not#remunerated#by#the#Balaklava#project#budget,#as#the#research#was#resourced#
separately.#Some#mentoring,#training#and#resources#for#team#members#was#paid#for#also.#
•! My#role#as#facilitator#demanded#the#most#time#out#of#all#of#the#READ#team#roles.#
•! In#most#cases,#involvement#by#participating#stakeholders#was#unpaid,#inKkind#input.#The#
voluntary#contributions#from#stakeholders#were#usually#absorbed#by#their#paid#role#with#
partner#organisations.#In#some#instances,#community#partners#were#paid#fees#or#had#their#
expenses#covered#for#participation#as#their#input#was#not#covered#by#a#funded#organisation.#
However,#offers#to#do#so#were#often#not#accepted#by#the#community#partners.#For#
community#partners,#READ#was#seen#as#an#extension#of#existing#community#arts#activities,#
rather#than#being#viewed#as#completely#extraneous.#From#informal#feedback#it#appears#
partners#were#not#focused#on#remuneration#as#a#motivation#to#participate.#
•! An#important#aspect#of#managing#partner#contributions#was#ensuring#equity#amongst#
partners#in#respect#who#was#participating#via#paid#versus#inKkind#support.#Given#the#high#
level#of#inKkind#input,#the#READ#activities#were#managed#to#generate#mutually#beneficial#
outcomes#for#the#inKkind#stakeholders.#
•! No#equipment#needed#to#be#hired#or#purchased#for#data#collection#or#analysis,#as#video#and#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 160#
photography#equipment#was#already#available#for#the#community#arts#projects#already#
underway.#InKkind#contributions#by#institutional#partners#covered#some#nonKsalary#costs#
such#as#printing#of#evaluation#instruments.#
•! NonKsalary#costs#and#equipment#needs#were#mainly#associated#with#the#storage#and#
management#of#the#data#collected,#which#included#the#need#for#at#least#10#large#hard#drives#
for#storage#and#to#create#a#robust#backup#system.#An#enormous#volume#of#data#was#
generated#from#the#documentation#process.#
•! Creative#artists,#arts#workers,#community#partners#and#institutional#partners#considered#the#
READ#processes#and#outputs#to#be#useful,#yet#it#was#also#considered#that#the#data#collection#
and#processing#was#timeKconsuming.##
•! Feedback#from#the#artists#indicated#that#is#was#often#challenging#for#them#to#undertake#the#
dual#roles#of#artist#in#residence#and#READ#documenter.#In#some#cases,#the#dual#briefs#were#
very#demanding.#The#role#of#documenter#was#shared#by#several#people#to#help#appease#this.#
•! There#were#many#timeKconsuming#aspects#of#the#READ#application,#such#as#data#
management#and#storage,#plus#the#review,#synthesis#and#editing#of#the#data#collected.#
•! The#strategy#to#work#with#academics#through#the#community#of#practice#relied#on#longKterm#
processes#and#relationships,#which#extended#well#beyond#the#life#of#the#Balaklava#project.#
This#arrangement#has#ongoing#time#and#resource#implications#for#the#project#partners#and#
for#host#organisation#Alphaville.#Community#partners#and#stakeholders#working#with#the#
academics#or#participating#in#the#community#of#practice#are#doing#so#in#a#voluntary#capacity.#
Thus,#this#may#not#be#a#suitable#arrangement#for#shortKterm#community#arts#activities.##
•! LongKterm#strategies#and#methods#require#ongoing#partnership#brokerage,#a#resource#
demand#for#the#host#organisation,#and#will#require#sustained#community#engagement#and#
feedback#cycles#to#ensure#information#is#disseminated#back#to#communities.#
•! Another#longKterm#task#is#overseeing#and#resolving#coKauthorship#issues,#so#that#all#those#
contributing#to#the#READ#outputs#–#whether#it#be#directly#through#artKmaking#or#the#analysis#
process#–#are#appropriately#acknowledged.##
•! For#public#dissemination#of#READ#outputs,#it#is#not#known#what#level#of#vigilance#will#be#
required#by#Alphaville#to#periodically#the#check#the#materials#in#institutions#or#
collections/repositories.##
•! Even#after#the#project#had#finished#(and#partner#closure#for#the#project#was#undertaken),#
evaluation#and#READ#data#that#required#processing#and#storage#continued#to#emerge.#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 161#
Constraints(and(strengths(of(the(READ(model(and(evaluation(practice(
The#above#discussion#of#time#and#cost#implications#illuminates#some#of#the#constraints#of#the#
methodology.#One#consequence#of#the#wide#use#of#data#collection#methods#was#the#implications#for#
storing#and#managing#data,#not#merely#for#READ#analysis#processes#but#for#longerKterm#accessibility.#
The#four#READ#strategies#produced#very#large#quantities#of#data,#which#meant#a#system#for#
organising#(cataloguing)#and#safeguarding#storage#was#required.#There#was#also#systematic#
documentation#of#authorship#required#for#the#creative#and#research#content.#These#types#of#
associated#tasks#were#challenging#aspects#of#the#facilitation#role,#and#certainly#had#time#and#cost#
implications.#On#the#flipside,#the#archive#of#information#and#material#is#extremely#valuable#for#
communities#and#participating#organisations#as#they#are#assured#ongoing#access.#Copies#of#the#
digital#content#were#shared#with#community#partners#(based#on#community#selection/requests)#to#
ensure#they#retain#ownership#and/or#access#to#the#material.#
One#of#the#broader#constraints#of#the#READ#model#of#practice#identified#is#that#READ#will#not#offer#an#
approach#suitable#for#all#contexts#and#evaluation#needs,#and#will#have#limitations#for#application#
depending#on#the#capacity#of#the#host#organisation#to#drive#the#concurrent#READ#strategies,#partner#
interests,#and#the#resources#and#timeframes#allocated#to#the#READ#process.#It#is#worth#reiterating#
that#the#READ#model,#as#with#any#guiding#evaluation#framework,#will#not#guarantee#positive#
appraisal#outcomes,#participation,#inclusion#or#the#delivery#of#socially#and#politically#just#means#for#
data#collection,#analysis#and#management.#Likewise,#given#the#flexibility#of#the#model#and#the#extent#
of#methodological#variation#that#may#result,#the#READ#model#cannot#guarantee#validity#or#credibility#
of#the#findings#generated#in#future#applications.#These#are#matters#determined#by#the#organisation,#
management#and#methodological#design#of#READ#systems#in#the#future.#
I#interpret#this#latter#finding#to#be#both#a#weakness#and#a#strength#of#the#READ#model#developed#to#
date.#On#the#one#hand,#as#mentioned#above,#it#means#that#the#responsibility#is#transferred#to#the#
practitioners#implementing#the#READ#model#in#the#future,#to#ensure#that#their#tailored#methodology#
is#rigorous#and#robust,#based#on#the#needs#of#the#application.#As#a#strength#however,#it#ensures#
methodological#design#of#the#READ#approach#can#be#reflexive#and#contextKspecific,#and#have#the#
capacity#to#respond#to#variation.#During#the#READ#application,#there#were#two#instances#in#
particular,#that#demonstrated#the#benefit#of#the#reflexivity#of#the#evaluation#practice#afforded#by#the#
READ#approach#in#responding#to#variation.#
The#first#was#related#to#a#variation#in#project#delivery#from#the#originally#conceived#community#arts#
project,#and#also#a#time#schedule#extension.#During#the#project,#the#original#concept#for#the#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 162#
community#showcase#evolved,#shifting#from#a#performanceKbased#theatre#showcase#to#a#diversified#
multiKarts#approach#to#suit#the#emergent#community#interest#in#photography#and#visual#arts.#The#
personal#health#circumstances#of#one#of#the#creative#team’s#key#artists#were#also#a#consideration#in#
this#variation.#This#variation#represented#a#major#shift#in#our#original#creative#brief#and#had#
significant#ramifications#for#the#creative#practice#and#community#partnerships,#and#also#for#the#READ#
methodologies#that#were#already#being#designed#and#implemented.#
The#READ#methodology#proved#flexible#enough#to#respond#to#the#resulting#project#variations#and#
interestingly#was#effective#at#capturing#and#recording#the#changes#in#ways#that#were#revealing#of#the#
arts#practice#and#processes.#The#main#strategy#impacted#was#Evaluation,#and#this#needed#revising#in#
order#to#ensure#the#instruments#for#data#collection#were#suited#to#the#participants#and#artKmaking#
that#eventuated.#For#Analysis,#this#strategy#wasn’t#affected#as#the#community#of#practice#was#
working#along#longer#timeframes#than#the#Balaklava#project#alone.#It#did#shift#the#research#interests#
to#some#degree,#and#if#anything#the#shift#in#arts#practice#was#viewed#as#an#interesting#development#
in#the#project#from#a#research#perspective.##
Interestingly,#the#strategies#of#Reflection#and#Documentation#played#a#role#in#supporting#and#
documenting#the#shift#in#creative#practice,#which#was#an#unforeseen#value#of#these#strategies.#For#
example,#the#Reflection#methods#and#approach#did#not#need#to#change#in#response#to#the#project#
variation#and#was#a#means#to#capture#learning#and#feedback#from#the#creative#team#and#community#
partners#about#the#process,#and#sharing#these#insights#advanced#the#creative#development#in#a#new#
direction.#Reflection#was#a#lens#to#analyse#the#challenges#and#choices#at#hand#and#elicited#rich#
findings#in#this#respect.#Documentation#also#played#a#valuable#role#in#capturing#the#shift,#and#the#
implications#for#the#creative#process#and#the#unfolding#drama#of#the#partners#to#respond.#At#the#
same#time,#the#reflection#and#documentation#methods#kept#the#continuity#of#data#collection#
happening#while#we#were#reorienting#the#evaluation#questions#and#agreed#approach.#
The#second#variation,#instigated#by#external#circumstances#beyond#our#control,#was#the#
announcement#by#then#Federal#Arts#Minister,#George#Brandis,#of#proposed#budget#cuts#of#
significance#for#the#Australia#Council#for#the#Arts,#in#May#2014.#The#reforms#were#a#direct#threat#on#
the#arm’sKlength,#peerKreviewed#grants#model#of#the#Australia#Council#and#generated#a#massive#
reaction#from#the#arts#sector.#One#development#of#direct#relevance#for#the#Nuclear#Futures#program#
was#that#the#Australia#Council#terminated#its#CCPI#fund.#While#the#funding#commitment#for#the#
existing#CCPI#projects#(such#as#Nuclear#Futures)#was#honoured#and#there#was#no#budget#loss#for#
current#projects,#the#context#of#the#arts#practice,#and#consequently#the#relevance#of#evaluation#
changed#significantly.#In#responding#swiftly#to#the#proposed#reforms,#internal#structural#changes#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 163#
ensued#for#the#Australia#Council,#as#did#a#serious#level#of#uncertainty#within#the#organisation.#
These#events#unfolded#during#the#months#at#the#close#of#the#Balaklava#project,#when#the#evaluation#
findings#were#scheduled#to#be#delivered#to#partners.#One#implication#of#this#timing#was#that#the#
Australia#Council#had#reduced#capacity#for#engaging#with#the#program#and#with#the#formative#
evaluation#findings.#When#I#asked#my#colleague#what#the#changes#might#mean#for#our#reporting#to#
the#Australia#Council,#he#jokingly#replied#‘where#we#used#to#do#a#5Kpage#report#we’ll#be#doing#a#3K
pager’.#While#this#comment#was#in#jest,#it#stuck#with#me#in#a#reflective#practice#session,#and#in#
examining#the#impact#of#the#external#events,#it#led#me#to#reaffirm#the#merit#and#significance#of#our#
decision#to#direct#our#evaluation#efforts#towards#the#interests#of#all#partners,#and#not#only#the#needs#
of#the#major#funding#body.#I#contemplated#the#potentially#different#scenario#of#our#situation#if#the#
driving#motivation#for#the#project’s#evaluation#had#been#solely#directed#towards#an#evaluation#report#
for#the#Australia#Council#criteria,#and#how#redundant#the#efforts#may#have#been#compared#to#what#
developed#through#the#READ#approach,#where#learning#and#evaluation#outcomes#were#delivered#for#
many#partners.58#While#the#upshot#of#the#reforms#remain#unknown#(at#the#time#of#writing),#it#is#
possible#that#by#2017#–#when#the#summative#findings#from#the#READ#approach#are#presented#to#the#
Australia#Council#about#Nuclear#Futures#and#its#broader#social#implications#as#a#CCPI#–#that#the#
findings#could#have#increased#relevance#for#the#Australia#Council,#potentially#offering#policyKrelated#
information#on#a#wellKdocumented#and#rigorously#evaluated#case#example#that#demonstrates#the#
value#of#experimental#and#partnershipKoriented#community#arts#funding.#
Recommendations(
The#first#recommendation#for#future#evaluation#practice#is#that#the#methodology#for#the#collective#
process#of#review#and#interpretation#of#the#READ#data,#including#collective#processes#of#evaluative#
reasoning,#can#be#improved#and#further#developed.#From#my#analysis#of#the#READ#application,#it#was#
an#area#of#evaluation#practice#that#could#be#strengthened#at#the#site#of#practice.#One#observation#
from#the#Balaklava#project#was#that#participation#from#multiple#stakeholders#was#easier#to#facilitate#
during#the#data#collection#phase,#as#was#the#analysis#of#data#collected#from#the#early#project#
activities.#This#was#because#these#collection#processes#could#occur#faceKtoKface#and#concurrently#
with#the#creative#residencies#when#visiting#artists#were#on#the#ground#and#the#project’s#momentum#
was#high#and#visible#for#partners.#For#the#reflective#and#evaluative#assessment#processes#in#the#later#
stages#of#the#project,#as#in#the#case#of#the#public#showcase,#while#some#progress#was#made#at#the#
#############################################################
58#This#is#not#to#suggest#that#Australia#Council’s#reduced#capacity#for#serious#evaluation#affects#the#benefits#that#individuals#or#organisations#generate#from#wellKconstructed#evaluations#of#previous#projects,#when#applying#for#new#funding.#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 164#
close#of#the#season#and#during#bump#out#(where#several#facilitated#debriefs,#and#reflective#and#
evaluative#assessment#meetings#were#conducted),#the#process#extended#over#several#months#and#
required#virtual#communication#with#participants#given#our#geographic#dispersion.#For#the#
summative#evaluation#of#Nuclear#Futures,#this#will#be#one#of#the#challenges#of#facilitation,#and#will#
require#planning#and#processes#for#enabling#adequate#and#robust#systems#for#collective#input.#
The#second#key#recommendation#is#that#the#scope#and#scale#of#the#READ#application#needs#to#be#
clearly#defined,#and#this#proved#an#aspect#difficult#to#gauge#for#the#Balaklava#application#because#of#
its#academic#context#(as#outlined#in#previous#discussion).#For#the#application,#this#aspect#of#READ#
planning#was#not#managed#as#it#would#have#been#if#READ#was#undertaken#solely#in#the#industry#
context#as#a#practitioner.#Rather,#the#scope#and#scale#of#the#READ#application#were#deliberately#kept#
open#and#flexible#to#encourage#experimentation#and#innovation.#Defining#the#scope#and#scale#will#
need#to#be#adequately#addressed#by#the#READ#facilitator#and/or#program#manager#to#ensure#
viability#of#the#application.##
6.2.6! Opportunities0for0enriching0evaluation0practice0
This#section#reports#on#the#opportunities#identified#during#the#reflective#analysis.#The#opportunities#
represent#the#areas#of#the#READ#approach#that#made#a#significant#contribution#to#the#efficacy#and/or#
enrichment#of#evaluation#practice#in#the#community#arts#context.#
Reflective(practice(
Reflection#was#used#as#a#data#collection#method#across#the#READ#methodology.#Significant#outcomes#
were#achieved#through#the#approach#to#reflection#as#a#core#READ#strategy,#arising#from#the#culture#
of#reflective#practice#that#was#established.#Reflective#practice#represents#a#significant#opportunity#
for#future#evaluation#practice#in#several#ways.#First,#the#collective#program#of#reflective#practice#
enabled#artists#and#community#members#to#become#critical#analysts#of#the#practice#in#new#ways,#
and#they#were#encouraged#to#share#their#insights.#This#resulted#in#individual#benefits#and#
constructive#exchanges#between#peers.#One#artist,#for#example,#explained#that#she#would#not#have#
normally#conducted#reflective#practice,#if#not#for#the#systematic#and#collective#process#implemented#
through#the#READ#approach,#and#that#the#reflections#had#added#value#to#the#iterative#cycles#of#her#
creative#practice.#Numerous#critical#buddy#exchanges#and#small#group#reflections#were#undertaken#
also,#which#helped#to#progress#creative#development#(evidenced#by#feedback#and#observation).#
Second,#there#was#continuous#improvement#through#organisational#learning#and#the#sharing#of#
information#generated#from#the#reflections.#For#Balaklava#this#related#to#the#improvement#of#the#
program#management#and#community#arts#practice.#Based#on#my#experience#across#disciplines#in#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 165#
evaluation#and#program#management#previously,#systematic#reflective#practice#for#organisational#
learning#is#not#widely#used#and#this#presents#an#opportunity#for#community#arts#organisations.#
Davidson#and#Rowe#(2009)#explain#how#tacit#knowledge#within#organisational#culture#is#often#
untapped#because#procedural#systems#for#capturing#knowledge#fail#to#draw#out#practical#wisdom:#
Closely#linked#to#the#organizational#culture,#this#difficultKtoKdefine#knowledge#adds#a#perspective#that#comes#only#from#experience,#is#rarely#acknowledged#or#written#down,#and#may#be#intuitive,#yet#contributes#to#judgment#and#wisdom.#This#can#produce#the#creation#of#serendipitous#knowledge#creation#which#more#procedural#systems#inhibit#through#their#emphasis#on#capturing#explicit#knowledge#alone.#(Davidson#and#Rowe,#2009,#p.565)#
The#role#of#reflective#practice#in#capturing#and#translating#tacit#knowledge#for#the#benefits#of#the#
creative#team,#community#partners,#and#Alphaville#organisation#more#broadly,#was#a#significant#
lesson#for#the#management#of#the#Nuclear#Futures#creative#practice,#which#will#have#implications#for#
evaluation#practice#in#the#future.#
Third,#as#well#as#its#value#for#improving#creative#practice#and#program#management,#reflective#
practice#was#relevant#during#the#READ#application#as#an#evaluator’s#tool,#where#individual#and#
collective#reflection#focused#on#the#READ#processes#and#results,#and#this#improved#the#conduct#of#
evaluation#practice.#This#finding#is#consistent#with#theorist#such#as#Bolton#(2010),#who#argue#the#
relevance#of#reflective#practice#for#evaluators:#‘Reflection#on#the#part#of#the#professional#evaluators#
is#also#crucial,#given#the#inherently#politicised#and#valueKbased#nature#of#evaluation,#and#the#need#
for#critical#monitoring#of#bias’#(Bolton,#2010,#p.5).#Thorpe#asserts#that#‘only#through#contemplating#
what#one#is#doing#and#why#can#one#demonstrate#that#intelligent#and#competent#practice#ensues’#
(Thorpe,#2004,#p.330).#For#the#Balaklava#project,#I#concluded#that#the#Reflection#strategy#of#the#READ#
approach#contributed#to#the#continuous#improvement,#robustness#and#logic#of#the#overall#evaluation#
practice.#
Unlocking(the(academies(
Another#opportunity#for#enriching#evaluation#practice#is#demonstrated#by#the#READ#methods#aiming#
to#unlock#the#academies.#As#explained#in#Chapter#5,#Section#5.4.4,#this#concept#relates#to#arts#
organisations#partnering#with#academic#institutions#to#promote#knowledge#generation#and#exchange#
between#academic,#industry#and#community#partners#through#social#enquiry#to#promote#broader#
learning.#While#this#approach#is#not#unique#and#follows#several#other#successful#examples#in#the#field#
(such#as#by#Somebody’s#Daughter#Theatre#Company#and#Milk#Crate#Theatre),#it#is#relatively#
underutilised#as#a#knowledgeKgeneration#strategy#for#community#arts#and#cultural#development.#For#
Nuclear#Futures,#the#community#of#practice#that#resulted#augmented#partnership#development#and#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 166#
is#producing#research#outputs#that#engage#with#the#wider#implications#of#the#community#arts#
practice#in#ways#that#would#not#have#been#delivered#through#conventional#evaluation.#
Legacy(goals(
The#innovative#use#of#longKterm#artefacts#and#processes#to#keep#the#stories#circulating#and#
accessible#to#future#generations#of#artists,#communities#and#researchers#was#concluded#to#be#an#
original#and#socially#driven#outcome#of#the#READ#methodology.#The#impulse#of#the#legacy#approach#
was#motivated#by#the#objective#of#the#creative#practice#to#‘make#the#story#go#far’,#to#create#
intergenerational#messaging#and#promote#conversations#between#communities,#artists,#researchers#
and#audiences.#The#objective#aligns#with#ideas#by#Mark#(2011)#that#meaningful#evaluation#should#
have#consequences,#such#as#community#development#outcomes#or#social#justice.#The#legacy#goals#
that#were#included#for#READ#extended#the#purpose#of#the#evaluation#practice#beyond#those#more#
orthodox#purposes#such#as#accountability#and#improvement.#
Early#findings#reveal#that#the#iteration#of#the#READ#model#for#Balaklava#has#already#contributed#to#
the#initial#realisation#of#this#objective#by#producing#multimedia#accounts#of#the#broader#range#of#
artworks#and#commentary#generated#during#the#program#that#are#now#secured#in#cultural#
repositories#and#in#the#public#domain.#For#example,#for#the#Balaklava#project,#instigating#legacy#was#
explored#through#a#partnership#with#the#State#Library#of#South#Australia#that#established#longKterm#
collection#development#for#oral#history#and#artefacts#generated#or#unearthed#during#the#project.#
Arrangements#with#families#for#future#donations#have#also#resulted.#The#significant#historical#items#
arising#from#the#creative#research#have#been#transferred#between#the#private#and#public#domains,#
and#will#have#the#benefit#of#being#organised#and#resourced#by#a#cultural#institution#with#a#stake#in#
longKterm#management.#While#it#can#only#be#speculative#as#to#the#benefits#the#approach#will#achieve#
(given#the#short#timeframe#of#assessment),#this#aspect#of#the#READ#application#offers#insight#into#
how#and#why#legacy#objectives#can#be#incorporated#into#the#evaluation#practice.#
The#second#example#from#the#READ#application,#highlighting#the#opportunity#for#evaluation#practice#
to#incorporate#legacy#goals,#was#through#the#utility#of#the#documentation.#It#is#argued#in#the#
literature#that#documentation#of#process#and#artefacts#enables#a#story#to#be#taken#further#afield,#and#
to#be#moved#and#used#beyond#the#story’s#place#of#origin#and#context#(UNESCO#&#ICCROM,#2015).#
This#is#true#for#the#documentation#about#the#creative#works,#where#the#message#and#impact#of#the#
creative#works#can#be#represented#through#the#documentation,#and#in#some#instances#this#has#the#
potential#to#reach#wider#audiences#than#the#artwork#itself.#Documentation#offered#a#means#to#
maintain#and#extend#the#significance#of#the#work#produced.#Future#plans#for#a#series#of#webisodes#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 167#
and#a#featureKlength#documentary#film,#both#spinKoff#projects#from#the#original#Balaklava#project,#
promise#to#extend#the#utility#arising#from#the#creative#documentation#beyond#the#Nuclear#Futures#
CCPI#funding#and#become#resources#for#public#dissemination.#
Linking(creative(and(evaluation(practice(
The#fourth#opportunity#identified#to#enrich#evaluation#practice#in#the#community#arts#context#relates#
to#the#compelling#examples#demonstrating#the#benefits#of#the#creative#documentation#methods#
used,#which#extended#data#collection#and#diversified#the#endKuse#possibilities#for#READ#outputs#
(RIPP:#Article#26K27#p.98K104#illustrates#some#examples#of#creative#documentation).#The#impulse#
for#including#creative#documentation#in#the#READ#methodology#was#to#draw#on#the#strengths#of#the#
creative#practice,#which#in#the#case#of#the#Balaklava#project#was#inspired#by#documentary#art#
traditions.#From#the#outset,#this#meant#looking#for#opportunities#that#utilised#documentary#artK
making#practices#as#methods#for#achieving#the#goals#and/or#enquiry#of#the#evaluation#practice.#One#
way#this#was#realised#was#via#multiKplatform#creative#documentation,#using#collaborative#CACD#
processes.#Implemented#by#filmmakers,#multiKmedia#artists,#community#members,#program#
administrators#and#partners,#the#creative#documentation#methods#generated#extensive#raw#data#
and#produced#many#evaluationKrelevant#outputs.#This#included#a#vast#collection#of#photomedia#
and#video#footage,#the#development#of#promotional#assets,#storyKbased#artworks/#
documentation,#video#interviews#and#website#content#(see#RIPP:#Article#27,#p.100).##
In#the#READ#application,#creative#documentation#added#to#the#richness#of#the#results#when#used#
alongside#conventional#evaluation#instruments#such#as#surveys#and#questionnaires.#Creative#
documentation#methods#served#to#capture#nuanced#and#ephemeral#data#that#was#not#revealed#
by#the#quantitative#methods#(RIPP:#Article#15,#p.36#has#an#image#that#illustrates#this).#One#of#the#
strengths#of#the#data#collected#through#creative#documentation#was#that#it#provided#insights#
about#the#engagement#and#personal#reactions#of#participants#and#audiences.#Thomas#(2015),#in#
research#on#a#gallery#curation#case#study#at#Queensland#University#of#Technology,#makes#this#
observation#on#the#limitations#of#traditional#reporting#for#representing#engagement:#
What#is#lost#in#the#traditional#reporting#system#is#the#recording#of#information#pertaining#to#engagement.#Engagement#quality#is#a#less#tangible#KPI#[key#performance#indicators]#than#numbers#but#neverKtheKless#was#identified#as#an#intended#and#highly#relevant#deliverable#outcome#of#the#program.#Beyond#the#attendance#figures#the#richness#and#diversity#of#the#engagement#remained#hidden.#(Thomas,#2015,#p.50)#
Based#on#the#DCI#findings,#the#suite#of#creative#methods#used#in#the#READ#application#were#effective#
for#revealing#the#subtle#artistic,#intuitive#and#emotive#expressions#of#engagement#and#experience,#
#
#
#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 168#
making#a#valuable#contribution#to#the#data#collection,#as#well#as#serving#as#an#engaging#means#to#
review#and#present#data#(on#both#the#processes#and#products#of#the#community#arts#activities).#
This#finding#aligns#with#Patton’s#(1981)#assertion#that#creative#methods#(such#as#photomedia)#are#
useful#for#data#collection#as#well#as#making#meaning#and#communicating#results#(even#when#findings#
are#derived#from#other#more#conventional#and/or#nonKcreative#methods).#In#the#READ#application,#
analysis#revealed#that#the#process#of#making#and#sharing#documentary#art#was#an#entry#point#for#
engagement#with#evaluation#practice.#Overall,#the#central#activity#of#creative#documentation#
enhanced,#legitimated#and#contextualised#the#evaluation#practice#for#the#partners#and#artists#
because#of#the#links#it#made#with#the#arts#practice#and#program#goals.#This#was#true#for#several#
constituencies#within#the#partnerships,#and#of#note,#it#was#particularly#true#for#the#creative#team.##
In#a#context#where#trepidation#of#evaluation#was#overt,#another#related#finding#relevant#for#future#
practice#was#that#the#diversified#READ#approach#shifted#emphasis#off#‘evaluation’,#and#contradicted#
the#assumption#that#evaluation#and#the#creative#practice#under#review#are#mutually#exclusive.#As#a#
result,#it#transformed#group#perceptions#of#what#and#who#could#be#involved#in#the#practice#of#
evaluation#and#in#the#documentary#artKmaking#(see#Section#6.2.2#for#further#background#discussion).#
In#many#instances#the#lines#between#these#practices#blurred,#and#artists#and#community#partners#
used#the#art#methods#for#simultaneous#purposes.#Importantly,#the#creative#documentation#approach#
raised#the#question:#what#is#evaluation#practice#and#what#is#documentary#artKmaking?##
In#challenging#the#assumption#at#the#site#of#practice#that#evaluation#practice#is#extraneous#to#arts#
practice,#traditional#binaries#of#community#arts#and#evaluation#of#community#arts#were#discarded.#
Produced#were#artworks#that#were#evaluation#products,#and#evaluation#products#that#were#
artworks.#Highlighted#were#the#many#linkages#able#to#be#created#and#leveraged#between#artKmaking,#
evaluation,#documentation#and#storytelling.#In#turn,#this#led#to#fluidity#and#resource#efficiencies#
across#the#domains#of#the#READ#implementation,#creative#arts#practice#and#program#management.#
The#incorporation#of#creative#documentation#into#the#READ#practice#supported#the#notion#that#there#
are#many#ways#of#knowing,#and#many#valid#sources#of#knowledge#relevant#for#the#creative#
community#partnership#initiative.#More#broadly,#Hawkesworth#(1989,#p.551)#states#that#‘perception,#
intuition,#conceptualisation,#inference,#representation,#reflection,#imagination,#remembrance’#are#all#
forms#of#knowing.#This#highlights#the#opportunity#practitioners#and#partners#have#for#evaluating#
community#arts#using#creative#methods#as#an#alternative#way#of#learning,#knowing#and#representing#
the#arts#practice#and#partnerships,#to#complement#conventional#methods.#For#Nuclear#Futures,#
creative#documentation#broadened#the#emergent#language#of#READ#processes#and#findings#and#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 169#
offered#an#aesthetic#description#of#the#community#arts,#which#is#so#often#lacking#in#evaluation#
outputs#in#this#context.59#Doloughan#(2002)#describes#such#a#discourse,#in#the#context#of#art#and#
design,#as#one#that#integrates#‘the#discursive,#the#pictorial,#the#persuasive,#and#the#instrumental’#
(Doloughan,#2002,#p.62)#and#challenges#the#privilege#traditionally#given#to#written#language.#In#the#
community#arts#evaluation#context,#this#is#relevant#given#the#current#tendency#in#the#field#to#
attempt#to#represent#the#vibrancy#and#impact#of#community#arts#through#written#reports,#
bureaucratised#language,#and#static#communication#formats.##
Finally,#reflection#on#the#role#of#creative#methods#for#learning#about#and#appraising#arts#practice#led#
me#to#revisit#the#discourse#canvassed#in#the#contextual#scan,#such#as#Goldbard’s#(2015)#argument#for#
qualitative#storyKbased#methods#as#a#means#to#demonstrate#impact#without#the#narrow#prescription#
of#cultural#measurement#and#impact#quantification;#or#Badham’s#(2015)#notion#of#approaching#the#
notion#of#cultural#value#from#the#bottom#up#so#as#to#best#reflect#and#encapsulate#stakeholder#values#
and#context.#With#the#increased#affordability#and#access#to#good#quality#audio#and#video#equipment#
and#software,#as#well#as#evolving#online#media#platforms#and#creative#art#methods,#practitioners#are#
offered#new#scope#and#sophistication#for#documentation,#analysis,#learning#and#appraisal#in#this#
arena.#In#conclusion,#significant#opportunities#are#being#missed#for#linking#the#domains#of#creative#
practice,#evaluation#practice#and#program#management#of#community#arts,#as#was#experimented#
with#in#the#READ#application.#Greater#exploration#of#these#linkages#would#be#of#interest#for#the#field.#
6.3! The0final0READ0product0
6.3.1! Product0development0
The#findings#and#practical#lessons#arising#from#the#research#enabled#a#workable#model#of#practice#to#
be#established,#and#has#resulted#in#original#evaluation#theory.#In#response#to#the#shared#problem#of#
community#arts#evaluation#at#my#site#of#practice,#I#determined#that#one#useful#output#of#the#DCI#
would#be#a#simplified#READ#product#able#to#guide#future#evaluation#practice.#During#the#DCI#research#
I#engaged#in#the#product#development#of#such#a#tool,#to#consolidate#the#READ#methodology#and#
model#of#practice#developed#to#date.60#The#READ#product#presented#in#this#section#is#the#culmination#
of#the#READ#application#that#I#facilitated#at#the#site#of#practice#(see#Chapter#5)#and#the#critical#review#
I#conducted#using#forensic#reflective#practice,#including#the#metaKevaluation.#It#was#inspired#by#the#
need#to#communicate#the#emergent#READ#model#in#a#simple#and#engaging#way.##
#############################################################
59#One#of#the#best#examples#of#creative#documentation/evaluation#output#reviewed#was#the#memory#basket#by#Big#hArt#(2010)#for#Ngapartji#Ngapartji#(a#treasure#box#of#stories#and#symbolic#objects).#This#however#is#rare#in#the#field.#60#A#product#is#defined#as#‘a#good,#idea,#method,#information,#object#or#service#created#as#a#result#of#a#process#and#serves#a#need#or#satisfies#a#want.#It#has#a#combination#of#tangible#and#intangible#attributes’#(WebFinances,#2016).#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 170#
For#this#product#development#I#prioritised#my#commitment#to#reflexivity#and#creative#
experimentation#in#order#to#maximise#innovation,#as#well#as#the#use#of#the#systems#approach#
introduced#in#Chapter#5#(Section#5.1.2).#This#has#involved#generating#new#knowledge#about#each#of#
the#READ#strategies#and#methodologies,#and#the#system#as#a#whole,#in#order#to#substantiate,#reK
evaluate#or#redesign#ideas#from#the#conceptual#the#model.#Understanding#has#been#gained#into#
previously#unknown#aspects#of#the#model,#such#as#its#methodological#domains#and#flexibility,#the#
interrelationships#between#strategies,#how#the#integration#processes#achieve#a#wide#range#of#
evaluative#and#nonKevaluative#outputs,#and#the#nuances#of#the#operational#system#once#developed.#
Figure#6.3#shows#the#stages#of#the#structured#READ#product#development#through#the#DCI.##
#
############
#
###
##
Figure#6.3:#Structured#design#and#development#for#the#READ#product##
#(Diagram#is#based#on#‘Structured#Ideas#Development’#(author#unknown)#retrieved#May#9,#2014#from#http://www.processfacilitation.info/processKfacilitationKcreativeKcollaborationKstructuredKideaK
development/.#Link#no#longer#active.)##
#####
Early iteration READ concept
Tailored Balaklava design & initial
model development
Applying and testing at site
of practice
Model theorisation
refinement, & articulation
Final READ product
(consolidated findings)
2012 2016 (not to scale)
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 171#
6.3.2! Profile0for0the0READ0product0
Based#on#the#reflective#analysis,#I#developed#the#following#profile#for#the#READ#product.#The#profile#
lists#the#characteristics,#advantages#and#disadvantages#evident#during#model#development#and#
testing.##
Characteristics((
•! ParticipatoryKoriented#
•! Functions#as#an#integrated#or#modular#approach#to#learning#and#appraisal#
•! Organised#around#four#READ#strategies#–#Reflection,#Evaluation,#Analysis#and#Documentation##
•! Suitable#for#formative#or#summative#learning#and#appraisal#
•! Acknowledges#the#importance#of#the#processes#and#outcomes#of#community#arts#
•! Inclusive#of#culturally#diverse#partners#and#participants#
•! Incorporates#creative#and#artKbased#methods#
•! Participant#focus#and#program/arts#organisation#ownership#
•! Flexible,#able#to#respond#to#variation#
•! KnowledgeKgenerating#
Advantages(
•! Driven#by#locally#relevant#questions,#stakeholderKoriented#
•! Flexible#to#address#a#wide#range#of#stakeholder#needs#for#learning#and#appraisal#
•! Produces#multiKplatform#outputs#(evaluative#and#nonKevaluative)##
•! For#accountability,#improvement#(individual#and#organisational),#broader#learning#and#legacy##
•! Instrumental#and#developmental#benefits#
•! Adaptable#for#capacityKbuilding#and#participant#empowerment##
•! Participatory,#representative,#inclusive#
•! Sustains#organisational#learning#and#growth#
Disadvantages((
•! Time#and#commitment#of#all#involved#
•! ResourceKintensive,#depending#on#the#scope#of#READ#application#
•! Conflict#resolution/facilitation#may#be#needed#for#partnership#brokerage#
•! Not#necessarily#independent#findings#if#undertaken#internally#
•! LongKterm#management#and#maintenance#will#be#required#for#legacy#objectives#
#
#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 172#
6.3.3! Infographic0
Figure#6.4#presents#the#infographic#developed#to#summarise#the#transferable#elements#of#the#READ#
model#based#on#the#Balaklava#application#findings,#to#demonstrate#the#general#processes#involved#in#
the#READ#approach.#It#aims#to#simplify#and#describe#the#READ#product#as#a#visual#communication#
tool#and#as#a#means#for#branding#the#approach.#The#need#for#an#infographic#was#identified#during#
the#Balaklava#project,#as#I#became#aware#that#a#oneKstop#diagram#that#encapsulated#the#model#(in#
an#engaging#way)#would#be#useful#in#communications#with#stakeholders#and#project#participants.#
The#infographic#presented#in#Figure#6.4#aims#to#fulfil#this#purpose#and#will#be#used#for#future#Nuclear#
Futures#evaluation#practice,#and#to#present#the#model#of#practice#publicly#to#other#practitioners#and#
scholars.#It#has#not#yet#been#tested#at#the#site#of#practice,#and#it#is#anticipated#that#there#will#be#
further#iterations#of#the#infographic#developed#during#2016#based#on#stakeholder#feedback#and#my#
experience#of#using#it#in#practice.#
Design(rationale(
During#the#design#process,#I#experimented#with#several#approaches#for#visualising#the#READ#product.#
In#one#early#attempt,#I#explored#using#the#systems#approach#(based#on#Table#6.1,#and#further#
developing#Figure#5.1),#however#the#result#was#too#complex.#I#wanted#a#clear#and#visually#dynamic#
means#to#demonstrate#the#model.#In#this#section,#I#provide#an#explanation#of#the#emergent#design.#
The#circular#design#developed#as#a#way#to#represent#READ#as#a#summative#lens#on#community#arts#
and#cultural#development#practice,#which#has#been#included#at#the#centre#of#the#infographic.#In#the#
top#semiKcircle#are#the#four#READ#strategies,#clearly#distinguished,#with#their#respective#descriptions.#
The#impulse#to#use#the#pieKshaped#colour#pieces#for#the#READ#strategies#responded#to#the#need#to#
communicate#their#modular#and/or#integrated#potential.#The#READ#strategies#were#given#
prominence#in#the#design#and#visually#command#the#viewer#to#start#interpreting#the#infographic#
from#the#top.#
The#lower#half#of#the#infographic#represents#the#processes#involved#in#designing#and#implementing#
the#READ#approach.#The#inner#sphere#(of#the#lower#semiKcircle)#includes#colourKcoded#data#sources,#
to#provide#examples#of#the#data#collection#options#for#each#of#the#strategies.#It#aims#to#communicate#
the#scope#for#choice#for#practitioners.#Examples#include#gathering#data#from:#publications,#surveys,#
interviews,#field#notes,#sound#recordings,#documentation#and#so#on.#The#instructive#text#‘mix#+#
integrate’#signals#that#practitioners#tailor#the#selection#to#meet#their#needs#and#resources.##
# #
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 173#
#
#
Figure#6.4:#The#READ#model#infographic#
##
(Version#updates,#READ#model#summaries,#and#alternative#print#formats#will#be#made#available#in#the#future#via#www.nuclearfutures.org.)#
#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 174#
It#is#important#to#note#that#the#data#sources#included#in#the#infographic#represent#a#guide#only,#and#
are#not#prescriptive.#Also,#while#each#is#colourKcoded#to#offer#extra#insight#into#the#READ#strategies,#
some#methods#can#be#utilised#in#multiple#strategies.#For#example,#journaling#is#a#method#that#could#
be#used#in#all#four#of#the#READ#strategies,#but#for#the#purpose#of#the#infographic#this#has#been#colour#
labelled#within#Reflection.#This#is#one#potential#constraint#of#the#infographic#in#its#current#form.#
In#the#next#sphere,#in#the#lower#semiKcircle,#is#the#‘create#+#share’#band.#Included#in#this#section#are#
examples#of#the#resources#and#artefacts#able#to#be#produced#from#the#integrated#(synthesised)#data.#
This#band#communicates#the#products#potentially#generated#from#the#READ#model,#and#includes#
outputs#such#as#raw#data,#evaluation#reports,#publications,#artwork,#management#resources,#and#
project#stories.#To#keep#the#design#simplified,#I#elected#not#to#represent#the#types#of#analysis#
processes#that#may#be#relevant#for#the#integration#(see#Table#6.1).#While#the#infographic#provides#no#
instruction#for#the#integration#process,#the#act#of#integration#is#still#conveyed#and#I#considered#this#
sufficient#for#the#purpose#of#communicating#the#model’s#key#processes.#
The#outer#band,#in#the#lower#semiKcircle#communicates#the#purpose#for#which#READ#may#be#applied:#
for#accountability,#improvement,#broader#learning#and#instigating#legacy.#These#concepts#have#been#
described#in#Chapter#5#and#6,#so#are#not#covered#again#in#this#descriptive#section#on#the#infographic.#
This#band#aims#to#tell#the#viewer#what#can#be#expected#from#applying#the#model,#and#that#READ#can#
be#directed#towards#a#range#of#purposes.#Overall,#the#lower#half#of#the#infographic#conveys#that#the#
choice#of#data#sources#can#be#tailored#to#suit#the#ultimate#purpose#of#applying#the#model,#which#will#
vary#between#projects#and#contexts.#
I#envisage#that#when#used#in#practice#to#
communicate#the#READ#model#of#practice,#ideally#
the#infographic#will#be#verbally#explained#or#
accompanied#by#a#succinct#description#of#the#key#
concepts#via#a#basic#comprehension#guide#(see#
Chapter#7#for#future#research#and#dissemination#
plans).#Version#updates,#READ#model#summaries,#
and#alternative#print#formats#will#be#made#available#
in#the#future#via#www.nuclearfutures.org#(for#
example,#see#image#on#right#for#greyscale#version).# #
#
Greyscale#version#of#the#READ#infographic.#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 175#
6.3.4! Assumptions,0constraints0and0variables0
The#emergent#READ#product#presented#in#this#section#is#closely#linked#to#the#context#from#which#it#
has#been#developed.#In#the#literature,#it#is#well#documented#that#evaluation#approaches#are#
significantly#contextualised#and#politicised#based#on#their#underpinning#theoretical#concepts#and#
contextual#influences#(Edenloff,#2011;#Davidson,#2005).#Some#scholars#such#as#Boud#and#Walker#
(1998,#p.196)#argue#that#context#is#‘so#allKpervasive#that#it#is#difficult#to#recognise#its#influence’.#With#
this#in#mind#I#used#reflective#practice#to#critically#examine#the#assumptions#and#constraints#of#the#
READ#model,#and#identified#a#list#of#variables#influencing#future#practice.##
One#of#the#significant#assumptions#made#at#the#site#of#practice#when#deciding#to#experiment#with#
the#READ#concept#was#that#engaging#communities#and#partners#in#Reflection,#Evaluation,#Analysis#
and#Documentation#was#going#to#be#beneficial#for#participants/organisations,#and#produce#positive#
experiences#and#outcomes.#However,#as#SielbeckKBowen#et#al.#(2002,#p.4)#points#out,#‘the#evaluation#
or#research#process#can#lead#to#significant#negative#or#positive#effects#on#the#people#involved#in#the#
evaluation/research’.#In#reality,#evaluation#practice#has#the#potential#to#produce#harm#if#conducted#
unethically#or#cause#distress#to#participants#for#reasons#unknown#to#the#evaluators#(MacDowall,#et#
al.,#2013).#Related#is#another#assumption#of#the#model#of#practice:#that#encouraging#participation,#
collaboration#and#learning#as#part#of#the#integrated#READ#approach#will#result#in#positive#exchanges#
for#partners#and#participants.#In#recognising#these#assumptions,#it#is#recommended#that#
practitioners#using#the#READ#approach#in#the#future#constantly#challenge#and#scrutinise#the#
assumptions#for#their#application#and#context#(during#all#phases#of#the#project).#In#doing#so,#
measures#can#be#taken#to#avoid,#minimise#or#respond#to#potential#negative#harm#that#may#occur.##
A#number#of#variables#were#identified#that#will#have#a#bearing#on#the#performance#and#efficacy#of#
future#READ#applications.#These#variables#revolve#around#biases#produced#by#the#selection#of#the#
READ#facilitator#and#team,#the#context#in#which#it#is#implemented,#and#the#methodological#solutions#
designed#for#each#READ#strategy.#In#summary,#the#variables#identified#for#READ#practice#are#as#
follows:#
•! Interpretation:#Although#the#model#attempts#to#distinguish#Reflection,#Evaluation,#Analysis#
and#Documentation#as#independent#yet#interrelated#READ#strategies,#the#methods#
applicable#for#each#can#overlap#significantly,#and#the#strategies#and#methods#will#be#subject#
to#interpretation.##
•! Purpose:#This#includes#why#the#implementation#is#being#undertaken#and#what#stakeholder#
needs#are#to#be#served.#
•! Subject/Focus:#The#CACD#processes#or#activities#at#the#focus#of#the#READ#application.##
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 176#
•! Context:#This#includes#consideration#of#the#impact#of#different#institutional#settings#(public#
and#private#arts#organisations,#government#agencies,#independent#artist,#education#arts#
settings);#and#the#social#context#(individual#or#group/team,#internal#versus#external#
position);#as#well#as#the#cultural#(including#ideologies#and#protocols)#and#geographic#context#
(access,#safety,#resources).#
•! Partnerships#and#level#of#participation:#Questions#arise#about#which#stakeholder#interests#
are#reflected#and#represented#during#the#scoping,#design,#conduct,#implementation#and#
dissemination#phases,#and#how#participatory#the#application#needs#to#be.#(Note:#
Participation#does#not#guarantee#ethical#or#representative#input#or#validity#or#relevance#of#
findings#(SielbeckKBowen#et#al.,#2002))#
•! Stakeholder#values:#Davidson#(2005)#states#that#the#question#is#not#whose#values#should#be#
incorporated#but#what#values,#based#on#the#context#and#terms#of#reference.#
•! Scale,#scope#and#site#of#practice:#For#smallKscale#READ#applications#the#available#resources,#
time#and#expertise#may#be#limited#compared#to#large#organisations#that#may#have#resources#
to#engage#in#more#comprehensive#or#longitudinal#studies#for#national#or#international#
programs.#
•! EndKuses#and#formats:#This#includes#questions#of#immediate#versus#longKterm#utility,#the#
varying#audiences#involved#and#format#(or#language)#for#delivered#outputs#such#as#digital,#
hardcopy,#written,#audio#visual#and#so#on.#
•! Conduct:#There#will#be#different#and#conflicting#approaches#on#how#to#implement#the#READ#
framework#and#what#key#players#hope#to#get#out#of#it.#
•! Design#and#implementation:#This#includes#the#methodological#and#epistemological#aspects,#
and#methods#selection#(qualitative,#quantitative#and/or#rich#ephemeral#data).#
From#these#variables,#and#based#on#the#findings#from#the#contextual#scan,#I#then#developed#a#
preliminary#list#of#guiding#questions#for#future#evaluation#practice#(addressed#to#the#practitioner).#I#
include#the#question#set#as#a#demonstration#of#the#analysis#of#future#variables,#to#illustrate#the#
considerations#arising#from#the#reflective#exercise.#The#questions#for#future#practitioners#are#
presented#on#the#following#page.#
# #
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 177#
Preparation#questions#for#future#READ#practitioners:#
•! What#is#your#site#of#practice#or#context?#
•! Why#is#READ#being#undertaken?#
•! Why#is#improved#learning#and/or#appraisal#of#value?#
•! How#do#you#currently#learn/reflect/evaluate/analyse/share?#
•! What#is#the#scale#and#scope#of#the#activities#or#review?#
•! What#do#you#hope#to#achieve?#
•! What#barriers#exist#(such#as#for#evaluation,#improving#learning#and#sharing)?#
•! What#information#is#useful#and#for#whom?##
•! What#would#you/partners#like#to#know?#Are#you#prepared#to#be#surprised?#
•! Which#‘language’#will#best#represent#your#story#(text,#images,#film,#reports,#statistics,#
infographics)?#
•! How#rigorous#do#you#need#to#be?##
•! What#will#count#as#‘evidence’?#
•! Is#it#a#solo#or#shared#process?##
•! Who#will,#or#needs#to#be,#involved,#and#in#what#capacity?#
•! Will#you/participants#need#support?#Where#will#you#find#it?#
•! How#and#who#will#facilitate#and#broker#relationships?#
•! What#philosophies#and#priorities#will#frame#the#approach?#
•! Is#organisational#learning#and#a#culture#of#reflection#relevant?#
•! Would#you#like#to#advance#professional#practice?#
•! Are#the#evaluative#questions#set,#or#do#they#need#developing?#
•! What#evaluation#values#need#to#be#represented?#
•! What#methods#best#match#your#learning#and#appraisal#needs?#
•! Is#creative#documentation#of#interest?#
•! What#would#a#strengthKbased#approach#look#like?#
•! Where#are#the#current#gaps#in#information#and#communication?#
•! How#will#meaning#be#made#and#shared?#
•! What#validity#checks#will#be#included?#Is#external#review#relevant?#
•! How#will#you#engage,#empower#and/or#inspire#others#along#the#journey?#
•! What#are#the#dissemination#goals#and#platforms#for#information#sharing#most#relevant#to#
your#setting?#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 178#
6.4! Chapter0conclusion0
The#development#and#application#of#the#READ#model#offered#a#contextKspecific,#inKdepth#exploration#
of#the#theory#and#practice#of#integrating#Reflection,#Evaluation,#Analysis#and#Documentation#as#a#
summative#lens#for#learning#and#appraisal#within#a#dynamic#community#arts#setting.#The#READ#
application#conducted#for#the#Balaklava#Community#Arts#Project#was#guided#by#an#interpretative#
approach#that#allowed#for#a#detailed#understanding#about#the#model#in#action#to#be#developed.#
From#the#findings,#I#constructed#a#general#model#of#practice#and#positioned#this#within#the#field#
alongside#contemporary#evaluation#theories.#This#informed#the#development#of#a#consolidated#READ#
product#for#future#practice,#and#the#reflective#analysis#confirmed#a#number#of#key#findings.##
The#findings#demonstrate#that#the#emergent#READ#model#offers#flexibility#as#an#integrated#or#
modular#evaluation#approach#that#can#be#tailored#for#varying#needs.#It#is#argued#that#while#
reflection,#evaluation,#analysis#and#documentation#are#routine#methods#used#in#contemporary#
evaluation#practice,#the#innovation#of#the#READ#model#lies#in#the#integration#of#these#as#strategies.#
For#evaluation#practice,#this#means#that#the#model#can#be#compartmentalised#in#order#to#maximise#
the#desired#learning#and#appraisal#outcomes,#and#that#multiKplatform#outputs#can#be#produced#for#a#
range#of#purposes#including#accountability,#improvement,#broader#learning#and#creating#legacy.#The#
process#is#also#inclusive#of#creative#methods#that#offer#rich#descriptive#representations,#and#that#are#
not#necessarily#driven#by#valuation,#merit#attribution#or#judgement#as#the#central#concern.##
The#findings#also#indicate#that#READ’s#capacity#to#deliver#benefits#for#partners#was#enhanced#by#the#
internal#conduct#of#the#evaluation#practice.#My#dual#role#of#READ#facilitator/manager#created#
numerous#advantages,#such#as#realKtime#information#exchange#and#utilisation#across#READ#activities,#
management#and#creative#practice.#In#keeping#with#the#community#arts#ethos,#the#internal#position#
kept#the#power#to#govern#learning#and#appraisal#processes#and#outcomes#within#the#creative#
partnerships.#On#examination#of#the#constructivist#stance#adopted#for#the#READ#application,#the#
interpretive#approach#demonstrated#good#credentials#for#offering#an#inclusive#and#partnerKfriendly#
means#of#accommodating#the#divergent#world#views#and#evaluation#constructs#at#the#site#of#practice.##
In#the#chapter#I#also#presented#a#broader,#holistic#assessment#of#the#model,#based#on#results#using#
analytical#and#metaKevaluation#tools#(for#evaluating#the#evaluation)#where#efficacy#was#analysed#
against#the#criteria#of#validity,#utility,#conduct,#credibility#and#cost.#This#highlighted#the#strengths#and#
weaknesses#of#the#READ#model#during#the#Balaklava#trial,#and#the#opportunities#and#constraints#of#
the#READ#approach.#The#most#significant#constraint#identified#of#the#READ#methodology#was#the#
time#and#resources#required#to#manage#the#four#concurrent#READ#strategies,#and#the#subsequent#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 179#
information#management#demands#arising#from#the#volume#of#data#collected.#Further,#while#the#
model#provides#a#guiding#framework#for#learning#and#appraisal#through#the#four#READ#strategies,#
there#is#limited#prescription#for#the#design#of#the#methodology#or#participatory#processes,#given#that#
evaluations#are#best#designed#and#implemented#to#be#contextKspecific.#Practitioners#applying#the#
READ#model#will#thus#need#to#ensure#the#appropriateness,#rigour#and#credibility#of#their#tailored#
framework#in#response#to#partner#needs#and#context,#and#consider#strategies#that#may#be#required#
to#satisfactorily#manage#the#vulnerabilities#of#the#model#when#internally#delivered.#
Of#broader#relevance#are#the#opportunities#revealed#by#the#READ#approach,#and#the#insight#these#
offer#for#extending#and#enriching#evaluation#practice.#First,#significant#value#was#added#from#the#
collective#program#of#reflection,#which#delivered#benefits#as#an#evaluator’s#tool,#for#the#
improvement#of#creative#arts#practice,#and#as#a#strategy#for#organisational#learning.#Second,#the#
methods#implemented#for#achieving#the#broader#learning#and#legacy#objectives#were#shown#to#
extend#the#purpose#and#impact#of#the#evaluation#practice,#in#order#to#disseminate#and#safeguard#
READ#outputs#in#the#public#domain.#This#included#methods#to#unlock#the#academies,#and#the#
potential#to#involve#cultural#institutions#as#partners#in#the#evaluation#practice,#so#that#the#wider#
implications#of#the#community#arts#practice#become#known#and#shared.#Fitting#with#the#community#
partnerships#approach#defining#contemporary#community#arts,#these#methods#widened#the#project’s#
scope#for#creating#mutually#beneficial#partnerships#between#arts#and#nonKarts#organisations#(one#of#
the#goals#for#Creative#Communities#Partnership#Initiative#funding).#Last,#the#findings#point#to#the#
opportunities#offered#by#creative#documentation#–#for#engagement,#participation,#linking#project#
domains#and#drawing#on#documentary#art#traditions#–#to#the#point#where#artKmaking#and#evaluation#
practice#were#performed#simultaneously#on#many#occasions.#
The#findings#provide#a#scaffold#for#learning#and#appraisal#as#well#as#a#practical#tool#for#
communicating#READ’s#integrated#approach.#In#summing#up,#the#READ#application#demonstrated#
that#at#the#site#of#practice#the#original#knowledge#generated#through#the#READ#lens#impacted#and#
improved#evaluation#practice#and#contributed#to#advancing#the#community#partnerships.#Of#
significance#was#the#value#delivered#from#the#mix#of#evaluative#and#nonKevaluative#outputs#
produced,#a#consequence#of#READ’s#wide#focus#and#multiKstrategy#approach.#The#findings#in#Chapter#
6#establish#the#grounds#for#the#general#model#of#READ#practice#proposed#through#the#DCI#and#is#the#
foundation#for#the#original#knowledge#claims#argued#in#Chapter#7,#the#final#chapter#of#the#
dissertation.#In#Chapter#7,#I#reflect#on#the#two#overarching#research#questions#and#present#my#
summative#conclusions.#In#addition,#I#discuss#the#research#implications#on#my#own#evaluation#
practice,#at#my#site#of#practice#and#the#field,#and#my#plans#for#future#dissemination#and#research.#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 180#
#
Blank#page#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 181#
70 Conclusion0
This#research#aimed#to#develop#and#test#a#new,#integrated#approach#to#evaluation#for#the#
community#arts#context#–#the#READ#model.#The#READ#model#combines#Reflection,#Evaluation,#
Analysis#and#Documentation#(READ)#to#create#a#wide#summative#lens#for#accountability,#
improvement,#broader#learning,#and#legacy#purposes.#Prior#to#this#research,#the#idea#for#READ#
existed#in#concept#form#only.#It#was#coined#in#2012#by#SydneyKbased#arts#company#Alphaville#to#
meet#the#evaluation#needs#of#the#Nuclear#Futures#Partnership#Initiative#(Nuclear#Futures),#an#
international#arts#and#cultural#program,#and#the#site#of#my#professional#practice.#The#Doctor#of#
Creative#Industries#(DCI)#offered#a#means#to#advance#the#early#READ#concept#into#a#working#model#
of#practice,#and#trial#the#collectively#developed#methodology#in#an#industry#context.#To#study#the#
design,#efficacy#and#potential#of#the#READ#model,#I#utilised#forensic#reflective#practice#as#my#
principal#research#strategy.#This#involved#the#analysis#and#theorisation#of#the#emergent#READ#
methodology,#to#create#original#knowledge#on#the#READ#model#and#a#product#for#future#practice.##
Chapter#7#consolidates#the#research#and#presents#the#findings#from#the#investigation.#In#focus#are#
my#summative#reflections#on#the#Nuclear#Futures#application#of#the#READ#approach,#elucidating#the#
key#practitioner#insights#and#original#knowledge#arising,#to#address#the#two#DCI#research#questions:#
a)! How#does#the#READ#model#contribute#to#program#learning#and#appraisal#for#creative#community#partnerships#within#community#arts,#based#on#the#Nuclear#Futures#Partnership#Initiative#experience?##
b)! What#innovation#does#READ#offer#evaluation#practice#and#program#evaluation?#
The#chapter#begins#with#a#summation#of#the#shared#problem#of#community#arts#evaluation#tackled#in#
the#DCI,#and#next,#draws#out#the#key#elements#of#the#READ#model#of#practice#resulting#from#the#
research.#I#review#the#DCI#outputs#and#the#areas#of#innovation#identified,#as#well#as#the#assumptions#
and#constraints#of#the#findings.#The#relevance#and#impact#of#the#research#are#examined#for#my#own#
professional#evaluation#practice,#at#the#site#of#practice,#and#for#the#field#of#community#arts#and#
beyond.#Following#this,#I#provide#my#intent#and#direction#for#dissemination#and#future#research.#
The#summative#conclusions#corroborate#the#efficacy#and#impact#of#the#READ#model#as#a#learning#and#
appraisal#framework#at#the#site#of#practice,#and#it#is#argued#that#the#integrated#mixed#method#
approach#offers#both#rigour#and#reflexivity.#As#well#as#equitably#dealing#with#divergent#stakeholder#
needs#and#values#(including#accommodating#trepidation#of#evaluation),#READ#embraces#methods#for#
engagement#and#knowledgeKgeneration#embedded#in#community#arts#and#cultural#development.#In#
widening#the#scope#and#creativity#of#evaluation#practice,#the#research#and#the#developed#READ#
model#advance#evaluation#practice#and#deliver#an#innovative#new#approach#for#community#arts.##
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 182#
7.1! The0DCI0research0
My(approach(to(a(shared(problem(
The#DCI#research#was#inspired#by#the#shared#evaluation#problem#arising#at#my#site#of#practice,#during#
the#planning#and#establishment#of#the#Nuclear#Futures#program#during#2012#and#2013,#by#arts#
company#Alphaville.#Amongst#our#cohort,#we#considered#how#we#could#best#approach#the#requisite#
evaluation#of#this#large#and#multiKstakeholder#community#arts#program#(funded#by#the#Australia#
Council#for#the#Arts),#and#in#doing#so,#many#questions#were#raised.#As#program#manager,#I#was#a#key#
player#in#these#discussions#and#deliberations.#
As#discussed#in#Chapter#2,#evaluation#in#community#arts#is#contentious#and#widely#disputed.#The#
advent#of#increasingly#structured#evaluation#as#a#routine#requirement#for#external#funding#sees#
practitioners#face#mounting#pressure#to#demonstrate#the#value#of#their#work#in#competitive#
environments.#Coupled#with#this#is#the#broader#trend#towards#cultural#measurement,#and#the#
pursuit#to#quantify#the#value#and#impact#of#the#arts.#For#practitioners#undertaking#evaluation#of#
community#arts,#there#are#both#philosophical#and#technical#challenges#to#navigate,#and#no#agreed#
framework#or#criteria#for#best#practice#exists.#While#significant#developments#have#occurred#in#the#
field#over#the#past#30#years,#effective#evaluation#practice#for#community#arts#remains#elusive,#
unresolved#and#disputed.#At#one#extreme,#opponents#argue#evaluation#in#its#current#usage#and#
orientation#is#detrimental#for#the#field#(Goldbard,#2006,#2008,#2015;#Belfiore#&#Bennett,#2008).##
Inspired#by#the#needs#of#the#Nuclear#Futures#Partnership#Initiative,#with#its#diverse#community#
partnerships#and#multiKarts#creative#practice,#host#organisation#Alphaville#sought#an#original#solution#
to#the#dilemma#of#community#arts#evaluation.#For#the#Nuclear#Futures#program,#the#required#
evaluation#framework#would#need#to#assess#the#processes#and#outcomes#of#community#arts#with#a#
variety#of#stakeholders,#from#artists#to#atomic#survivor#communities#in#Australia,#Japan,#Kazakhstan#
and#the#Marshall#Islands,#as#well#as#to#evaluate#the#multiKplatform#creative#works#produced,#and#the#
impact#generated#for#partners#and#participants.#To#address#the#key#evaluation#questions#arising,#
Alphaville#devised#the#READ#concept#to#combine#Reflection,#Evaluation,#Analysis#and#Documentation#
as#concurrent#strategies#for#learning#and#appraisal.##
Prior#to#the#research,#only#a#basic#brief#for#the#conceived#READ#approach#existed#(see#Chapter#5,#
Section#5.1.1#for#the#280#word#READ#proposal#establishing#the#extent#of#previous#concept#
development).#To#create#a#working#READ#model,#the#DCI#research#has#centred#on#my#evaluation#
practice#as#lead#designer#and#facilitator#for#the#collective#development#process#of#a#READ#
methodology#for#the#Balaklava#Community#Arts#Project,#delivered#in#regional#South#Australia#in#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 183#
2014K5#as#the#first#of#the#Nuclear#Futures’#projects.#My#role#has#involved#operationalising#a#model#of#
practice#for#the#READ#approach,#and#consolidating#the#READ#epistemology#and#methodology#
through#a#process#of#critical#analysis#of#the#model’s#performance#in#action.#In#order#to#critically#
analyse#my#evaluation#practice,#and#the#emergent#READ#model#of#practice,#I#employed#forensic#
reflective#practice#as#the#principal#research#strategy#for#the#DCI,#and#this#utilised#research#methods#
such#as#written#and#video#reflection,#digital#storytelling,#interviews#and#observation#(see#Chapter#3).##
As#outlined#in#Chapter#5,#at#the#site#of#practice,#there#were#numerous#evaluation#needs#identified#in#
the#scoping#with#stakeholders#that#were#taken#into#account#during#the#collective#evaluation#practice,#
such#as#the#need#for:#formative#and#summative#outputs#(including#project#and#evaluation#reporting);#
storyKbased#and#creative#documentation;#community#archiving;#broader#learning#and#dissemination#
outcomes;#information#generation#for#continuous#improvement,#for#creative#practice,#evaluation#
and#management;#and#alignment#with#the#ethos#and#practice#of#the#community#arts.#The#focus#of#
the#scoping#was#both#the#Balaklava#Community#Arts#Project#and#the#Nuclear#Futures#program#
generally.#Based#on#the#findings#of#the#stakeholder#scoping,#I#determined#that#a#broad#definition#of#
evaluation#practice#would#need#to#be#adopted#in#order#to#cultivate#an#inclusive#and#engaged#culture#
of#learning#and#appraisal#at#the#site#of#practice,#and#to#produce#authentic,#representative#and#useful#
outputs#and#findings.#
To#ensure#relevance#and#credibility,#the#evaluation#solution#pursued#also#needed#to#suit#the#nuanced#
setting#of#the#Nuclear#Futures#creative#communities#partnership#initiative,#and#this#included#the#
need#to#embody#the#ethos#of#the#creative#practice#which#was#participationKoriented#and#strongly#
influenced#by#documentary#art#traditions#and#community#arts#and#cultural#development#(CACD)#
principles.#For#validity,#the#READ#model#of#practice#needed#to#rely#on#a#rigorous#and#substantiated#
methodology,#and#include#measures#for#managing#the#implications#of#the#internally#positioned#
evaluation#delivery.#In#addition,#overarching#the#siteKspecific#evaluation#needs#and#agendas#of#the#
participating#communities,#artists#and#institutions,#was#the#more#challenging#question#at#the#centre#
of#the#Nuclear#Futures#creative#practice:#What#creative#art#works#and#processes#can#endure#across#
the#deepKtime#of#the#nuclear#future?#This#question,#of#interest#to#the#Nuclear#Futures#cohort,#related#
to#the#objective#to#enable#the#program#stories#‘to#go#far’#into#the#deep#nuclear#future,#to#generate#
legacy#impacts#from#the#program#and#the#evaluation#practice#(Alphaville,#2015).#
During#the#DCI#I#facilitated#collective#input#and#participation#in#the#READ#model#development#prior,#
during#and#following#the#Balaklava#project,#and#this#was#combined#with#data#generated#from#my#
systematic#process#of#forensic#reflective#practice#to#formulate#the#articulated#READ#model#and#
product#presented#in#Chapters#6#and#7,#and#overall#findings.#While#the#READ#application#for#the#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 184#
Balaklava#Community#Arts#Project#was#the#central#research#focus,#it#was#intended#that#if#the#READ#
concept#was#deemed#to#be#viable#and#effective#as#a#potential#model#of#practice,#based#on#the#DCI#
findings,#that#READ#could#be#applied#across#all#projects#of#Nuclear#Futures#and#for#the#program’s#
summative#evaluation.#This#meant#that#the#DCI#research#promised#immediate#benefits#for#extending#
my#own#evaluation#practice,#as#well#as#a#prospective#means#to#develop#an#evaluation#model#for#the#
program#partners#and#host#arts#organisation#Alphaville.#Given#the#research#problem#also#resonated#
with#identified#research#and#knowledge#gaps#in#the#field#(see#Chapter#2),#the#academic#context#of#
the#research#presented#the#opportunity#to#share#any#innovations#in#evaluation#practice#with#other#
scholars#and#practitioners,#and#contribute#to#the#discourse#on#effective#community#arts#evaluation.#
DCI(outputs(
Overall,#the#DCI#research#has#advanced#my#professional#evaluation#practice,#and#the#evaluation#
practice#for#Nuclear#Futures,#and#makes#a#valuable#and#original#knowledge#contribution#to#the#
contested#field#of#community#arts#evaluation.#The#principal#DCI#research#outputs#are#the:#
1.! Dissertation#(this#document):#A#comprehensive#document#outlining#the#research#problem,#site#of#practice#and#research#findings.#Included#are#original#research#outputs#such#as#the#digital#story,#the#reflective#questions#set,#and#the#READ#product#(including#infographic).#
2.! READ#in#Practice#portfolio:#A#compilation#of#materials#and#resources#generated#through#the#evaluation#practice,#facilitating#the#READ#application.#Included#are#original#outputs#such#as#the#Nuclear#Futures#evaluation#plan,#designed#evaluation#instruments,#program#documentation#and#reports,#and#conference#presentations#(see#companion#document#submitted#for#examination#–#164#pages).#The#portfolio#is#intended#as#a#companion#document#to#the#dissertation#and#has#been#structured#as#a#standKalone#research#output.#
Assumptions(and(constraints(
In#presenting#this#research,#there#are#several#assumptions#and#constraints#to#acknowledge.#First,#the#
research#is#not#a#complete,#summative#analysis#of#the#READ#model#in#action.#It#focuses#on#the#early#
developmental#stage#of#the#conceptualised#model,#limited#to#the#study#of#READ’s#design#and#trial#for#
one#Nuclear#Futures#project.#For#some#aspects#of#READ,#such#as#the#Analysis#strand#and#academic#
outputs,#while#solid#progress#has#been#made,#several#of#the#anticipated#READ#outcomes#will#extend#
beyond#the#research.#Future#READ#applications#will#also#lead#to#the#further#refinement#of#the#model,#
such#as#addressing#the#weaknesses#identified#in#Chapter#6#(where#possible)#in#preparation#for#the#
summative#evaluation.#Second,#it#is#noted#that#the#considerations#and#scope#of#the#READ#application#
for#the#DCI#study#extends#well#beyond#the#usual#capacity#of#community#arts#evaluation#(where#
generally#evaluations#are#underKresourced).#For#this#reason,#subsequent#READ#applications#will#need#
to#be#adapted#to#suit#available#resources#and#expertise#using#the#infographic#and#guide#in#Chapter#6.##
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 185#
Third,#it#is#assumed#that#for#the#DCI#research#conducted,#that#the#validity#and#credibility#of#the#
research#have#been#adequately#addressed#in#the#research#design#(see#Chapter#3).#The#practiceK
informed#approach#and#qualitative#methodology#was#based#on#systematic#forensic#reflective#
practice,#and#utilised#methods#such#as#ongoing#reflection,#interviews,#digital#storytelling,#and#
observation.#I#consider#the#methodology#implemented#to#have#been#effective#for#accommodating#
the#practitionerKasKresearcher#approach#of#the#DCI#and#that#it#has#produced#reliable#and#valid#
results.#One#noted#constraint#of#the#DCI#methodology#is#that#there#are#limitations#for#generalising#
the#findings,#given#my#central#position#as#researcher#(as#evaluator#and#forensic#reflective#
practitioner)#and#the#strong#influence#of#the#context#on#the#evaluation#practice#(which#is#the#case#for#
all#evaluation#theory).#Other#methodKbased#constraints#limiting#the#potential#to#generalise#the#
findings#include#the#data#collected#from#the#interviews,#which#while#producing#rich#and#detailed#
information#of#relevance#for#the#context,#relied#on#a#very#small#sample#size#and#purposeful#selection,#
and#thus#is#not#representative#of#the#community#arts#field#in#entirety.##
Fourth,#the#DCI#findings#in#Chapter#6#are#contextKspecific,#inextricably#linked#to#the#Nuclear#Futures#
site#of#practice.#The#influence#of#context#has#had#an#impact#on#the#model’s#development#and#on#my#
own#reflective#practice.#For#example,#the#analysis#and#metaKevaluation#conducted#on#the#READ#
application#was#underpinned#by#criteria#for#good#practice#grounded#in#the#Balaklava#project#context.#
The#measures#and#constructs#of#good#evaluation#applied#in#the#reflective#practice#were#informed#by#
the#metaKevaluation#tools#selected,#my#values#and#interpretations#of#the#tools,#and#the#values#and#
priorities#identified#by#project#stakeholders.#As#a#result,#the#research#is#not#replicable,#as#no#two#
READ#applications#will#be#the#same.#Thus,#the#claims#made#about#the#outcomes#and#impact#of#the#
READ#model#in#this#dissertation#relate#specifically#to#the#Balaklava#designed#application.#
Despite#such#constraints,#the#findings#do#contain#significant#and#relevant#practical#instruction#about#
the#READ#model#of#practice#and#integrated#approach#beyond#the#site#of#practice,#and#is#certainly#
relatable#for#and#transferable#by#other#practitioners.#While#the#Balaklava#project#(and#Nuclear#
Futures#site#of#practice)#is#not#characteristic#of#all#creative#community#partnerships,#it#does#
exemplify#many#of#the#common#attributes#of#such#initiatives#(see#Chapter#2,#Section#2.1.2).#Thus,#it#is#
assumed#as#reasonable#to#present#the#READ#model#as#an#emergent#evaluation#theory#of#relevance#to#
other#sites#of#practice,#and#offer#it#as#an#original#and#credible#approach#for#community#arts#
evaluation,#based#on#its#success#in#the#Balaklava#context.#Fitzpatrick,#et#al.,#(2012,#p.112)#argue#‘the#
various#approaches,#or#theories,#proposed#by#Evaluators#make#up#the#content#of#the#field#of#
evaluation’.#In#this#way,#the#READ#product#becomes#a#generalised#model,#offering#a#guide#framework#
for#learning#and#appraisal#of#potential#interest#to#practitioners#beyond#the#site#of#practice.#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 186#
7.2! Summative0conclusions0
In#this#section#I#present#my#summative#conclusions#based#on#the#findings#from#Chapter#6.#I#argue#that#
the#READ#model#achieved#effective#and#appropriate#evaluation#outcomes#for#the#Balaklava#
Community#Arts#Project,#and#that#the#model#demonstrated#innovation#by#applying#an#integrated#
approach#to#learning#and#appraisal#that#utilised#the#creative#strengths#of#the#partnerships#and#arts#
practice.#In#doing#so,#I#argue#that#the#READ#approach#makes#an#original#and#relevant#contribution#as#a#
model#of#evaluation#practice#for#this#field.#While#contextKspecific,#the#conclusions#demonstrate#a#
number#of#innovative#developments#in#evaluation#practice,#and#provide#practical#insight#into#the#
epistemological#and#technical#aspects#of#the#READ#model#as#an#emergent#approach#for#community#
arts#evaluation,#which#is#of#potential#interest#to#other#scholars#and#practitioners.##
How(does(the(READ(model(contribute(to(program(learning(and(appraisal(for(creative(community(
partnerships(within(community(arts,(based(on(the(Nuclear(Futures(Partnership(Initiative(experience?((
The#READ#model#makes#a#strong#contribution#as#a#learning#framework.#It#produces#a#variety#of#
outcomes#from#utilising#the#evaluation#practice#as#a#knowledgeKdriven#lens#for#gaining#understanding#
into#the#community#arts#practice.#This#included#the#fulfilment#of#contextKspecific#and#broader#
learning#goals.#The#application#demonstrated#that#the#developed#READ#model#had#the#capacity#to#
incorporate#learning#processes#across#all#four#READ#strategies,#and#this#resulted#in#a#range#of#diverse#
learning#outcomes,#evidenced#across#three#planes:#the#individual#participant#level,#at#the#small#
groups/teams#level,#and#at#the#program#and#organisational#level.##
Examples#of#the#learning#outcomes#achieved#during#the#collective#design#and#application#process#
taken#from#Chapter#6#included:#
•! Professional#development#(via#mentoring,#training,#conference#attendance);#
•! Skills#development#(via#mentoring,#training,#workshops,#information#dissemination,#exposure#
to#new#information/partner#approaches#and#collaboration);#
•! Advancement#of#creative#practice#(via#individual#and#collective#reflective#practice#findings,#
application#of#evaluation#and#research#findings#during#creative#development);#
•! Advancement#of#evaluation#practice#(via#individual#and#collective#capacity#building#during#
design#and#implementation,#technical#training,#experimentation#and#empowerment);#
•! Development#of#a#sustained#culture#of#reflection#(particularly#for#Alphaville#and#the#core#
Nuclear#Futures#personnel);#
•! Establishment#of#a#community#of#practice#(practitioners#and#researchers#working#towards#
academic#analysis#goals#and#publication#of#research);#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 187#
•! Organisational#learning#(for#Alphaville#through#reflection,#evaluation,#analysis#and#
documentation#processes#and#outputs);#
•! Original#knowledge#generation#(via#artKmaking,#reflective#practice,#evaluation,#research);#
•! Continuous#improvement#(use#of#READ#data#to#improve#program#and#creative#practice,#for#
the#Balaklava#project#and#future#Nuclear#Futures#projects);#
•! Education#and#broader#learning#(academic#and#industry#presentations,#research#
dissemination;#contribution#to#collection#development#and#dissemination#of#educationK
related#resources);##
•! Partnership#development#(via#learning#exchange#and#information#sharing#between#partners).#
Importantly#for#the#various#future#contexts#to#which#the#READ#model#may#be#applicable,#the#learning#
outcomes#produced#benefited#a#mix#of#stakeholders#including#community#members,#artists,#
participants,#management#personnel,#staff,#and#representatives#from#partner#organisations.#Through#
the#allied#learning#strategies#of#READ,#knowledge#from#a#range#of#sources#was#explored,#valued#and#
incorporated,#including#tacit,#local,#indigenous#and#scientific#data.#The#information#sharing#occurred#
in#structured#and#spontaneous#ways,#enhanced#by#my#dual#role#of#READ#facilitator/manager,#and#
this#in#turn#had#positive#impacts#for#the#creative#practice,#partnership#development,#program#
management#and#overall#learning#outcomes.#In#terms#of#maximising#the#utility#and#dissemination#of#
the#data#generated#through#READ,#the#internal#position#and#dual#role#of#the#READ#facilitator/#
program#manager#provided#an#enormous#advantage.#
For#the#evaluation#and#analysis#of#READ’s#performance#as#a#framework#for#appraisal,#I#defined#this#to#
encompass#the#broad#definition#of#evaluation#formulated#at#the#site#of#practice,#which#included#
more#general#sensemaking#processes.#At#the#site#of#practice,#appraisal#focused#on:#the#creative#
process,#artworks#produced,#community#engagement,#project#progress,#and#partnership#quality,#
dynamics#and#needs.#Based#on#the#data#generated,#findings#confirmed#that#the#mix#of#knowledgeK
driven#appraisal#methods#employed#across#the#four#READ#strategies#sufficiently#met#the#identified#
evaluation#needs#of#the#stakeholders.#Stakeholder#feedback#on#the#evaluation#processes#and#
outcomes#(and#the#overall#experience#of#READ)#indicated#consistently#that#READ#deliverables#in#the#
domain#of#appraisal/evaluation#exceeded#expectations#and#requirements.#The#Balaklava#READ#
application#also#performed#well#against#Davidson’s#metaKevaluation#criteria#(2005):#validity,#utility,#
conduct#and#credibility#(see#Chapter#6).#On#assessing#cost,#the#final#criterion#from#Davidson’s#set,#no#
conclusion#was#possible#as#the#READ#application#did#not#follow#the#usual#scope#of#practice#(in#terms#
of#industry/practitioner#delivery)#and#conformed#to#academic#research#conventions.##
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 188#
Interview/consultation#data#and#the#forensic#reflective#practice#findings#revealed#that#in#addition#to#
the#instrumental#value#arising#from#the#READ#model#(in#terms#of#its#direct#use),#the#integrated#model#
had#conceptual#value,#which#changed#the#way#evaluation#was#practiced#and#generated#new#
understandings#of#the#interrelationships#between#evaluation,#creative#practice#and#program#
management.#One#interesting#observation#as#facilitator,#was#that#once#the#model#was#known#to#
stakeholders,#many#appeared#to#be#more#receptive#and#engaged#with#the#terminology#and#concept#
of#READ,#as#opposed#to#‘evaluation’.#In#taking#the#emphasis#off#‘evaluation’#as#terminology,#which#
came#with#previously#constructed#notions#of#the#practice,#the#diversified#READ#methodology#
developed#in#the#research#(and#the#acronym#itself)#changed#the#perception#and#nature#of#the#
evaluation#practice.#As#the#READ#facilitator#for#the#pending#summative#evaluation#of#the#Nuclear#
Futures#program#(across#many#other#projects),#this#is#an#important#finding,#which#will#be#influential#
for#future#evaluation#practice#at#the#site.#In#the#community#arts#context#more#broadly,#where#there#is#
evident#trepidation#of#evaluation#for#some#stakeholders,#this#finding#on#the#impact#of#READ’s#
alternative#rhetoric#for#appraisal#and#evaluation#is#also#of#significance.#
For#program#appraisal,#the#READ#application#involved#numerous#approaches#to#evaluative#thinking#
and#analysis,#and#this#increased#the#scope#of#what#was#included#as#evaluation#practice#and#the#
outcomes/outputs#this#produced.#The#use#of#creative#documentation#methods#for#example,#
supported#evaluative#processes#driven#by#an#accountability#agenda#while#utilising#moreKsubtle#
appraisal#processes#for#exploring#creative,#aesthetic,#cultural#and#social#questions.#Both#the#
Reflection#and#Documentation#strategies#of#the#READ#approach#played#a#particularly#important#role#
in#engaging#with#the#often#underKvalued#or#disregarded#tacit#systems#of#knowledge#generation#
inherent#in#community#arts#practice#and#program#management,#allowing#experimental,#artsKbased#or#
partnerKspecific#suggested#methods#to#be#explored.#While#these#strategies#tapped#into#tacit#
processes#that#were#already#being#employed#by#the#creative#team,#the#activities#appeared#to#be#
extended,#made#explicit,#and#shared#because#of#the#collective#READ#practice.#The#findings#revealed#
that#the#use#of#creative#documentation#positively#influenced#participation#and#engagement#in#
evaluation#practice,#and#legitimated#evaluative#activities,#particularly#for#the#creative#team.#Securing#
the#interest#and#participation#of#the#creative#team#in#the#evaluation#practice#was#important,#and#
meant#data#collection#became#a#component#of#creative#practice,#and#built#on#the#relationships#the#
artists#had#established#with#community#partners#through#the#artKmaking.##
The#findings#in#Chapter#6#also#elucidate#some#of#the#potential#issues#and#challenges#of#adopting#an#
integrated#approach#to#evaluation#practice,#and#identified#several#areas#of#the#formulated#READ#
approach#that#will#require#further#consideration.#Specifically,#the#potential#cost,#time#commitment#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 189#
and#the#issue#of#managing#the#large#volume#of#data#generated#to#ensure#accessibility#and#utility#
present#real#challenges#for#using#the#fourKstrand#READ#model.#In#order#to#manage#the#concerns#
identified,#and#minimise#the#associated#risks#during#future#applications,#it#is#essential#that#the#scope#
and#goals#of#future#READ#applications#are#clearly#defined.#It#is#acknowledged#that#for#more#
sophisticated#and#demanding#applications,#such#as#the#pending#Nuclear#Futures#summative#
evaluation,#new#domains#of#evaluation#practice#may#need#to#be#explored#and#potentially#integrated,#
such#as#external#review#processes#as#a#check#for#validity#of#internal#findings,#and#the#possible#use#of#
indicators#and#metrics.##
Another#constraint#identified#in#the#research#is#that#the#READ#approach#and#methods#trialled#were#
part#of#a#contextKspecific#application#designed#in#response#to#the#unique#evaluation#needs#at#the#site#
of#practice.#It#cannot#be#claimed#that#the#integrated#READ#methodology#will#suit#all#contexts#and#
community#partnerships#in#community#arts,#and#this#is#an#accepted#limitation#of#the#model#for#
future#practice.#Applying#the#READ#model#as#a#guide#to#evaluation#practice#cannot#guarantee#
consistent#learning#or#appraisal#outcomes,#validity#of#findings,#credibility,#participation,#or#the#
delivery#of#a#socially#and#politically#just#means#for#data#collection,#analysis#and#management.#Any#
future#applications#of#the#model#will#be#different#to#the#case#studied#in#this#research,#and#will#be#
shaped#by#variables#such#as#stakeholder#needs#and#dynamics,#resource#allocation,#organisational#
capacity,#and#the#rigour#of#the#contextKspecific#methodological#design.#
Despite#such#limitations#however,#the#READ#approach#was#effective#in#engaging#with#the#
complexities#of#the#community#arts#practice,#rather#than#succumb#to#an#oversimplified#or#toolkit#
methodology.#As#a#learning#and#appraisal#framework#it#demonstrated#its#potential#for#shifting#the#
focus#beyond#an#instrumental#benefits#lens,#to#be#inclusive#of#multiple#learning#and#knowledge#
systems#which#is#important#when#working#with#culturally#diverse#partners,#as#is#the#case#for#Nuclear#
Futures.#Further,#the#research#demonstrated#that#the#integrated#READ#approach#achieved#an#
alignment#between#the#practices#of#evaluation#and#community#arts,#where#the#vibrancy#and#ethos#of#
the#arts#practice#became#exemplified#and#delivered#through#the#evaluation#practice.##
As#a#practitioner#negotiating#multiple#roles#within#the#complex#creative#community#partnership#
initiative,#the#research#produced#valuable#insights#into#how#to#strategically#integrate#the#
management,#READ#practice#and#artKmaking#of#the#program#to#produce#benefits#for#each#domain.#
The#result#for#the#Balaklava#application#of#READ#was#a#blurring#of#lines#between#these#three#domains#
at#the#site#of#practice,#and#I#would#argue#that#this#contributed#to#the#model’s#effectiveness.#The#
implication#for#my#own#practice#is#that#for#future#iterations#of#using#the#READ#approach#I#will#
prioritise#establishing#strategic#links#between#these#domains#as#a#core#part#of#practice.##
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 190#
I#refer#back#to#the#relationship#illustrated#in#Figure#5.1#between#READ,#program#management#and#
creative#practice:##
#
#
#
#
#
#
#
As#well#as#working#well#across#the#above#three#spheres#of#practice,#the#research#demonstrated#that#
the#READ#model#offers#a#flexible#evaluation#approach#that#can#be#used#as#an#integrated#or#modular#
framework,#and#it#is#this#attribute#of#the#model#that#increases#its#potential#relevance#for#the#varying#
contexts#within#community#arts.#This#means#that#individual#READ#strategies#can#be#employed#
independently,#or#in#tandem,#to#cater#for#particular#learning#and#appraisal#objectives,#or#
accommodate#stakeholderKpreferred#learning#modalities.#For#example,#a#solo#creative#practitioner#
wanting#to#understand#and/or#promote#their#arts#practice#for#the#purpose#of#improvement#or#
audience#development#could#apply#Reflection#and#Documentation#as#a#combined#solution.#The#
viability#and#broader#implications#of#using#the#READ#model#in#its#integrated#or#modular#capacity#was#
critically#assessed#during#the#reflective#analysis#and#metaKevaluation#of#the#Balaklava#application,#
and#was#identified#as#an#area#of#innovation#demonstrated#by#the#model#(see#Chapter#6).#
Overall,#the#READ#model#has#the#potential#to#present#as#both#a#simple#and/or#complex#system.#In#its#
Balaklava#iteration,#READ#was#ambitiously#designed#and#comprehensively#enacted.#The#convenient#
READ#acronym#offered#a#useful#mantra#for#those#participating,#a#shared#compass#for#the#collective#to#
guide#our#evaluation#practice#that#produced#outcomes#and#impact#beyond#the#expectation#of#the#
partner#collective.#While#it#is#beyond#the#scope#of#this#research#to#verify#whether#the#efforts#and#
input#employed#for#the#wideKranging#Balaklava#application#have#or#will#be#matched#by#the#benefits,#
in#my#position#as#dual#READ#facilitator/manager#I#can#see#that#the#benefits#of#the#READ#application#
will#be#of#value#to#Alphaville#(at#the#very#least)#for#years#to#come.#Another#question#arising#from#the#
research#is#about#READ’s#integrated#approach,#and#whether#the#outcomes#of#the#READ#system#as#a#
whole#add#up#to#more#than#the#sum#of#its#parts.#For#the#DCI,#with#the#limited#industry#application#
#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 191#
conducted,#my#conclusion#is#that#it#does.#However,#to#what#extent#this#occurs#is#a#question#for#future#
research.##
In#closing#this#section,#I#present#a#list#made#during#the#forensic#reflective#practice#(a#kind#of#‘note#to#
self’#reflection)#of#the#key#lessons#and#conclusions#from#the#evaluation#practice#that#I#will#take#with#
me#as#READ#facilitator#for#use#in#the#next#round#of#Nuclear#Futures#evaluation#practice,#and#beyond.#
While#there#are#still#some#issues#to#improve,#and#further#applications#to#implement#and#analyse,#this#
list#was#my#summation#of#the#practical#lessons#I#drew#from#the#Balaklava#application:#
•! The#internal#position#meant#previous#knowledge#of#context#was#utilised,#and#advantages#
were#created#from#this.#
•! The#dual#position#of#READ#facilitator/manager#enabled#strategic#pathways#to#be#established#
and#maintained#between#READ,#creative#practice#and#program#management.##
•! Data#collection#and#analysis#could#be#spontaneous#and#reflexive,#as#well#as#
structured/planned.#
•! Access#to#data#in#realKtime#increased#the#capacity#of#both#positions,#and#in#turn#increased#
responsiveness#and#reflexivity,#and#utility#of#READ#or#program#management#data.#
•! The#socially#constructivist#interpretative#stance#was#an#effective#means#to#deal#with#the#
diverse#range#of#evaluation#constructs#and#values#present#for#stakeholders#and#partners.#
•! Scoping#was#fundamental#for#establishing#the#terms#of#reference#and#determining#partner#
capacity#and#interest#to#participate.#
•! If#negotiated#and#agreed#with#partners,#the#terms#of#reference#for#evaluation#practice#can#
extend#to#evaluative#and/or#nonKevaluative#processes#(in#terms#of#attributing#worth).#
•! The#creative#documentation#methods#were#of#high#relevance#to#the#project#and#collective,#
producing#a#range#of#benefits#including#increased#engagement,#ownership,#participation#and#
utility,#whereby#creative#processes#became#evaluation#practice.#
•! The#culture#of#reflection#that#developed#contributed#to#individual#and#organisational#
learning.#Particular#value#was#gained#internally#for#the#creative#team#through#sharing#and#
responding#to#reflections.#Data#was#often#of#immediate#use.#
•! Reflective#practice#performed#well#as#an#evaluator’s#tool.#
•! A#range#of#developmental#outcomes#was#evident,#including#organisational#learning#
(Alphaville#and#partners),#partnership#development,#professional#and#skills#development,#
and#program#development.#
•! Unlocking#the#academies#(by#partnering#researcher#institutions)#is#a#longerKterm#strategy#
that#added#credibility,#created#a#community#of#practice#and#added#value#for#community#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 192#
partners,#participants#and#audiences#(see#Chapter#5:#Section#5.4.4#for#background#on#
unlocking#the#academies#concept).#
•! The#methodological#flexibility#of#the#model#was#important#in#a#changing#environment,#and#
for#responding#to#project#or#creative#practice#variations.#
•! Introducing#legacy#and#archival#objectives#to#the#evaluation#practice#has#the#potential#(if#
successful)#to#extend#the#impact#of#evaluation#for#social#justice#(speculative).#
What(innovation(is(offered(for(program(evaluation(and(practice?(
Based#on#the#DCI#findings,#innovation#was#identified#in#the#following#areas.#
First,#the#READ#model#demonstrated#innovation#in#the#Nuclear#Futures#application#and#this#was#
closely#linked#to#its#originality,#efficacy#and#appropriateness#of#the#model#in#meeting#the#
requirements#of#the#evaluation#context.#This#included#methodological#innovation#that#fulfilled#the#
information#and#resource#needs#demanded#at#the#site#of#practice.#The#systematic#and#tailored#
framework#designed#for#the#Balaklava#project#used#an#innovative#suite#of#methods#drawn#from#the#
latest#academic#literature#to#substantiate#the#robust#integration#of#all#four#READ#strategies.#The#
outputs#that#were#produced#fulfilled#the#shortKterm#evaluation#and#documentation#needs#of#
partners.#The#strength#of#READ#as#a#learning#framework#complemented#the#evaluative#activities,#and#
had#significant#outcomes#for#the#continuous#improvement#of#our#evaluation#approach#and#arts#
practice,#and#in#building#the#evaluation#capacity#at#the#collective#level.##
The#second#demonstration#of#innovation#is#the#READ#infographic,#a#stylised#visual#representation#of#
the#READ#model,#which#is#presented#in#Section#6.3.3.#The#infographic#is#an#original#and#creative#
solution#to#the#challenge#of#communicating#the#READ#approach,#and#will#be#of#immediate#relevance#
at#the#site#of#practice.#By#encapsulating#a#large#volume#of#descriptive#information#about#the#model,#I#
plan#to#use#the#infographic#as#a#communication#tool#for#the#broader#dissemination#of#information#
about#READ#as#an#emergent#evaluation#approach,#in#both#industry#and#academic#contexts.#
Third,#is#the#innovative#approach#of#the#READ#model#for#establishing#processes#aimed#at#generating#
longKterm#outcomes,#and#to#create#and#disseminate#outputs#in#a#way#that#sustains#relevance#and#
accessibility#beyond#the#project#timeframe#and#site#of#practice.#Examples#include#READ’s#approach#
to#creative#documentation,#the#intent#to#unlock#the#academies,#and#production#and#public#sharing#of#
legacy#artefacts.#While#it#is#too#early#to#verify#the#developed#model’s#effectiveness#in#delivering#the#
longerKterm#and#deep#future#goals,#the#READ#approach#has#demonstrated#promise#for#this#in#the#
community#arts#context.#One#known#outcome#from#the#trial#is#that#the#pursuit#of#broader#learning#
and#legacy#goals#does#extend#the#management#and#partner#brokerage#demands#of#the#evaluation#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 193#
practice,#and#therefore#will#be#better#suited#to#mediumKtoKlong#term#community#arts#projects#rather#
than#shortKterm#activities.#
Fourth,#while#reflection,#evaluation,#analysis#and#documentation#are#known#and#used#in#varying#
ways#by#evaluators#and#creative#practitioners,#the#innovation#of#the#READ#model#lies#in#the#
integration#of#these#core#strategies#via#a#consolidated#framework.#The#model#brings#together#data#
generated#from#four#established#knowledgeKgenerating#and#appraisal#strategies#to#deliver#a#wide#
summative#lens#for#recording,#reviewing#and#representing#community#arts#activities.#Furthermore,#
the#model#has#the#flexibility#to#be#applied#as#an#integrated#or#modular#framework,#and#to#
accommodate#the#preferred#learning#modalities,#budgets#and#interests#of#stakeholders.#It#is#
imperative#here#to#note#that#in#the#READ#model,#reflection,#analysis#and#documentation#are#not#
merely#methods#of#an#evaluation#methodology,#but#are#independent#(yet#interKrelated)#learning#and#
appraisal#strategies#across#creative#practice,#management#and#evaluation#practice.#In#this#way,#the#
developed#READ#model#approaches#evaluation#as#part#of#an#integrated#practice.#
Lastly,#innovation#was#demonstrated#within#the#READ#strategies.#For#instance,#in#the#case#of#creative#
documentation,#the#READ#model#methodology#was#innovative#in#two#ways.#First,#it#drew#on#the#
strengths#of#the#creative#practice#of#the#project#and#second,#in#utilising#documentary#art#methods#it#
enabled#new#creative#content#and#artworks#to#be#generated#through#the#evaluation#practice.#Future#
utility#of#the#creative#content#is#already#planned#as#spinKoffs#beyond#the#Balaklava#project,#including#
plans#for#a#feature#length#documentary#film.#The#prospect#that#such#artefacts#can#be#generated#from#
the#combined#creative#and#evaluative#practice#of#READ#has#the#potential#to#raise#new#goals#for#CACD#
practitioners#interested#in#creating#longKterm#outcomes#through#community#arts#and#evaluation.##
7.3! Implications0and0significance0of0the0research0
In#this#section#I#examine#the#significance#of#the#research#and#findings#across#three#spheres#of#
evaluation#practice:#as#a#practitioner,#at#the#site#of#practice#and#for#the#industry#field.#
As(a(practitioner(
The#research#is#embedded#in,#and#furthers,#my#own#professional#evaluation#practice#advancing#my#
expertise#in#the#design#and#delivery#of#reflexive#learning#and#appraisal#systems#suited#to#the#creative#
arts#context.#The#DCI#offered#a#unique#opportunity#as#an#industry#practitioner#to#contextualise#and#
critically#examine#my#own#evaluation#practice.#Throughout#the#DCI#I#have#challenged#my#espoused#
assumptions#of#evaluation#and#experimented#with#new#approaches#and#methods.#I#have#also#
developed#my#theoretical#understanding#of#both#evaluation#and#program#management.#As#a#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 194#
consequence#of#the#research#enquiry,#my#evaluation#practice#has#transformed#and#advanced#by#
utilising#the#process#of#continuous#improvement#via#forensic#reflective#practice.#The#DCI#coursework#
has#also#had#implications#for#my#practice#through#skills#development#in#transmedia#design#and#digital#
storytelling,#and#both#have#been#incorporated#into#the#creative#approach#at#the#site#of#practice.#Of#
immediate#significance#is#the#impact#that#the#research#has#had#on#building#my#capacity#and#expertise#
to#deliver#evaluation#for#Alphaville#and#the#Nuclear#Futures#program,#and#my#ability#to#convey#new#
approaches#to#others#at#an#academic#level.#Overall,#the#new#expertise#and#experience#gained,#and#
the#resulting#evaluation#theory,#will#impact#the#quality#and#scope#of#my#future#evaluation#practice.#
Site(of(practice(
The#research#has#generated#transformative#impacts#at#the#site#of#practice,#and#the#outputs#
developed#through#the#research#process#–#including#the#clarification#of#the#program#logic,#the#
evaluation#plan,#the#READ#product#development#and#the#design#of#evaluation#instruments#and#so#on#
–#have#produced#myriad#benefits#for#host#organisation#Alphaville,#and#for#the#Nuclear#Futures#
program#and#partners.#Essentially#the#findings#have#produced#a#set#of#established#protocols#for#
evaluation#at#the#site#of#practice,#and#a#working#knowledge#of#the#baseline#steps#involved#in#
implementing#the#READ#approach.#The#findings#also#establish#an#important#foundation#for#the#
further#development#of#innovative#evaluation#practices#across#the#Nuclear#Futures#program,#
including#the#identification#of#the#model’s#strengths#and#constraints#critical#for#the#program’s#
summative#evaluation#in#2017.#At#the#site#of#practice,#the#successful#READ#approach#created#impact#
through#an#enhanced#understanding#of#the#artistic,#theoretical,#academic#and#the#wider#social#
implications#of#the#community#arts#practice,#and#promoted#engagement,#acceptance#and#
participation#in#evaluation#by#mixed#stakeholder#groups.##
Feedback#collected#from#Alphaville#stated#that#there#has#been#direct#and#significant#value#for#the#
organisation#from#the#research#throughout#the#DCI#period.#The#research#and#findings#will#also#be#
directly#relevant#for#the#evaluation#approach#to#be#implemented#during#the#final#evaluation#of#the#
initiative.#As#well#as#meeting#the#evaluation#needs#at#the#site#of#practice,#the#READ#application#
delivered#capacity#building#and#a#broad#range#of#learning#outcomes#for#Alphaville.#In#particular,#the#
culture#of#reflection#generated#amongst#the#creative#team#has#and#will#continue#to#extend#across#
other#projects,#and#the#community#of#practice#will#result#in#valuable#learning,#dissemination#and#
promotional#outcomes#for#the#organisation.#For#community#partners,#the#research#has#produced#a#
mix#of#creative#and#informationKbased#resources#that#encapsulate#the#stories,#artworks#and#
testimonies#from#the#project,#which#has#implications#for#future#campaigning#and#advocacy#work.#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 195#
Community(arts(evaluation(
While#the#DCI#findings#are#context#specific,#the#READ#product#developed#in#the#research#is#relevant#
for#practitioners#charged#with#the#task#of#evaluating#community#arts#practice#or#interested#in#
program#evaluation.#As#an#emergent#and#tested#approach,#the#developed#READ#model#contributes#
more#broadly#to#the#discourse#on#effective#approaches#to#community#arts#evaluation#and#offers#an#
original#and#relevant#model#of#evaluation#practice,#especially#in#the#domain#of#community#
partnerships.#For#practitioners,#the#research#speaks#to#individuals#such#as#myself,#who#traverse#
multiple#professional#roles#across#program#management,#creative#practice#and/or#evaluation.#In#
detailing#the#system’s#operations#and#nuances#of#the#READ#approach,#the#DCI#research#provides#
practical#guidance#for#an#integrated#approach#to#program#evaluation,#designed#for#multiple#and#
complex#creative#partnerships#initiatives.#In#drawing#from#the#documentary#art#traditions,#the#
research#also#demonstrates#the#potential#role#of#creative#practice#in#evaluation.#Although#arising#
from#the#community#arts#context,#the#research#has#implications#for#evaluation#practice#in#other#
fields,#including#the#creative#industries,#education#and#sustainability.#
In#addressing#identified#knowledge#gaps#on#effective#community#arts#evaluation,#the#research#
engages#with#the#complexities#of#evaluation#of#creative#partnership#initiatives,#delivering#an#original#
and#practical#contribution#that#will#have#impact#within#the#field.##For#the#field,#the#READ#model#
offers:#
•! an#integrated#evaluation#method#that#can#be#done#‘in#house’;#
•! a#model#of#evaluation#best#practice#that#embraces#the#ethos#of#community#arts;#•! a#flexible#framework#for#generating#robust#data,#that#is#linked#with#management#and#
creative#arts#practice#so#that#data#can#be#fed#back,#continuously,#into#the#project#for#realKtime#impact;#
•! original#strategies#for#engagement#and#for#developing#stakeholder#ownership#of#project#evaluation,#particularly#where#trepidation#or#resistance#to#traditional#evaluation#means#exists;#
•! possibilities#for#more#general#application#in#(researchKled)#creative#management.#
When#framed#within#the#broader#discourse#on#cultural#value#currently#unfolding#in#the#field,#the#
research#advances#the#pursuit#of#alternative#approaches#to#the#current#orthodoxy#of#topKdown,#
funding#agency#oriented#evaluation.#As#discussed#in#Chapter#2,#scholars#such#as#Goldbard#(2008,#
2015),#Belfiore#(2014)#and#Badham#(2015)#concerned#by#the#current#trends#and#blinkered#
approaches#to#community#arts#evaluation,#call#for#a#new#era#of#evaluation#practice#that#can#embrace#
wider#notions#of#cultural#value#and#operate#outside#of#the#constraints#of#quantitative#impact#
measurement#alone.#The#READ#model,#as#developed#in#this#research,#responds#to#such#appeals.#The#
application#of#evaluation#as#part#of#an#integrated#approach#to#learning#and#appraisal#is#inclusive#of,#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 196#
yet#not#limited#to,#meritKattribution#as#a#lens#for#understanding#and#representing#community#arts#
practice#and#impact.#In#addition,#the#developed#READ#model#embraces#practice#and#partnerKled#
knowledge#and#sensemaking#arising#from#creative#practice,#collective#reflection,#research#for#
broader#learning,#and#legacyKdriven#activities,#and#in#doing#so,#offers#new#rhetoric,#conceptualisation#
and#scope#for#evaluation#practice#in#community#arts.#
7.4! Dissemination0
The#two#audiences#that#dissemination#is#directed#to#are#industry#practitioners#(in#community#arts#
and#beyond)#and#academic#scholars#in#community#arts#and#evaluation.#Previous#and#futures#plans#
for#dissemination#are#provided#below.##
Academic((
Dissemination#of#early#findings#during#the#DCI#has#included:#
•! Conference#presentation:#Harnessing#evaluation#through#integration:#A#participatory#model#
for#Reflection,#Evaluation,#Analysis#and#Documentation#(the#‘READ’#model)#in#CommunityJ
based#Arts#by#E.#Barkley#at#the#Australasian#Evaluation#Society#Conference,#Darwin#10K12#September,#2014.#
•! Seminar#presentation:#Approaches#for#Reflection,#Evaluation,#Analysis#and#Documentation#by#E.#Barkley#at#Creative#Arts#and#Nuclear#Futures#Academic#Seminar,#The#Glasshouse,#QUT,#7#August#2015.##
Other#academic#events#and#forums#in#which#research#was#disseminated#with#industry#peers#and#
networks#include:#
•! Evaluation#workshop#to#CSIRO#social#research#teams#(as#part#of#a#professional#development#day)#in#Brisbane,#December#2013.#
•! Sharing#of#research#at#the#masterclass#for#postgraduate#students#prior#to#the#Spectres#of#Evaluation#conference#in#Footscray,#February#2014#
•! Networking#and#discussion#with#members#of#international#community#of#community#arts#workers#and#evaluators,#at#Spectres#of#Evaluation#conference#in#Footscray,#February#2014.#
#
Future#dissemination#opportunities#will#be#created#through#the#publication#of#papers#within#targeted#
journals.#One#example#is#a#planned#paper#for#submission#to#the#refereed#Journal#of#Arts#and#
Communities,#which#has#a#relevant#and#international#audience.#Another#example#is#the#planned#eK
book#discussed#below.#
#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 197#
Industry(dissemination((
Dissemination#of#selected#research#findings#has#been#undertaken#with#Nuclear#Futures#stakeholders/#
partners#(for#example,#via#partner#summaries#and#presentations).#During#the#research#I#have#
communicated#key#findings#at#the#site#of#practice#for#the#purpose#of#promoting#understanding#about#
the#READ#practice#(during#and#following#the#READ#application#research)#and#for#inspiring#action.#
Dissemination#to#the#Nuclear#Futures#partnership#and#stakeholder#audience#is#important#for#our#
continued#work#on#the#model#and#for#improving#practice#for#the#summative#application#of#READ#for#
the#whole#program.#Sharing#the#findings#and#examples#of#good#practice#of#the#READ#application#
from#the#Balaklava#experience#provides#information#and#helps#spread#ideas#that#are#relevant#beyond#
the#site#of#practice.#The#dissemination#methods#utilised#include#meeting#forums,#online#information#
via#the#Nuclear#Futures#website#and#partner#summaries#at#the#completion#of#the#research.#Given#the#
collaborative#process#involved#in#this#DCI#research,#it#will#be#important#to#deliver#results#back#to#the#
participants#and#stakeholders.#While#findings#from#READ#have#been#regularly#reported#to#the#
Australia#Council#for#the#Arts,#during#the#project,#a#more#comprehensive#presentation#on#the#
developed#READ#model#and#findings#is#planned#following#the#summative#evaluation#of#Nuclear#
Futures#in#2017.#
Beyond#the#dissemination#at#my#site#of#practice,#research#findings#have#also#been#circulated#with#
selected#industry#peers.#Dissemination#of#the#research#to#date#in#the#industry#arena#has#received#
positive#external#feedback#from#affiliated#industry#networks#and#peak#funding#agencies.##
eKbook(
For#future#communication#and#dissemination#of#the#research#I#plan#to#produce#and#8K10#page#eKbook#
that#introduces#and#describes#the#READ#model#and#approach.#The#publication#will#be#primarily#aimed#
at#creative#practitioners#and#evaluators.#Included#will#be#the#infographic#and#a#basic#‘how#to’#guide#
for#applying#the#integrated#approach.#
7.5! Future0research0
As#the#application#of#READ#is#relatively#recent#and#there#is#no#other#data#published#about#the#READ#
model,#it#is#important#to#expand#knowledge#on#this#subject.#Future#research#would#benefit#in#
developing#an#understanding#of#how#the#READ#model#can#best#be#designed#and#implemented#in#
other#community#arts#settings,#and#to#what#effect.#Subsequent#research#could#also#be#a#means#to#
innovate#the#methodological#designs#and#practices,#and#further#refine#the#systems#and#practice#of#
READ.#Keeping#pace#with#the#pending#Nuclear#Futures#applications#of#the#READ#model#it#would#be#of#
interest#to#examine#the#model#in#the#international#context.#Likewise,#application#in#other#fields#would#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 198#
also#provide#insight#into#the#functioning#and#potential#versatility#of#the#developed#evaluation#
approach.##
At#the#site#of#practice,#research#will#continue#into#the#READ#model’s#applications#across#the#Nuclear#
Futures#program,#with#the#research#questions#and#analysis#being#framed#across#a#number#of#
research#areas.#For#example,#in#a#planned#book#Australian#Art#on#the#Nuclear#Age:#Land,#Peoples,#
and#their#Futures,#to#be#written#by#Nuclear#Futures#program#collaborators,#I#will#examine#the#wider#
implications#of#the#program#revealed#by#the#READ#findings,#focusing#in#particular#on#the#legacy#
objectives#and#methods.#The#chapter#is#tentatively#titled#‘Making#the#story#go#far’.#The#second#
chapter,#which#I#will#coKauthor#with#projection#designer#Nic#Mollison,#has#the#prospective#title#
‘Staging#the#bomb’.#This#chapter#is#set#to#explore#the#implications#of#the#production#design#of#the#
immersive#projection#installation#developed#for#the#Balaklava#showcase,#and#investigates#the#role#
the#installation’s#innovative#audioKvisual#technologies#played#in#enhancing#the#community#arts#
practice#and#transforming#our#capacity#to#engage#with#community#about#the#nuclear#age.##
Finally,#there#would#be#benefits#derived#from#investigating#READ#to#understand#its#role#in#
organisational#learning#and#program#management,#without#evaluation#being#the#central#focus#of#the#
study#as#it#has#been#in#this#research.#Future#research#across#all#of#the#above#mentioned#areas#will#
increase#the#depth#of#details#and#descriptions#for#using#the#developed#READ#approach#across#varied#
contexts#and#for#many#potential#purposes.#
The#following#section#presents#the#overall#dissertation#conclusion.#
# #
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 199#
7.6! Dissertation0conclusion0
This#Doctor#of#Creative#Industries#(DCI)#research#has#focused#on#the#development,#operational#use,#
and#critical#review#of#the#READ#model#K#a#dynamic#and#rigorous#model#for#community#arts#evaluation.#
Designed#for#the#community#arts#context,#the#READ#model#extends#beyond#conventional#approaches#
to#integrate#Reflection,#Evaluation,#Analysis#and#Documentation#(READ),#to#create#a#wide#summative#
lens#for#community#arts#practice.#The#READ#approach#was#collaboratively#devised#at#the#site#of#my#
professional#practice#with#Alphaville#arts#company.#Prior#to#this#research,#the#proposed#READ#model#
existed#in#concept#form#only,#and#no#proof#of#concept#or#methodology#had#been#established.##
Through#the#DCI#my#role#as#researcher#has#involved#driving#the#twoKyear#process#of#developing#the#
READ#concept#into#to#a#substantiated,#working#model#in#the#industry#context.#I#have#achieved#this#by#
engaging#with#community#arts#partners#and#participants,#leading#collective#design#and#delivery#
processes,#articulating#and#refining#the#model,#and#evaluating#the#developed#READ#model#in#action.#
The#body#of#evidence#presented#in#this#dissertation#represents#the#first#iteration#of#critical#analysis#
on#the#model#during#the#initial#phase#of#the#Nuclear#Futures#Partnership#Initiative#(2013#K#2015),#an#
international#arts#program#funded#by#the#Australia#Council#for#the#Arts#(2014#K#2016)#and#the#site#of#
my#professional#practice.#Resulting#from#the#DCI#research#is#a#comprehensive#and#integrated#model#
of#practice#for#community#arts#evaluation,#as#well#as#a#practitionerKoriented#READ#product#for#future#
practice,#of#broader#relevance#to#the#field,#suited#to#nonKexperts#and#experts#in#evaluation.#
The#enquiry#fits#within#a#tradition#of#artists,#managers#and#evaluators#seeking#to#address#the#
persistent#epistemological#and#technical#problems#raised#by#evaluation,#in#the#face#of#rising#industry#
demands#to#demonstrate#the#value#of#community#arts#practice.#Within#the#field,#the#research#is#
positioned#within#the#unfolding#debate#on#cultural#value#and#measurement,#and#contributes#to#the#
discourse#on#effective#models#of#practice#for#community#arts#evaluation.#This#research#establishes#a#
verified#alternative#to#the#progressively#measurementKoriented#and#quantitative#approaches#gaining#
traction#in#the#field,#which#favour#economic#and#social#conceptualisations#of#cultural#value#and#lead#
to#the#further#instrumentalisation#of#the#arts.#Demonstrated#in#this#dissertation,#and#the#
accompanying#READ#in#Practice#portfolio,#are#the#diverse#accomplishments#of#the#wideKlens#READ#
approach#that#integrates#evaluation#as#part#of#a#broader#methodology#for#learning#and#appraisal.##
To#study#the#design,#efficacy#and#potential#of#the#READ#model#developed#in#this#research,#I#utilised#
forensic#reflective#practice#as#my#principal#research#strategy#for#the#DCI.#The#forensic#reflective#
practice#was#knowledgeKdriven,#involving#systematic#examination#and#critical#enquiry#of#the#model#
devised.#The#research#design#also#utilised#structured#and#reflexive#qualitative#methods,#including#
written#and#video#reflection,#semiKstructured#interviews,#digital#storytelling#and#observation.#The#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 200#
qualitative#research#methodology#elicited#rich#data#and#siteKspecific#learnings.#Of#significance#were#a#
number#of#key#findings#on#the#emergent#evaluation#practice#and#the#consolidated#READ#model.#
The#READ#model#developed#in#the#DCI#research#operated#as#a#flexible#and#robust#framework#that#
harnessed#evaluation#and#increased#the#scope#for#learning#and#appraisal#via#its#integrated#approach.#
The#READ#methodology#tailored#for#the#Balaklava#Community#Arts#Project#addressed#stakeholder#
identified#needs,#as#well#as#facilitated#a#collective#process#of#sensemaking#about#the#creative#
practice#and#broader#implications#of#the#community#arts.#The#mix#of#evaluative#and#nonKevaluative#
methods#utilised#across#the#READ#strategies#diversified#the#opportunities#for#data#collection,#
engagement#and#participation,#and#generated#rigorous#and#credible#findings#for#the#context#and#
needs#of#the#evaluation#process.#It#was#identified#that#the#READ#model#fulfilled#a#range#of#purposes,#
including#accountability,#improvement,#broader#learning#and#instituting#legacy#outcomes.##
In#terms#of#utility,#the#multiKplatform#portfolio#of#READ#outputs#produced#during#the#Balaklava#
application#were#highly#valued#by#stakeholders.#Immediate#and#longKterm#benefits#resulted#for#host#
organisation#Alphaville#and#for#others#within#the#partnerships.#The#findings#revealed#that#the#
developed#READ#approach#generated#transformative#impacts#at#the#Nuclear#Futures#site#of#practice#
and#advanced#evaluation#practice#for#the#collective.#In#shifting#the#focus#beyond#a#conventional#
evaluationKonly#lens,#and#through#the#incorporation#of#reflective#and#creative#methods#closely#linked#
into#the#arts#practice,#the#vibrancy#and#ethos#of#the#community#arts#practice#was#exemplified#in#the#
evaluation#practice.#Furthermore,#for#some#stakeholders#harbouring#trepidation#of#conventional#
evaluation,#the#diversified#READ#approach#allowed#prior#evaluation#constructs#to#be#challenged#and#
renegotiated,#so#that#evaluative#processes#and#activities#were#contextualised#and#legitimated#as#part#
of#an#accepted#broader#framework#for#artKmaking,#learning#and#appraisal#of#value#to#all#partners.##
The#research#demonstrates#that#the#innovation#of#the#emergent#READ#model#of#practice#lies#in#the#
integration#of#the#four#key#appraisal#and#learning#strategiesK#Reflection,#Evaluation,#Analysis#and#
Documentation.#For#the#field,#the#tested#READ#model#arising#from#this#research#offers:#
•! an#integrated#evaluation#method#that#can#be#done#‘in#house’;#
•! a#model#of#evaluation#best#practice#that#embraces#the#ethos#of#community#arts;#
•! a#flexible#framework#for#generating#robust#data,#that#is#linked#with#management#and#creative#arts#practice#so#that#data#can#be#fed#back,#continuously,#into#the#project#for#realKtime#impact;#
•! original#strategies#for#engagement#and#for#developing#stakeholder#ownership#of#project#evaluation,#particularly#where#trepidation/#resistance#to#traditional#evaluation#means#exists;#
•! possibilities#for#more#general#application#in#(researchKled)#creative#management.#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 201#
Specific#areas#demonstrating#innovation#in#evaluation#practice#were#evidenced,#including#the#use#of#
creative#documentation#and#original#creative#work#for#evaluation#purposes,#as#well#as#the#utilisation#
of#long#term#goals#and#broader#learning#outcomes#to#enhance#the#benefits#returned#to#community#
partners.#The#prospect#of#the#developed#READ#approach#being#able#to#address#longKterm#objectives#
arising#from#creative#practice#has#the#potential#to#raise#new#goals#for#CACD#practitioners#interested#
in#creating#diverse#legacy#outcomes#through#community#arts.#In#addition,#the#model#demonstrated#a#
capacity#to#produce#organisational#and#broader#learning#outcomes#through#the#established#
community#of#practice#so#as#to#‘unlock#the#academies’,#and#via#a#sustained#culture#of#reflection#by#
artists#and#community#partners#K#that#has#carried#on#across#subsequent#Nuclear#Futures#projects.#
Methodologically,#the#READ#model#demonstrates#innovation#as#an#original,#versatile#and#adaptable#
framework,#able#to#be#used#by#practitioners#as#either#an#integrated#or#modular#evaluation#approach.#
One#result#of#integrating#the#four#established#strategies#(Reflection,#Evaluation,#Analysis#and#
Documentation)#was#that#the#frame#for#evaluation#practice#was#widened#at#the#site#of#practice#and#
this#extended#READ’s#applicability#as#a#knowledgeKdriven#framework#for#learning#and#appraisal.#The#
broader#implication#is#that#the#integrated#approach#offers#the#model#a#reflexive#edge#in#meeting#
multiple#needs#as#defined#by#the#context,#partnerships#and#arts#practices#under#review.#It#also#
means#that#the#scale#and#scope#of#future#READ#applications#can#range#greatly,#from#creative#
practitioners#adapting#the#READ#approach#for#their#individual#arts#practice#to#application#by#an#
evaluation#team#across#large#complex#multiKarts#creative#partnerships.#The#capacity#of#the#model#to#
be#inclusive#of#different#learning#approaches#and#knowledge#paradigms#is#also#significant,#given#the#
role#evaluation#practice#plays#in#brokering#partnerships,#which#are#often#culturally#diverse.##
In#the#dissertation,#the#impact#of#the#research#is#evidenced#across#three#spheres#of#evaluation#
practice:#as#a#practitioner,#at#the#site#of#practice#and#for#the#industry#field.#In#my#own#professional#
evaluation#practice,#the#research#has#advanced#my#expertise#in#the#design#and#delivery#of#reflexive#
learning#and#appraisal#systems#suited#to#the#creative#arts#context#and#provides#a#valuable#evaluation#
tool#for#future#practice.#At#the#site#of#practice,#the#evaluation#practice#and#READ#model#developed#in#
the#research#created#impact#by#generating#an#enhanced#understanding#of#the#artistic,#theoretical,#
academic#and#wider#social#implications#of#the#community#arts#practice.#The#research#findings#also#
establish#an#important#foundation#for#the#further#development#of#innovative#evaluation#practices#
across#the#Nuclear#Futures#program,#including#the#identification#of#the#model’s#strengths#and#
constraints#critical#for#the#program’s#summative#evaluation#in#2017.#For#the#field,#the#DCI#research#
provides#a#timely#contribution#to#the#discourse#on#effective#evaluation#of#community#arts#by#
demonstrating#a#number#of#original#developments#in#evaluation#practice#that#respond#to#calls#from#
within#the#field#to#diversify#approaches#to#community#arts#evaluation.#Practical#insights#into#an#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 202#
integrated#stakeholderKoriented#approach#to#evaluation#are#revealed,#as#well#as#demonstration#of#
how#documentary#art,#collective#reflection,#community#archiving,#and#academic#enquiry#can#offer#
adept#methods#for#understanding,#appraising#and#representing#community#arts#practice.##
In#summing#up,#the#findings#within#this#dissertation#identify#a#number#of#key#areas#of#innovation#for#
evaluation#practice,#particularly#for#the#evaluation#of#complex#creative#partnershipKoriented#
programs.#The#READ#model#developed#and#presented#in#the#DCI#demonstrates#innovation#in#the#
originality,#efficacy#and#appropriateness#of#the#evaluation#methods#and#conduct#that#it#brought#to#
delivering#the#evaluation#needs#at#the#site#of#practice.#Of#particular#interest#were#the#benefits#
resulting#from#the#use#of#creative#documentation#and#the#inclusion#of#legacy#goals#as#a#longKterm#
purpose#for#evaluation.#Both#of#these#examples#drew#on#the#strengths#of#the#creative#practice#and#
partners#of#the#project,#and#extended#the#scope#of#evaluation#practice,#resulting#in#original#artefacts#
and#creative#works#that#are#of#relevance#to#partners#beyond#the#evaluation#phase.#As#a#body#of#
work,#the#DCI#findings,#the#developed#READ#model,#and#the#consolidated#READ#product,#contribute#
original#knowledge#to#the#field#and#to#the#discourse#on#the#effective#evaluation#of#community#arts#
and#impact#measurement.#For#my#own#professional#practice,#the#research#has#transformed#my#
approach#to,#and#definition#of,#evaluation,#and#elucidated#the#benefits#of#cultivating#interKlinked#
opportunities#for#learning#and#appraisal#across#the#domains#of#creative#practice,#management,#and#
evaluation.#At#the#site#of#practice,#the#research#represents#a#genuine#effort#to#engage#with#the#
complexities#of#community#arts#evaluation#in#a#practical#way,#and#has#resulted#in#improved#
evaluation#practice#and#the#establishment#of#protocols#for#the#future#use#of#the#READ#model.#
Beyond#the#site#of#practice,#the#findings#have#broader#relevance#and#will#be#of#interest#to#scholars#
and#practitioners#in#community#arts,#the#creative#industries,#and#other#fields#with#an#interest#in#
program#evaluation,#artsKbased#and/or#stakeholderKoriented#evaluation,#community#engagement,#
organisational#learning,#and#program#management#more#generally.#
#
*************************#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 203#
Appendix#1## Interview#design#for#scoping#interviews###Example:0Interview0Design0Schedule0
Code0 Code0category0 Aim0of0Data0 Questions0to0generate0data0 Relevance0Partner0000000Industry0
Notes0for0analysis0
StakHs0 Stakeholder0profiles0
Gather0data0on0background0of0participant0and0positioning0within0CBA0stakeholder0groupings0
Could0you0please0briefly0describe0the0field0that0you0work0in0and0the0type0of0work0you0do?0(i.e.0your0current0role)0What0 type0 of0 organization0 do0 you0 work0for?0
*0 *0 Participants0to0be0analysed0in0terms0of0stakeholder0group0and0as0individuals.0Analysis0may0be0relevant0relating0to0years0of0experience0and0evaluation0lessons0etc.0
How0long0have0you0worked0in0this0area?0 *0 *0
EvUse0 Evaluation0Uses0 Gather0a0multiple0perspective0snapshot0of0evaluation0uses0and0dissemination0
What0 is0 your0 experience0 in0 participating0in,0 implementing,0 reviewing0 or0 using0evaluation?0
*0 *0 Provides0a0gauge0to0participant’s0experience0with0evaluation0
Are0you0involved0with0evaluation0practice0in0your0current0role?00If0so,0what0type0of0evaluative0activities0do0you0do?0If0not,0how0does0your0organization0use0or0implement0evaluation?0
*0 *0 Analysis0on0dominant0evaluation0uses0and0dissemination0relevant0to0participants0
Does0 your0 organization0 (or0 yourself)0disseminate0 or0 share0 evaluation0 findings0to0other0practitioners0or0organisations?0If0so,0how0and0why?0
*0 *0
EvDef0 Evaluation0Definitions0
Gather0a0multiple0perspective0snapshot0of0evaluation0definitions0
How0do0you0define0evaluation?0What0is0the0purpose0of0evaluation?0
*0 *0 Analysis0aims0to0identify0the0dominant0underlying0values,0assumptions0and0constructs0of0CBA0evaluation0by0key0stakeholder0groups0to0provide0insight0into0the0partnership0context0of0CCPI0and0CBA.0
Does0 this0 definition0 reflect0 how0 your0organization0defines0evaluation?0
*0 *0 Examine0if0there0is0conflict0or0difference0in0personal0vs0organizational0agendas.0
EvVal0 Evaluation0Value0
Identify0what0participants0value0about0evaluation00
What0 is0 your0 personal0 opinion0 of0 the0value0of0evaluation?0
*0 *0 Analysis0of0dominant0uses,0values0and0barriers0of0CBA0evaluation0by0key0stakeholder0groups0to0provide0insight0into0the0functionality0points0for0NFs0evaluation0design00
How0 does0 your0 organization0 view0evaluation0practice0do0you0think?0
*0 *0
What0are0some0of0the0negative0aspects0of0evaluation?0
*0 *0
Beyond0 the0 evaluation0 findings0themselves,0 what0 other0 benefits0 does0evaluation0offer,0if0any?0
0 *0
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 204#
Code0 Code0category0 Aim0of0Data0 Questions0to0generate0data0 Relevance0Partner0000000Industry0
Notes0for0analysis0
EvCBA0 Evaluation0Community)based0Arts0
Gather0a0multiple0perspective0snapshot0of0evaluation0within0the0Community)based0arts0context;00Identify0what0principles0or0factors0make0for0‘good’0evaluation0
Do0you0think0 it0 is0possible0 to0successfully0evaluate0 Community)based0 Arts0 projects0or0 initiatives0 (or0 CCPI0 or0 relevant0language)?0
*0 *0 Analysis0of0context0specific0issues0for0undertaking0evaluation,0may0be0approached0by0stakeholder0groupings0or0theme0based0analysis.0
What0do0 you0 consider0 to0be0 the0barriers0to0 successful0 evaluation0 in0 Community)based0Arts?0
*0 *0
What0 aspects0 of0 community)based0 arts0are0 important0 to0 ‘measure’0 or0 evaluate?00Why?0
*0 *0
Are0 evaluation0 reports0 from0 other0Community)based0 Arts0 projects0 of0interest0to0you?00Do0you0get0 an0opportunity0 to0 learn0 from0other0projects0via0evaluations?0
0 *0
What0do0you0think0are0key0elements0that0you0 think0 makes0 a0 ‘good’0 evaluation0 in0the0Community)based0Arts0context?0
*0 *0 Analysis0for0building0criteria0for0successful0design0(can0be0analysed0in0terms0of0meta)evaluation0scholarship)0
PartEv0 Participatory0Evaluation0
Identify0dominant0definitions0and0levels0of0experience/0familiarity0of0participants0
Have0 you0 heard0 of0 ‘participatory0evaluation?’00If0 so,0 what0 is0 your0 understanding0 of0 this0approach?0If0not,0define0participatory0evaluation0
*0 *0 Analysis0of0key0features0or0examples0provided0of0participatory0approaches0and0levels0of0experience0with0participatory0approaches0by0partners0
Have0 you0 been0 involved0 in0 evaluation0before0 that0 used0 participatory0approaches?0If0 so,0what0were0 the0 key0 benefits0 of0 this0type0of0approach?0If0 so,0 what0 were0 the0 issues0 or0disadvantages0 arising0 from0 using0 a0participatory0approach?0
*0 *0 Looking0for0evidence/0links0between0participatory0approaches0and0benefits0(or0potential0benefits)0to0the0program0that0will0assist0in00theory0development0on0participatory0approaches0given0constraints0of0CBA0projects0
What0do0you0think0the0benefit0might0be0for0using0a0participatory0approach0for0a0program0like0NFs?00Is0there0any0evidence0of0this0so0far?0
*0 0
What0do0you0think0the0benefit0might0be0for0using0a0participatory0approach0for0Community)based0arts0or0CCPI0programs?0Do0you0know0of0examples0or0case0studies0where0this0has0been0the0case?0
0 *0 Analysis0of0potential0strengths0and0constraints0of0participatory0approaches0for0particular0elements0or0complexities0within0the0CCPI0case0study0
0
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 205#
Bibliography0
Alkin,#M.#C.#(2004).#ContextKadapted#utilization:#A#personal#journey.#In#M.#C.#Alkin#(Ed.),#Evaluation#Roots:#Tracing#theorist’s#view#and#influences#(pp.#293–303).#Thousand#Oaks,#CA:#Sage.#
Alphaville.#(2015).#Nuclear#Futures:#Exposing#the#legacies#of#the#atomic#age#through#creative#arts.#[Website].#Retrieved#February#13,#2015,#from#http://nuclearfutures.org.#
Alvesson,#M.#&#Skoldberg,#K.#(1999).#Reflexive#Methodology:#new#vistas#for#qualitative#research.#London:#Sage.##
Anderson,#L.#W.,#&#Postlethwaite,#T.#N.#(2007).#Program#evaluation:#LargeKscale#and#smallKscale#studies.#International#Institute#for#Educational#Planning#Series#Booklet,#(8).#
Archer,#B.#(1995).#The#Nature#of#Research.#CoJdesign,#January#issue,#pp.#6K13.#[Paper#transcribed#from#a#photocopy#of#the#original#by#Chris#Rust#on#17.1.09.]#Retrieved#April#20,#2014#from#http://www.metu.edu.tr/~baykan/arch586/Archer95.pdf#(no#longer#available).#See#alternative#link,#https://archive.org/stream/TheNatureOfResearch/Archer1995Codesign#page/n0/mode/2up##
Arvanitakis,#J.#(2008).#Staging#Maralinga#and#Looking#for#Community#(Or#Why#We#Must#Desire#Community#Before#We#Can#Find#It).#Research#in#Drama#Education,#13(3),#295K306.#
Australasian#Evaluation#Society#(AES).#(2010).#Guidelines#for#the#Ethical#Conduct#of#Evaluations.#[Booklet].#Retrieved#May#20,#2013#from#http://www.aes.asn.au/images/stories/files/membership/AES_Guidelines_web.pdf##
The#Australia#Business#Arts#Foundation#(AbaF)#&#Australia#Council#for#the#Arts.#(2010).#Arts#and#business:#partnerships#that#work#[Factsheet].#Sydney.#Retrieved#May,#5,#2013#from#
http://www.creativepartnershipsaustralia.org.au#(no#longer#available).#See#full#report,#http://apo.org.au/resource/artsKandKbusinessKpartnershipsKwork##
Australia#Council#for#Arts.#(2011).#Community#Partnerships#sector#plan#2010J2012#for#community#arts#and#
cultural#development#(revised#version#May#2011).#Retrieved#May,#15,#2013#from#http://www.australiacouncil.gov.au/__data/assets/pdf_file/0018/103482/COMMUNITY_PARTNERSHIPS_2011_Sector_Plan.pdf#(no#longer#available).#See#alternative#link,#http://static.placestories.com/pool/project/0001/0008319/docs/docK2Z.pdf##
Australian#Government.#(2000).##Does#Your#Project#Make#a#Difference?#A#Guide#to#evaluating#environmental#education#
projects#and#programs.#Sydney:#Department#of#Environment#and#Conservation#(NSW).#
Babbie,#E.#(1992).#The#Practice#of#Social#Research#(6th#ed.).#Belmont:#Wadsworth#Publishing#Company.#
Badham,#M.#(2010).#The#case#for#‘socially#engaged#arts’:#navigating#art#history,#cultural#development#and#arts#funding#narratives.#Local#Global:#Identity,#Security,#Community,#7,#84K99.#
Badham,#M.#(2015).#Democratising#Cultural#Indicators:#Developing#a#Shared#Sense#of#Progress.#In#L.MacDowall,#M.Badham,#E.Blomkamp,#&#K.Dunphy#(Eds.).#Making#Culture#Count:#The#Politics#of#
Cultural#Measurement#(pp.195K213).#London:#Palgrave#Macmillan.##
Bambino,#D.#(2002).#Critical#friends.#Educational#leadership,#59(6),#25K27.#
Becker,#H.#S.#(1996).#The#Epistemology#of#Qualitative#Research.#In#R.Jessor,#A.Colby#&#R.#A.Shweder#(Eds.),#Ethnography#and#Human#Development:#Context#and#Meaning#in#Social#Inquiry#(p.530).#Chicago:#University#of#Chicago#Press.##
Belenky,#M.,#Clinchy,#B.,#Goldberger,#N.,#and#Tarule,#J.#(1986).#Women’s#Ways#of#Knowing.#New#York:#Basic#Books.#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 206#
Belfiore,#E.#(2014).#‘Impact’,#‘value’#and#‘bad#economics’:#Making#sense#of#the#problem#of#value#in#the#arts#and#humanities.#Arts#&#Humanities#in#Higher#Education,#14(1),#95–110.#
Belfiore,#E.#&#Bennett,#O.#(2008).#The#Social#Impact#of#the#Arts:#An#Intellectual#History.#New#York:#Palgrave#Macmilllan.#
Belfiore,#E.#&#Bennett,#O.#(2010).#Beyond#the#Toolkit#Approach:#Arts#Impact#Evaluation#Research#and#the#realities#of#Cultural#PolicyKmaking.#Journal#for#Cultural#Research#14,#2#(2010),#121K142#from#http://dx.doi.org/10.1080/14797580903481280.#
BetterEvaluation.#(2013).#Rainbow#Framework#Overview.#[website]#Retrieved#September#4,#2013#from:#www.betterevaluation.org/plan###
Big#hART.#(2010).#Ngapartji#Ngapartji#Memory#Basket.#[Boxed#creative#documentation#with#information#and#artefact#resources].#Available#via#Big#hART#organisation.#
Blichfeldt,#B.#S.,#&#Andersen,#J.#R.#(2006).#Creating#a#wider#audience#for#action#research:#Learning#from#caseKstudy#research.#Journal#of#Research#Practice,#2(1),#ArticleKD2.#
Blomkamp,#E.#(2015).#A#Critical#History#of#Cultural#Indicators.#In#L.MacDowall,#M.Badham,#E.Blomkamp,#&#K.Dunphy#(Eds.).#Making#Culture#Count:#The#Politics#of#Cultural#Measurement#(pp.11K26).#London:#Palgrave#Macmillan.##
Bolton,#G.#(2010).##Reflective#Practice:#Writing#&#Professional#Development#(3rd#ed.).#London:#Sage.##
Boud,#D.#(2001).#Using#Journal#Writing#to#Enhance#Reflective#Practice.#New#Directions#For#Adult#and#Continuing#Education,#90#(Summer),#9K17.##
Boud,#D.,#&#Walker,#D.#(1998).#Promoting#reflection#in#professional#courses:#The#challenge#of#context.#Studies#in#higher#education,#23(2),#191K206.#
Bourner,#T.#(2003).#Assessing#reflective#learning.#Education+#Training,#45(5),#267K272.#
Braun,#V.,#&#Clarke,#V.#(2006).#Using#thematic#analysis#in#psychology.#Qualitative#research#in#psychology,#3(2),#77K101.#
Brown,#P.F.#(2006).#Maralinga:#Theatre#from#a#Place#of#War.#In#G.McAuley#(Ed.),#Ground#Work:#The#
Politics#of#Place#and#Performance.#Berne:#Peter#Lang.#
Brown,#P.F.#(2007).#British#nuclear#testing#in#Australia:#Performing#the#Maralinga#Experiment#through#verbatim#theatre.#Journal#of#the#Royal#Society#of#New#South#Wales,#139,#39K50.#
Brown,#P.F.#(Ed.).#(2010).#Verbatim:#staging#memory#and#community.#Sydney:#Currency#Press.#
Brown,#P.F.#and#the#Maralinga#Research#Group.#(2006).#Half#a#Life#[Stage#Play#based#on#testimonies#of#Australian#and#British#Nuclear#Test#Veterans].#Available#from#Alphaville#arts#organisation.#
Burgess,#J.#(2006).#Hearing#ordinary#voices:#Cultural#studies,#vernacular#creativity#and#digital#storytelling.#Continuum:#Journal#of#Media#&#Cultural#Studies,#20(2),#201K214.#
Burgess,#J.#&#Klaebe,#H.#(2009).#Digital#storytelling#as#participatory#public#history#in#Australia.#In#J.Hartley#&#K.McWilliam,#(Eds.)#Story#Circle:#Digital#Storytelling#Around#the#World#(pp.#155K166).#Chichester,#UK;#Malden,#MA:#WileyKBlackwell.##
Burnaford,#G.#(2006).#Moving#Toward#a#Culture#of#Evidence:#Documentation#and#Action#Research#inside#
CAPE#Veteran#Partnerships#[Veteran#Partnerships#Report#2005K6].#Retrieved#May,#21,#2013#from#http://www.capeweb.org/##
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 207#
Butterfoss,#F.#D.,#Francisco,#V.,#&#Capwell,#E.#M.#(2001).#Stakeholder#participation#in#evaluation.#Health#Promotion#Practice,#2(2),#114K119.#
Campbell,#D.#T.,#Stanley,#J.#C.,#&#Gage,#N.#L.#(1963).#Experimental#and#quasiJexperimental#designs#for#research#(No.#04;#Q175,#C3.).#Boston:#Houghton#Mifflin.#
Carter,#P.#(2004).#Material#thinking:#The#theory#and#practice#of#creative#research.#Melbourne#University#Press.#
Carter,#P.#(2010).#Symposium:#‘The#Artist#as#Researcher’#[Recorded#lecture].#Royal#Academy#of#the#Art#Conference,#The#Hague#(5K6th#February,#2010).#Retrieved#November#20,#2015#from#https://vimeo.com/23632430##
Centre#for#Digital#Storytelling.#(2014).#How#it#all#began.#[Website].#Retrieved#January#4,#2014#from#http://storycenter.org/history/#(no#longer#available).#See#‘Our#Story’#on#alternative#link,#http://www.storycenter.org/press/##
Charmaz,#K.#(2000).#Grounded#Theory:#Objectivist#and#Constructivist#methods.#In#N.K.Denzin#&#Y.S.Lincoln#(Eds.),#Handbook#of#Qualitative#Research#(2nd#ed.)#(pp.509K535).#Thousand#Oaks,#CA:#Sage.#
Charmaz,#K.#(2006).#Constructing#Grounded#Theory:#A#practical#guide#through#Qualitative#Analysis.#Newbury#Park,#CA:#Pine#Forge#Press.#
Christie,#C.#A.,#&#Alkin,#M.#C.#(2008).#Evaluation#theory#tree#reKexamined.#Studies#in#Educational#Evaluation,#34(3),#131K135.#
Clarke,#A.#(1999).#Evaluation#Research.#An#introduction#to#principle,#methods#and#practice.#London:#Sage.#
Clements,#P.#(2007).#The#evaluation#of#community#arts#projects#and#the#problems#with#social#impact#methodology.#International#Journal#of#Art#&#Design#Education,#26(3),#325K335.#
Couldry,#N.#(2008).#Digital#storytelling,#media#research#and#democracy.#In#K.Lundby#(Ed.),#Digital#storytelling,#mediatized#stories:#SelfJrepresentations#in#new#media#(pp.#42K60).#New#York:#Peter#Lang.#
Cousins,#J.#B.,#&#Earl,#L.#M.#(1995).#The#case#for#participatory#evaluation:#Theory,#research,#practice.#In#J.B.#Cousins#&#L.M.#Earl#(Eds),#Participatory#evaluation#in#education:#Studies#in#evaluation#use#and#organizational#learning#(pp.#3K18).#London:#The#Falmer#Press.#
Cousins,#J.#B.,#&#Whitmore,#E.#(1998).#Framing#participatory#evaluation.#New#directions#for#evaluation.#(80),#5K23.#
Cox,#A.#(2005).#What#are#communities#of#practice?#A#comparative#review#of#four#seminal#works.#Journal#of#Information#Science,#31(6),#527–540.##
Cresswell,#J.#W.#(1994).#Qualitative#and#quantitative#approaches.#Qualitative#and#quantitative#approaches.#New#York:#Sage.#
Cresswell,#J.W.,#&#Clark,#V.#L.#P.#(2007).#The#Nature#of#Mixed#Methods#Research.#In#J.W.Cresswell#&#V.L.P.Clark#(Eds.),#Designing#and#Conducting#Mixed#Methods#Research#(pp.1K19).#Thousand#Oaks,#CA:#Sage.#
Cross,#R.,#&#Hudson,#A.#(2006).#Beyond#belief:#the#British#bomb#tests:#Australia's#veterans#speak#out.#Adelaide:#Wakefield#Press.##
Curthoys,#A.#(1991).#Unlocking#the#Academies:#Responses#and#Strategies#[online].#Meanjin,#50(2/3),#WinterKSpring#1991:#386K393.#Retrieved#December#4,#2015,#from#http://search.informit.com.au.ezp01.library.qut.edu.au/documentSummary;dn=136325485529146;res=IELLCC##
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 208#
Darabi,#A.#(2002).#Teaching#Program#Evaluation:#Using#a#Systems#Approach.#American#Journal#of#Evaluation,#23#(June),#219K228.#
Dart,#J.#&#Davies,#R.#(2003).#A#dialogical,#storyKbased#evaluation#tool:#The#Most#Significant#Change#Technique.#American#Journal#of#Evaluation,#24(2),#137–55.#
Davidson,#E.#J.#(Ed.).#(2005).#Evaluation#methodology#basics:#The#nuts#and#bolts#of#sound#evaluation.#London:#Sage.#
Davidson,#E.#J.#(2010).#Actionable#evaluation#for#realKworld#decision#makers.#In#10th#Australasian#Higher#Education#Evaluation#Forum,#[Proceedings].#November#2010,#Lincoln:#New#Zealand.#
Davidson,#P.,#&#Rowe,#J.#(2009).#Systematising#knowledge#management#in#projects.#International#Journal#of#Managing#Projects#in#Business,#2(4),#561K576.#
Davies,#R.#&#Dart,#J.#(2005).#The#Most#Significant#Change#(MSC)#Technique:#A#guide#to#its#use.#Retrieved#20.5.13#from#http://www.mande.co.uk/docs/MSCGuide.pdf##
Denzin,#N.K.#&#Lincoln,#Y.S.#(Eds.).#(2011).#The#SAGE#handbook#of#qualitative#research.#Thousand#Oaks,#CA:#Sage.#
Dewey,#J.#(1933).#How#we#think:#A#restatement#of#the#relation#of#reflective#thinking#to#the#educational#process.#Lexington,#MA:#Heath.#
DiCiccoKBloom,#B.,#&#Crabtree,#B.#F.#(2006).#The#qualitative#research#interview.#Medical#education,#40(4),#314K321.#
Dobson,#P.#J.#(1999).#Approaches#to#theory#use#in#interpretive#case#studies–a#critical#realist#perspective.#In#Australasian#Conference#on#Information#System,#Wellington,#New#Zealand.#Retrieved#September#20,#2013#from#http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.10.722&rep=rep1&type=pdf##
Doloughan,#F.#J.#(2002).#The#language#of#reflective#practice#in#art#and#design.#Design#Issues,#18(2),#57K64.#
Donaldson,#S.#I.,#Christie,#C.#A.,#&#Mark,#M.#(Eds.).#(2008).#What#Counts#as#Credible#Evidence#in#Evaluation#and#
EvidenceJbased#Practice?#Thousand#Oaks,#CA:#Sage.##
Donovan,#C.#(2012).#Priceless?#A#blog#on#the#very#idea#of#measuring#cultural#value,#Interactive#media#blog#on#Department#of#Culture,#Media#and#Sport,#United#Kingdom.#Retrieved#August#20,#2012#from#http://blogs.culture.gov.uk/main/2012/01/welcome_to_the_priceless_blog.html#(no#longer#available).#See#alternative#link#http://dcmsblog.uk/2012/01/welcomeKtoKtheKpricelessKblog/##
Drotner,#K.#(2008).#Boundaries#and#bridges:#Digital#storytelling#in#education#studies#and#media#studies.#In#K.#Lundby#(Ed.),#Digital#storytelling,#mediatized#stories:#SelfJrepresentations#in#new#media#(pp.#61K84).#Berne:#Peter#Lang.#
Dunphy,#K.#(2015).#A#Holistic#Framework#for#Evaluation#of#Arts#Engagement.#In#L.MacDowall,#M.Badham,#E.Blomkamp,#&#K.Dunphy#(Eds.).#Making#Culture#Count:#The#Politics#of#Cultural#
Measurement#(pp.243K263).#London:#Palgrave#Macmillan.##
Dunn,#A.#(2006).#Community#Partnerships#Scoping#Study:#Creative#Communities#[Commissioned#report]#Retrieved#May#22,#2013#from#http://www.australiacouncil.gov.au/__data/assets/pdf_file/0004/33952/cp_scoping_study.pdf#(no#longer#available).#See#alternative#link#http://static.placestories.com/pool/project/0001/0007950/docs/docK2E.pdf##
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 209#
Dunst,#C.#J.,#Trivette,#C.#M.,#&#LaPointe,#N.#(1992).#Toward#clarification#of#the#meaning#and#key#elements#of#empowerment.#Family#Science#Review,#5(1K2),#111–130.#
Edenloff,#J.#(2011).#ArtsJBased#Evaluation#Tools#for#Community#Arts#Programs:#A#Case#Study#of#Art#City’s#
‘Green#Art’#in#Winnipeg,#Manitoba#(Doctoral#dissertation,#University#of#Manitoba).#
Ezzy,#D.#(2006).#The#Research#Process.#In#M.#Walter#(Ed.),#Social#Research#Methods:#An#Australian#
Perspective#(pp.30K51).#Melbourne:#Oxford#University#Press.##
Fetterman,#D.#M.,#&#Wandersman,#A.#(Eds.).#(2005).#Empowerment#evaluation#principles#in#practice.#New#York:#Guilford#Press.#
Finlay,#L.#(2008).#Reflecting#on#‘Reflective#practice’#[PracticeKbased#Professional#Learning#Centre#paper#52.#A#discussion#paper#prepared#for#PBPL#CETL].#Retrieved#June#5,#2013,#from#http://www.open.ac.uk/opencetl/resources/pbplKresources/finlayKlK2008KreflectingKreflectiveKpracticeKpbplKpaperK52##
Fitzpatrick,#J.#L.,#Sanders,#J.#R.,#&#Worthen,#B.#R.#(2012).#Program#evaluation:#Alternative#approaches#and#practical#guidelines#(4th#ed.).#Upper#Saddle#River,#NJ:#Pearson.#
#Fonow,#M.,#and#Cook,#J.#(Eds.).(1991).#Beyond#Methodology:#Feminist#Scholarship#as#Lived#Research.#
Bloomington:#Indiana#University#Press.##
Friedman,#A.#(Ed.).#(2008).#Framework#for#Evaluating#Impacts#of#Informal#Science#Education#Projects#[OnKline],#p.#36.#Retrieved#12.12.13,#from#http://insci.org/resources/Eval_Framework.pdf#(no#longer#available).#See#alternative#link#http://www.informalscience.org/sites/default/files/2008_PI_Summit_Plenary_Framework_Introduction.pdf##
Gattenhof,#S.,#&#Seffrin,#G.#(2014).#Evaluation#of#the#Australian#Performing#Arts#Market#2014K2018KYear#One#Report.#
Geursen,#G.,#&#Rentschler,#R.#(2003).#Unraveling#cultural#value.#The#Journal#of#Arts#Management,#Law,#and#
Society,#33(3),#196K210.##
Gibbs,#G.#(1988).#Learning#by#Doing:#A#Guide#to#Teaching#and#Learning#Methods.#London:#Further#Education#Unit.#Retrieved#June#8,#2013#from:#http://www2.glos.ac.uk/gdn/gibbs/index.htm##
Glaser,#B.G#&#Strauss,#A.L.#(2009).#The#discovery#of#grounded#theory:#Strategies#for#qualitative#research.#New#Brunswick,#USA:#Transaction#Books.#
Goldbard,#A.#(1993).#Postscript#to#the#past:#Notes#toward#a#history#of#community#arts.#High#Performance,#16(4),#23K27.#
Goldbard,#A.#(2006).#New#Creative#Community:#The#Art#of#Cultural#Development.#Oakland:#New#Village#Press.#
Goldbard,#A.#(2008).#The#Metrics#Syndrome#[Online#article].#Retrieved#May#23,#2013#from#http://arlenegoldbard.com/wpKcontent/uploads/2005/12/MetricsKSyndromeK10K13K08.pdf#!
Goldbard,#A.#(2015).#The#Metrics#Syndrome:#Cultural#Scientism#and#Its#Discontents.#In#L.MacDowall,#M.Badham,#E.Blomkamp,#&#K.Dunphy#(Eds.).#Making#Culture#Count:#The#Politics#of#Cultural#
Measurement#(pp.214K227).#London:#Palgrave#Macmillan.##
Gray,#C.#(1996).#Inquiry#through#practice:#developing#appropriate#research#strategies.#No#Guru,#No#Method.##UIAH#Helsinki.#Retrieved#April#1,#2013,#from#http://carolegray.net/Papers%20PDFs/ngnm.pdf##
Greene,#J.#C.#(2005).#A#valueKengaged#approach#for#evaluating#the#BuncheKDa#Vinci#learning#academy.#New#Directions#for#Evaluation,#106,#27–47.�#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 210#
Greenwood,#J.#(1993).#Reflective#practice:#a#critique#of#the#work#of#Argyris#and#Schon.#Journal#of#Advanced#Nursing,#18,#1183K1187.#
Guba,#E.G.,#&#Lincoln,#Y.S.#(1981).#Effective#Evaluation.#Beverly#Hills,#CA:#Sage.#
Guba,#E.G.,#&#Lincoln,#Y.S.#(1985).#Naturalistic#inquiry.#San#Francisco:#JosseyKBass.#
Guba,#E.G,#&#Lincoln,#Y.S#(2004).#The#roots#of#fourth#generation#evaluation:#Theoretical#and#methodological#origins.#In#M.Alkin#(Ed.),#Evaluation#roots:#Tracing#theorists’#views#and#influences.#Thousand#Oaks,#CA:#Sage.�#
Guijt,#I.,#Brouwers,#J.,#Kusters,#C.,#Prins,#E.#&#Zeynalova,#B.#(2011).#Evaluation#Revisited:#Improving#the#Quality#
of#Evaluative#Practice#by#Embracing#Complexity.#[Conference#Report#of#the#same#name#held#in#Utrecht,#the#Netherlands,#in#May#2010]#Retrieved#May#7,#2013#from#http://www.capacity.org/capacity/export/sites/capacity/documents/topicKreadings/110412KevaluationKrevisitedKmayK2010_smallKversion.pdf#(no#longer#available).#See#alternative#link#https://evaluationrevisited.wordpress.com/2011/04/13/conferenceKreportKonline/##
Hadley,#B.J.#&#Gattenhof,#S.J.#(2012).#Brokering#evaluations#of#partnerships#in#Australian#community#arts:#responding#to#entrepreneurial#tendencies.#Journal#of#Arts#and#Communities#(JAAC),#4(3),#231K249.#
Hartley,#S.#(2008).#Project#Management:#Principles,#processes#and#practice.#Pearson#Education#Australia,#Frenchs#Forest.#
Hartley,#J.#&#McWilliam,#K.#(Eds.).#(2009).#Story#Circle:#Digital#Storytelling#Around#the#World.#Malden,#MA:#WileyKBlackwell.#
Haseman,#B.#(2006).#Manifesto#for#Performative#Research.#Media#International#Australia#incorporating#
Culture#and#Policy,#(118),#99K106.##
Haseman,#B.#&#Mafe,#D.#(2009).#Creative#Practice#and#Research#in#Education#and#Politics.#In#H.Smith#&#R.T.#Dean#(Eds.),#PracticeJled#Research,#ResearchJled#Practice#in#the#Creative#Arts#(pp.212K228).#Edinburgh:#Edinburgh#University#Press.#
Hatton,#N.,#&#Smith,#D.#(1995).#Reflection#in#teacher#education:#Towards#definition#and#implementation.#Teaching#and#teacher#education,#11(1),#33K49.#
Hawe,#P.,#Degeling,#D.#&#Hall,#J.#(1990).#Impact#and#Evaluation#Outcome.#In#P.#Hawe,#D.#Degeling#&#J.#Hall#(Eds.),#Evaluating#Health#Promotion:#A#Health#Workers’#Guide.#Sydney:#Maclennon#&#Petty#Publishing.#
Hawkesworth,#M.#E.#(1989).#Knowers,#knowing,#known:#Feminist#theory#and#claims#of#truth.#Signs,#533K557.#
Hawkins,#G.#(1993).#From#Nimbin#to#Mardi#Gras:#Constructing#Community#Arts.#Australian#Cultural#Studies.#St#Leonards,#NSW:#Allen#&#Unwin.#
Hertzberg#Kaare,#B.#&#Lundby,#K.#(2008).#Mediatized#lives:#Autobiography#and#assumed#authenticity#in#digital#storytelling.#In#K.Lundby#(Ed.),#Digital#storytelling,#mediatized#stories:#SelfJrepresentations#in#
new#media#(pp.#105K122).#Berne:#Peter#Lang.##
Hildreth,#P.#&#Kimble,#C.#(2004).#Knowledge#Networks:#Innovation#through#Communities#of#Practice.#London:#Hershey,#Idea#Group#Inc.#
#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 211#
Hlubinka,#M.#I.#(2003).#Behind#the#Screens:#Digital#Storytelling#as#a#Tool#for#Reflective#Practice.#[submission#for#Master#of#Science#at#the#Massachusetts#Institute#of#Technology].#Massachusetts:#Massachusetts#Institute#of#Technology.##
Hood,#D.#W.#(2002).#Feminist#Evaluation#and#the#Inclusion#of#Difference.#In#New#directions#for#evaluation,#96.96,#27K2.#
House,#E.#R.#(1993).#Professional#evaluation.#Newbury#Park,#CA:#Sage.##
House,#E.#R.,#&#Howe,#K.#R.#(2000).#Deliberative#democratic#evaluation#in#practice.#In#Evaluation#Models#(pp.#409K421).#Springer#Netherlands.#
Ilcan,#S.,#&#Basok,#T.#(2004).#Community#government:#voluntary#agencies,#social#justice,#and#the#responsibilization#of#citizens#1.#Citizenship#studies,#8(2),#129K144.#
Jackson,#M.R.#(2009).#Shifting#Expectations:#An#Urban#Planner’s#Reflections#on#Evaluation#of#CommunityJbased#
Arts#[Essay#for].#Retrieved#May#5,#2013#from#http://conference2012.culturaldevelopment.com.au/wpKcontent/uploads/2012/03/ShiftingExpectations.pdf#(no#longer#available).#See#alternative#link,#http://animatingdemocracy.org/sites/default/files/shifting_expectations.pdf##
Jackson,#M.R.,#Harranz,#J.#&#KabwasaKGreen,#F.#(2003).#Art#and#culture#in#communities:#a#framework#for#
measurement#[Policy#Brief#No.1#of#the#Culture,#Creativity#and#Communities#Program].#Washington#DC:#The#Urban#Institute.##
Jennings,#M.,#&#Baldwin,#A.#(2010).#“Filling#out#the#Forms#was#a#Nightmare”:#Project#evaluation#and#the#reflective#practitioner#in#community#theatre#in#contemporary#Northern#Ireland.#Music#and#Arts#in#Action,#2(2),#72K89.#
Johns,#C.#(2006).#Engaging#reflection#in#practice:#A#narrative#approach.#Oxford,#UK:#Blackwell#Publishing.#
Jonassen,#D.#H.#(1992).#Evaluating#constructivistic#learning.#In#T.M.Duffy#and#D.H.Jonassen#(Eds.),#Constructivism#and#the#technology#of#instruction:#A#conversation#(pp.137K148).#Hillsdale:#Lawrence#Erlbaum#Associates.#
Keating,#C.#(2002).#Evaluating#Community#Arts#and#Community#WellJBeing:#An#Evaluation#Guide#for#Community#
Arts#Practitioners.#(Eds.),#Arts#Victoria#and#Effective#Change.#Retrieved#May#5,#2013#from#http://creative.vic.gov.au/Research_Resources/Resources/Evaluating_Community_Arts_and_Wellbeing###
Kelaher,#M.,#Dunt,#D.,#Curry,#S.,#Berman,#N.,#&#Joubert,#L.#(2009).#Evaluation#of#the#Community#Arts#
Development#Scheme:#Improving#Health#and#Wellbeing#Through#Social#Connection#and#the#Arts.#Retrieved#May#15,#2014#from#https://www.vichealth.vic.gov.au/search/evaluationKofKtheKcommunityKartsKdevelopmentKscheme##
Kendall,#K.#E.,#&#Kendall,#J.#E.,#(2001).#Systems#analysis#and#design#(5th#Edn).#Upper#Saddle#River,#NJ:#PrenticeKHall.#
Klaebe,#H.,#Ellison,#E.,#&#Kirk,#G.#(2015).#Spotlight#on#animating#spaces:#Evaluation#of#Artslink#Queensland's#Animating#Spaces#2013#ߞ.#Brisbane:#Artslink#Queensland.#Available#from#http://eprints.qut.edu.au/89149/.#
Kotvojs,#F.#&#Hurworth,#R.#(2013).#Evaluating#capacity#development:#What#do#we#really#want#to#know?#Evaluation#Journal#of#Australasia,#13(1),#4.#
Kotzee,#B.#(2012).#Private#practice:#Exploring#the#missing#social#dimension#in#‘reflective#practice’.#Studies#in#Continuing#Education,#34(1),#5K16.#
Lambert,#J.#(2010).#Digital#Storytelling#Cookbook.#Berkeley:#Digital#Diner#Press.##
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 212#
Lambert,#J.#(2012).#Digital#storytelling:#Capturing#lives,#creating#community.#New#York:#Routledge.#
Lamont,#M.#(2011).#An#environment#of#collegial#reflective#dialogue#for#inservice#teacher#educators.#In#Processes#of#Inquiry#(pp.#63K88).#Rottadam:#Sense#Publishers.#
Larner,#W.,#&#Butler,#M.#(2005).#Governmentalities#of#local#partnerships:#The#rise#of#a"#partnering#state"#in#New#Zealand.#Studies#in#Political#Economy,#75.#
Lave,#J.#&#Wenger,#E.#(1991).#Situated#Learning:#Legitimate#Peripheral#Participation.#Cambridge:#Cambridge#University#Press.##
Lay,#M.,#&#Papadopoulos,#I.#(2007).#An#exploration#of#fourth#generation#evaluation#in#practice.#Evaluation,#13(4),#495K504.#
Leavy,#P.#(2009).#Method#meets#art:#Social#research#and#the#creative#arts.#London;#Guildford.#
Lehmann,#H.#(2001).#Using#grounded#theory#with#technology#cases:#distilling#critical#theory#from#a#multinational#information#systems#development#project.#Journal#of#Global#Information#Technology#
Management,#4(1),#45K60.#
Leitch,#R.,#&#Day,#C.#(2000).#Action#research#and#reflective#practice:#Towards#a#holistic#view.#Educational#Action#Research,#8(1),#179K193.#
Lennie,#J.#(2006).#Increasing#the#rigour#and#trustworthiness#of#participatory#evaluations:#learnings#from#the#field.#Evaluation#Journal#of#Australasia,#6(1),#7K35.#
Lippard,#L.R.#(1984).#Trojan#Horses:#Activist#Art#and#Power.#In#B.#Wallis#(Ed.),#Art#after#Modernism:#Rethinking#
Representation#(pp.341K358).#New#York:#New#Museum#of#Contemporary#Art.#
Loughran,#J.J.#(2002).#Effective#reflective#practice:#in#search#of#meaning#in#learning#about#teaching.#Journal#of#Teacher#Education,#53(1),#33K43.#
Lundby,#K.#(Ed.).#(2008).#Digital#storytelling,#mediatized#stories:#SelfJrepresentations#in#new#media.#Berne:#Peter#Lang.##
MacDowall,#L.#Introduction:#Making#Culture#Count.#In#L.MacDowall,#M.Badham,#E.Blomkamp,#&#K.Dunphy#(Eds.).#Making#Culture#Count:#The#Politics#of#Cultural#Measurement#(pp.1K8).#London:#Palgrave#Macmillan.##
MacDowall,#L.,#Badham,#M.,#Blomkamp,#E.,#&#Dunphy,#K.#(Eds.).#(2015).#Making#Culture#Count:#The#
Politics#of#Cultural#Measurement.#London:#Palgrave#Macmillan.##
MacDowall,#L.,#Mulligan,#M.,#Panucci,#F.#&#Badham,#M.#(2013).#Spectres#of#Evaluation#[Unpublished#workshop#paper].#Centre#for#Cultural#Partnerships,#University#of#Melbourne.##
Mark,#M.#M.,#Henry,#G.,#Julnes,#G.#(2000).#Evaluation:#An#integrated#framework#for#understanding,#guiding,#and#improving#policies#and#programs.#San#Francisco,#CA:#JosseyKBass.##
Markiewicz,#A.#(2012).#Closing#the#gap#through#respect,#relevance,#reciprocity#and#responsibility:#Issues#in#the#evaluation#of#programs#for#Indigenous#communities#in#Australia.#Evaluation#Journal#of#Australasia,#12(1),#19.#
Mark,#M.#M.#(2011).#Toward#better#research#on—And#thinking#about—Evaluation#influence,#especially#in#multisite#evaluations.#New#Directions#for#Evaluation,#2011(129),#107K119.#
Mason,#J.#(2002).#Qualitative#researching#(2nd#ed.).#London:#Sage.#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 213#
Mathison,#S.#(Ed.).#(2005).#Encyclopedia#of#evaluation.#Thousand#Oaks,#CA:#Sage.##
Matarasso,#F.#(1996).#Defining#Values:#Evaluating#Arts#Programs#[Social#Impact#of#the#Arts:#Working#Paper#1].#Bournes#Green:#Comedia.#
Matarasso,#F.#(1997).#Use#Or#Ornament?:#The#Social#Impact#of#Participation#in#the#Arts.#Stroud:#Comedia.#
McClelland,#J.#R.#(1985).#The#Report#of#the#Royal#Commission#into#British#Nuclear#Tests#in#Australia.#Australian#Government#Publication,#Appendix#G#Chronology.#
McKay,#J.,#&#Marshall,#P.#(2001).#The#dual#imperatives#of#action#research.#Information#Technology#&#People,#14(1),#46K59.##
McNeill,#P.#(1985).#Research#Methods.#London:#Tavistock.#
McNiff,#S.#(2008).#ArtsKBased#Research.#In#J.G.#Knowles#and#A.#L.#Cole#(Eds.),#Handbook#of#the#Arts#in#Qualitative#Research#(pp.#29K40).#Thousand#Oaks,#CA:#Sage.##
Milbourne,#L.#(2009).#Remodelling#the#Third#Sector:#Advancing#collaboration#or#competition#in#communityKbased#initiatives?#Journal#of#Social#Policy,#38(02),#277K297.#
Mills,#D.#&#Brown,#P.#(2004).#Art#and#wellbeing:#a#guide#to#the#connections#between#community#cultural#
development#and#health,#ecologically#sustainable#development,#public#housing#and#place,#rural#
revitalisation,#community#strengthening,#active#citizenship,#social#inclusion#and#cultural#diversity.#Surry#Hills:#Australia#Council#for#the#Arts.#
Moffatt,#A.#C.#(2014).#Stepping#outside#the#circle:#a#reflective#practice#framework#for#creative#facilitators.#Retrieved#May#4,#2015#from#http://eprints.qut.edu.au/78130/2/Amanda_Moffatt_Practitioner_Resource.pdf##
Moon,#J.#(2001).#Learning#through#reflection.#In#Early#professional#development#for#teachers#(pp.#364K378).#London:#David#Fulton.#
Moreau,#K.#A.,#&#Cousins,#J.#(2012).#Program#evaluation#in#familyKcentred#pediatric#rehabilitation#settings:#A#review#of#evaluation#studies#and#the#potential#use#of#participatory#and#collaborative#evaluation#approaches.#Evaluation#Journal#of#Australasia,#11(2),#3.#
Mulligan,#M.#&#Smith,#P.#(2009).#Art,#Government#and#the#Turn#to#Community:#Putting#Art#at#the#Heart#
of#Local#Government.#Melbourne:#RMIT.#
National#Archives#of#Australia.#(2013).#British#nuclear#tests#at#MaralingaJFactsheet#129.#Retrieved#June#3,#2013#from#http://www.naa.gov.au/collection/factKsheets/fs129.aspx##
National#Ideas#Summit#&#Australia#Council#(1990).#Unlocking#the#academies:#discussion#paper#and#proposals#for#
action#from#forum#I#following#the#National#Ideas#Summit#(1990:#Canberra).#Redfern:#Australia#Council.#
Newman,#S.#(1999).#Constructing#and#critiquing#reflective#practice.#Educational#Action#Research,#7:1,#145K163.#
Newman,#T.,#Curtis,#K.,#&#Stephens,#J.#(2003).#Do#communityKbased#arts#projects#result#in#social#gain?#A#review#of#literature.#Community#Development#Journal,#38(4),#310K322.#
Osfield#Consultants.#(2007).#HighWater#Theatre#the#Partnership#–#Somebody’s#Daughter#Theatre#Company,#
Upper#Hume#Community#Health#Service.#Melbourne:#Victorian#Department#of#Education.##
#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 214#
Osmotherly#J.#(2002).#The#HighWater#Youth#Arts#Program#Somebody’s#Daughter#Theatre#Company.#Paper#presented#by#Somebody’s#Daughter#Theatre#Company#at#the#Role#of#Schools#in#Crime#Prevention#
Conference#convened#in#Melbourne:#Australian#Institute#of#Criminology.##
Owen,#J.M.#&#Rogers,#P.J.#(1999).#Program#Evaluation:#Forms#and#Approaches.#St#Leonards,#NSW:#Allen#&#Unwin.#
Palmer,#D.#(2010).#Ngapartji#Ngapartji:#the#consequences#of#kindness.#Perth:#Murdoch#University.##
Parsons,#H.#(2015).#Growth#and#Depression#in#Hayek’s#Garden:#The#Emotional#Language#of#Accounting.#In#L.MacDowall,#M.Badham,#E.Blomkamp,#&#K.Dunphy#(Eds.).#Making#Culture#Count:#The#Politics#of#
Cultural#Measurement#(pp.53K66).#London:#Palgrave#Macmillan.##
Patton,#M.#Q.#(1994).#Developmental#evaluation.#Evaluation#practice,#15(3),#311K319.#
Patton,#M.#Q.#(2008).#UtilizationJfocused#evaluation.#Thousand#Oaks,#CA:#Sage.#
Patton,#M.#Q.#(2010).#Developmental#evaluation:#Applying#complexity#concepts#to#enhance#innovation#and#use.#New#York:#Guilford#Press.#
Price,#M.,#McCoy,#B.,#&#Mafi,#S.#(2012).#Progressing#the#dialogue#about#a#framework#for#Aboriginal#evaluations:#Sharing#methods#and#key#learnings.#Evaluation#Journal#of#Australasia,#12(1),#32.#
Richardson,#K.#(2004).#Introduction:#General#systems#theoryK#the#skeleton#of#science.#E:CO#Special#Double#Issue,#(6)(1K2),#127K139.#
Rist,#R.C.#(Ed.).#(1999).#Program#evaluation#and#the#management#of#government:#Patterns#and#prospects#across#
eight#nations.#New#Brunswick,#JF:#Transaction#Publishers.#
Rodgers,#C.#(2002).#Defining#reflection:#Another#look#at#John#Dewey#and#reflective#thinking.#The#Teachers#College#Record,#104(4),#842K866.#
Rowley,#J.#(2012).#Conducting#research#interviews.#Management#Research#Review,#35(3/4),#260K271.#
Ryan,#K.,#&#Cousins,#J.#B.#(Eds.).#(2009).#The#SAGE#international#handbook#of#educational#evaluation.#Newbury#Park,#CA:#Sage.#
Scanlan,#J.#M.,#&#Chernomas,#W.#M.#(1997).#Developing#the#reflective#teacher.#Journal#of#Advanced#Nursing,#25(6),#1138K1143.#
Schon,#D.#A.#(1983).#The#Crisis#of#Confidence#in#Professional#Knowledge.#In#D.#A.#Schon#(Ed.),#The#reflective#practitioner:#how#professionals#think#in#action#(pp.3K20).#New#York:#Basic#Books.##
Schratz,#M.#(1993).#From#Cooperative#Action#to#Collective#SelfKreflection:#A#sociodynamic#approach#to#educational#research.#In#M.#Schratz#(Ed.),#Qualitative#voices#in#educational#research#(pp.56K70).#London:#The#Falmer#Press.##
Schwandt,#T.#A.#(2009).#Globalization#influences#on#the#western#evaluation#imaginary.#The#SAGE#international#handbook#of#educational#evaluation,#19K36.#
Schwandt,#T.#(2000).#Three#Epistemological#Stances#for#Qualitative#Inquiry:#Interpretivism,#Hermeneutics,#and#Social#Constructionism.#In#N.K#Denzin#&#Y.#Lincoln#(Eds.),#Handbook#of#Qualitative#Research#(2nd#ed.),#(pp.189K213),#Thousand#Oaks,#CA:#Sage.##
Scriven,#M.#(1967).#The#methodology#of#evaluation.#In#R.E.#Stake#(Ed.),#Curriculum#evaluation.#(American#Educational#Research#Association#Monograph#Series#on#Evaluation),#No.#1,#pp.#39K83).#Chicago:#Rand#McNally.#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 215#
Scriven,#M.#(1991).#Key#evaluation#checklist.#In#M.#Scriven#(Ed.),#Evaluation#thesaurus.#Thousand#Oaks,#CA:#Sage.#
Scriven,#M.#(2003a).#Evaluation#in#the#new#millennium:#The#transdisciplinary#view.#In#S.#I.#Donaldson#&#M.#Scriven#(Eds.),#Evaluating#Social#Programs#and#Problems:#Vision#for#the#New#Millennium.#Mahwah,#NJ:#Erlbaum.#pp.#19–42.�#
Scriven,#M.#(2003b).#The#Key#Evaluation#Checklist.#Available#online:#http://evaluation.wmich.edu/checklists/#
Scriven,#M.#(2012).#Conceptual#Revolutions#About#Evaluation;#Past#and#Future.#Recorded#lecture#from#The#Evaluation#Centre#(Evaluation#Café:#September#11th,#2012).#Retrieved#June#4,#2014,#from#http://vimeo.com/49308493##
Selwood,#S.#(2002).#Measuring#Culture#[Online#article#on#Spiked#website].#Retrieved#May#23,#2013#from#http://www.spikedKonline.com/articles/00000006DBAF.htm#(no#longer#available).#See#alternative#link,#http://www.spikedKonline.com/newsite/article/6851#.V4YZO5N94n0##
Sharp,#C.#(2003).#Development#of#Program#Evaluation#in#Australia#and#the#Australasian#Evaluation#Society#–#the#early#decades.#Evaluation#Journal#of#Australasia,#3(2),#6K16.#
Shaw,#I.,#Shaw,#I.#G.#R.,#Greene,#J.#C.,#&#Mark,#M.#M.#(Eds.).#(2006).#The#Sage#handbook#of#evaluation.#Thousand#Oaks,#CA:#Sage.#
SielbeckKBowen,#K.#A.,#Brisolara,#S.,#Seigart,#D.,#Tischler,#C.,#&#Whitmore,#E.#(2002).#Exploring#feminist#evaluation:#The#ground#from#which#we#rise.#New#directions#for#evaluation,#2002(96),#3K8.#
Smith,#H.#&#Dean,#R.T.#(Eds.).#(2009).#PracticeJled#Research,#ResearchJled#Practice#in#the#Creative#Arts.#Edinburgh:#Edinburgh#University#Press.#
Stake,#R.#(1967).#The#countenance#of#educational#evaluation.#The#Teachers#College#Record,#68(7),#523K540.#
Stauffer,#B.#(2015).#‘Process#Documentation’.#[An#adapted#online#summary#of:#Moriarty,#P.#(2007).#The#EMPOWERS#approach#to#water#governance:#guidelines,#methods#and#tools.#INWRDAM]#Retrieved#December#7,#2015#from#http://www.sswm.info/content/processKdocumentation#reference_book139##
Stock,#C.#(2013).#Course#Overview#and#Forensic#Reflective#Practice#Intensive#Modules#[Doctor#of#Creative#Industries,#QUT,#Unpublished#course#materials].#Retrieved#April#3,#2013#from#QUT#Blackboard#online#community#site#(password#protected).#
Sullivan,#G.#(2009).#Making#Space:#The#Purpose#and#place#of#PracticeKled#Research.#In#H.#Smith#&#R.T.#Dean,#PracticeJled#Research,#ResearchJled#Practice#in#the#Creative#Arts#(pp.#41K65).#Edinburgh:#Edinburgh#University#Press.#
Sullivan,#G.#(Ed.).#(2010).#Art#practice#as#research:#Inquiry#in#visual#arts.#Thousand#Oaks,#CA:#Sage.#
Swan,#P.,#&#Atkinson,#S.#(2012).#Managing#evaluation:#A#community#arts#organisation's#perspective.#Arts#&#Health,#4(3),#217K229.#
Tharp,#T.#(2003).#Twyla#Tharp:#The#creative#habit:#Learn#it#and#use#it#for#life:#A#practical#guide.#New#York:#Simon#&#Schuster#
Thomas,#L.#(2015).#Curating#in#uncharted#territories:#An#examination#of#a#programming#model#&#
communicative#platform#for#curation#in#nonJtraditional#art#spaces.#Masters#by#Research#thesis,#Queensland#University#of#Technology.#Retrieved#January#22,#2016#from#http://eprints.qut.edu.au/84621/##
#
AN#INTEGRATED#APPROACH#TO#EVALUATION# ELLISE#BARKLEY#
# 216#
Thorpe,#K.#(2004).#Reflective#learning#journals:#From#concept#to#practice.#Reflective#practice,#5(3),#327K343.#
UNESCO#and#ICCROM.#(2015).#‘Why#Document’.#ReKorg#[website].#Retrieved#December#7,#2015#from#http://reKorg.info/en/documentationKsystem/1KwhyKdocument###
van#Erven,#E.#(2001).#Community#Theatre:#Global#Perspectives.#London:#Routledge.#
Wadsworth,#Y.#(1998).#What#is#Participatory#Action#Research?.#Action#Research#International,#Paper#2.#Retrieved#May#20,#2013#from#http://www.uq.net.au/action_research/ari/pKywadsworth98.html#(no#longer#available).#See#alternative#link#http://www.aral.com.au/ari/pKywadsworth98.html##
Wakefield#Regional#Council.#(2015).#[website].#Retrieved#April#3,#2015#from#http://www.wakefieldrc.sa.gov.au/page.aspx##
Watkins,#J.#and#Tacchi,#J.#(2008).#Participatory#Content#Creation#for#Development:#Principles#and#Practices#(Research#from#the#Finding#a#Voice#Project),#India:#UNESCO#House.#
WebFinances.#(2016).#‘Product’.#[Online#Business#Dictionary]#http://www.businessdictionary.com/definition/product.html#ixzz3w843DBlD##
Weiss,#C.H.#(1972).#Evaluation#research:#Methods#for#assessing#program#effectiveness.#Englewood#Cliffs,#NJ:#PrenticeKHall.#
Wenger,#E.,#McDermott,#R.#&#Snyder,#W.M.#(2002).#Cultivating#Communities#of#Practice#(Hardcover),#Boston:#Harvard#Business#Press.#
Wilcox,#D.#(1996).#The#Guide#to#Effective#Participation,#Retrieved#from#May#20,#2013#from#http://partnerships.org.uk/guide/main1.html##
Wilde,#J.#&#Sockey,#S.#(1995)#Evaluation#Handbook.#Albuquerque,#NM,#Evaluation#Assistance#CenterKWestern#Region,#New#Mexico:#New#Mexico#Highlands#University.#Retrieved#May#2,#2013#from#http://www.ncela.us/files/rcd/BE020502/Evaluation_Handbook.pdf##
Williams,#D.#(1997)#How#the#Arts#Measure#Up:#Australian#Research#into#Social#Impact#[Social#Impact#of#the#Arts#Working#Paper#No.8.].#Gloustershire:#Comdedia.#
Woolcock,#G.,#&#Davern,#M.#(2015).#Creative#Accounts:#Reimagining#Culture#and#Wellbeing#by#Tapping#into#the#Global#Movement#to#Redefine#Progress.#In#L.MacDowall,#M.Badham,#E.Blomkamp,#&#K.Dunphy#(Eds.).#Making#Culture#Count:#The#Politics#of#Cultural#Measurement#(pp.129K144).#London:#Palgrave#Macmillan.##
Yalata#&#Oak#Valley#Communities#with#Mattingley,#C#(2009).#Maralinga:#the#Anangu#story.#St#Leonards,#NSW:#Allen#&#Unwin.#
#
#
************************************end************************************#