![Page 1: FoME Symposium 2015 | Workshop 8: Current Evaluation Practices and Perspectives on more Effective Media Development Evaluation | Jessica Noske-Turner: Towards more effective approaches](https://reader035.vdocuments.us/reader035/viewer/2022070512/589ae3821a28abee708b5625/html5/thumbnails/1.jpg)
Towards more effective evaluation approaches for media development
![Page 2: FoME Symposium 2015 | Workshop 8: Current Evaluation Practices and Perspectives on more Effective Media Development Evaluation | Jessica Noske-Turner: Towards more effective approaches](https://reader035.vdocuments.us/reader035/viewer/2022070512/589ae3821a28abee708b5625/html5/thumbnails/2.jpg)
Current projects
Evaluating Communication for development: supporting adaptive and accountable development
Mobilising Media for Sustainable Outcomes in the Pacific Region
RMIT University School of Media & Communication 2
![Page 3: FoME Symposium 2015 | Workshop 8: Current Evaluation Practices and Perspectives on more Effective Media Development Evaluation | Jessica Noske-Turner: Towards more effective approaches](https://reader035.vdocuments.us/reader035/viewer/2022070512/589ae3821a28abee708b5625/html5/thumbnails/3.jpg)
PhD Research Question & Design
Research Question: How can the impacts of Australian media assistance on social change and governance be most effectively evaluated and understood?
Phase 1: Document analysis: Media Assistance Evaluations between 2002-2012 (total of 47) Phase 2: InterviewsInterviews with media assistance evaluators (consultants and in-house, total of 10)
Phase 3: Case study:Cambodian Communication Assistance Project (CCAP), funded by AusAID-funded project, managed by ABC International Development.
![Page 4: FoME Symposium 2015 | Workshop 8: Current Evaluation Practices and Perspectives on more Effective Media Development Evaluation | Jessica Noske-Turner: Towards more effective approaches](https://reader035.vdocuments.us/reader035/viewer/2022070512/589ae3821a28abee708b5625/html5/thumbnails/4.jpg)
Findings: Key challenges
• Conceptual ambiguities
• Bureaucratic systems
• Complex epistemological and political undercurrents in the evaluation discipline
RMIT University School of Media & Communication 4
![Page 5: FoME Symposium 2015 | Workshop 8: Current Evaluation Practices and Perspectives on more Effective Media Development Evaluation | Jessica Noske-Turner: Towards more effective approaches](https://reader035.vdocuments.us/reader035/viewer/2022070512/589ae3821a28abee708b5625/html5/thumbnails/5.jpg)
1. Conceptual ambiguities
RMIT University School of Media & Communication 5
![Page 6: FoME Symposium 2015 | Workshop 8: Current Evaluation Practices and Perspectives on more Effective Media Development Evaluation | Jessica Noske-Turner: Towards more effective approaches](https://reader035.vdocuments.us/reader035/viewer/2022070512/589ae3821a28abee708b5625/html5/thumbnails/6.jpg)
2. Bureaucratic systems
the codification of approaches that are meant to accomplish positive outcomes into mechanical checklists and templates that not only fail to achieve their intent but actually lead to even worse outcomes. (Anderson, Brown and Jean 2012: 67)
Proceduralization:
![Page 7: FoME Symposium 2015 | Workshop 8: Current Evaluation Practices and Perspectives on more Effective Media Development Evaluation | Jessica Noske-Turner: Towards more effective approaches](https://reader035.vdocuments.us/reader035/viewer/2022070512/589ae3821a28abee708b5625/html5/thumbnails/7.jpg)
2. Bureaucratic systems
Authored by Total Commissioned (/required) by
donor
Commissioned (/required) by
projectExternal consultant 27 19 8
Donor 5 5
Project 5 2 3
Consultant + Donor 2 2
Consultant + Project 1 1
Donor + Project 1 1
unknown 6 6
Total 47 35 12
![Page 8: FoME Symposium 2015 | Workshop 8: Current Evaluation Practices and Perspectives on more Effective Media Development Evaluation | Jessica Noske-Turner: Towards more effective approaches](https://reader035.vdocuments.us/reader035/viewer/2022070512/589ae3821a28abee708b5625/html5/thumbnails/8.jpg)
2. Bureaucratic systems
![Page 9: FoME Symposium 2015 | Workshop 8: Current Evaluation Practices and Perspectives on more Effective Media Development Evaluation | Jessica Noske-Turner: Towards more effective approaches](https://reader035.vdocuments.us/reader035/viewer/2022070512/589ae3821a28abee708b5625/html5/thumbnails/9.jpg)
2. Bureaucratic systems
![Page 10: FoME Symposium 2015 | Workshop 8: Current Evaluation Practices and Perspectives on more Effective Media Development Evaluation | Jessica Noske-Turner: Towards more effective approaches](https://reader035.vdocuments.us/reader035/viewer/2022070512/589ae3821a28abee708b5625/html5/thumbnails/10.jpg)
• ‘Nuancing’ reports• Being ‘circumspect’• Pre-cooking evaluations
Dependence on Independence
2. Bureaucratic systems
![Page 11: FoME Symposium 2015 | Workshop 8: Current Evaluation Practices and Perspectives on more Effective Media Development Evaluation | Jessica Noske-Turner: Towards more effective approaches](https://reader035.vdocuments.us/reader035/viewer/2022070512/589ae3821a28abee708b5625/html5/thumbnails/11.jpg)
•Simplicity and complexity•Proving vs improving•Positivism vs participation
3. Evaluation epistemologies
![Page 12: FoME Symposium 2015 | Workshop 8: Current Evaluation Practices and Perspectives on more Effective Media Development Evaluation | Jessica Noske-Turner: Towards more effective approaches](https://reader035.vdocuments.us/reader035/viewer/2022070512/589ae3821a28abee708b5625/html5/thumbnails/12.jpg)
Adapted from Manyozo (2012)
Media DevelopmentFocus on industryAims to improve governance.
Media for DevelopmentFocus on contentAims to educate and inform.
Participatory Communication: Focus on dialogueAims to foster participation and self-determination.
Conceptual ambiguities
![Page 13: FoME Symposium 2015 | Workshop 8: Current Evaluation Practices and Perspectives on more Effective Media Development Evaluation | Jessica Noske-Turner: Towards more effective approaches](https://reader035.vdocuments.us/reader035/viewer/2022070512/589ae3821a28abee708b5625/html5/thumbnails/13.jpg)
Prototype
![Page 14: FoME Symposium 2015 | Workshop 8: Current Evaluation Practices and Perspectives on more Effective Media Development Evaluation | Jessica Noske-Turner: Towards more effective approaches](https://reader035.vdocuments.us/reader035/viewer/2022070512/589ae3821a28abee708b5625/html5/thumbnails/14.jpg)
Prototype
![Page 15: FoME Symposium 2015 | Workshop 8: Current Evaluation Practices and Perspectives on more Effective Media Development Evaluation | Jessica Noske-Turner: Towards more effective approaches](https://reader035.vdocuments.us/reader035/viewer/2022070512/589ae3821a28abee708b5625/html5/thumbnails/15.jpg)
C4D Design & Evaluation in the Pacific
RMIT University School of Media & Communication 15
• Workshops, field visits, pilot– Capacity building – ‘Innovation’– Proportionate– Piloting the ‘LEAD4innovation
toolkit’or C4D IDEAS Toolkit– Incudes a facilitator’s guide
Rainbow Framework Toolkit Module Objectives and Activities
MANAGE an evaluation or evaluation system
Module 2: Stakeholder Mapping Who who has an interest or role in our project and evaluation? How can we communicate with them?
Module 3: Ethics
What are the potentially ethical issues in media and communication projects and evaluation? How can we prevent these from becoming problems?
REPORT & SUPPORT USE of findings
Module 4: Planning Data Use
What is our macro-timeline for sharing evaluation insights? How can we build in continuous learning? What are the most appropriate ways to share our insights?
DEFINE what is to be evaluated
Module 5: DefineWhat are we trying to do, how are we trying to do it?
FRAME the boundaries for an evaluation
Module 6: Ask QuestionsBased on our main goals, what are our key evaluation questions?
DESCRIBE activities, outcomes, impacts and context
Module 7: Planning Information and Data Collection
Which data collection methods will give us the best information to answer our questions? Which methods are realistic?
UNDERSTAND CAUSES of outcomes and impacts
Module 8: Making Sense of DataHow do we manage our data? How can we make sense of and analyse different kinds of data?
(SYNTHESISE data from one or more evaluations)
(Included in the Facilitator’s Guide)
(Meta-analysis of groups of projects)
Contents of the LEAD4innovation/ C4D IDEAS Toolkit against the Rainbow Framework
![Page 16: FoME Symposium 2015 | Workshop 8: Current Evaluation Practices and Perspectives on more Effective Media Development Evaluation | Jessica Noske-Turner: Towards more effective approaches](https://reader035.vdocuments.us/reader035/viewer/2022070512/589ae3821a28abee708b5625/html5/thumbnails/16.jpg)
RMIT University School of Media & Communication 16
![Page 17: FoME Symposium 2015 | Workshop 8: Current Evaluation Practices and Perspectives on more Effective Media Development Evaluation | Jessica Noske-Turner: Towards more effective approaches](https://reader035.vdocuments.us/reader035/viewer/2022070512/589ae3821a28abee708b5625/html5/thumbnails/17.jpg)
Conventional Baselines Living BaselinesOriginates from experimental designs Challenges conventional notion of
‘baselines’Assumes controlled environment Acknowledges complexity (feedback
loops, interrelationships)
Simple model of cause and effect Social change occurs through interaction between multiple factors and agents
Anticipates linear trajectories of change Anticipates multiple, emergent changes which are unknowable in advance
For measuring For generating insights and learning
Research is undertaken twice (before and after), or at most, three times (if there is a midline)
Research is undertaken periodically
Fixed, contained Adaptive, expandingPositions development knowledge in the domain of experts
Privileges local knowledge
“Living Baseline”
![Page 18: FoME Symposium 2015 | Workshop 8: Current Evaluation Practices and Perspectives on more Effective Media Development Evaluation | Jessica Noske-Turner: Towards more effective approaches](https://reader035.vdocuments.us/reader035/viewer/2022070512/589ae3821a28abee708b5625/html5/thumbnails/18.jpg)
RMIT University School of Media & Communication 18
Noske-Turner, J. (2015). 10 Years of Evaluation Practice in Media Assistance: Who, When, Why and How?. In Nordicom Review (Vol. 36, No. Special Issue, pp. 41-56)
Noske-Turner, J. (2014). Evaluating the impacts of media assistance: problems, and principles. Global Media Journal German Edition, 4(2), 1-21.
Publications