evaluating media assistance programs: what we have done and what we have learned lee b. becker &...
TRANSCRIPT
![Page 1: Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d155503460f949e9aa1/html5/thumbnails/1.jpg)
Evaluating Media Assistance Programs:
What We Have Done and What We Have Learned
Lee B. Becker & Tudor Vlad
![Page 2: Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d155503460f949e9aa1/html5/thumbnails/2.jpg)
2
Background• University of Georgia
• Grady College of Journalism & Mass Communication
• James M. Cox Jr. Center for International Mass Communication Training and Research– Conducts training programs– Focuses on evaluation
Athens and the university in 1840 as depicted in the painting by George Cooke.
![Page 3: Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d155503460f949e9aa1/html5/thumbnails/3.jpg)
3
My Research Interests
• Media effects– Opinion and learning
• Organizational behavior and message construction– Characteristics of workers, including
educational credentials
• Sociology of education and labor markets– Needs assessment– Evaluation of impact
![Page 4: Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d155503460f949e9aa1/html5/thumbnails/4.jpg)
4
Evaluation of the Knight International Press Fellowship
Program
1. Interviewed 531 persons in 8 European and 3 Latin American Countries
2. At 31 people in each of the 11 countries
3. Field work completed in 1999.
![Page 5: Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d155503460f949e9aa1/html5/thumbnails/5.jpg)
5
Basic Distinction
• Assessing and monitoring program process.
• Measuring and monitoring program outcomes.
![Page 6: Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d155503460f949e9aa1/html5/thumbnails/6.jpg)
6
Locus of Impact
• On journalists– Attitudes– Behaviors
• On organizations
• On media system
• On society overall
![Page 7: Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d155503460f949e9aa1/html5/thumbnails/7.jpg)
7
Problems with Self-Reports
• Timing is important– Immediately after training is too soon– Duration of effect is variable
• Likely to overestimate effect– To satisfy the program provider– To justify own investment of time and effort
• Lacks reference– Most participants in training on projectile to change
• Participant may not know the answer
![Page 8: Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d155503460f949e9aa1/html5/thumbnails/8.jpg)
8
Journalists in Philippines
• Overestimated the amount of coverage they gave to the issues behind the conflict.
• Underestimated the amount of coverage of the Manila media to issues other than the conflict.
![Page 9: Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d155503460f949e9aa1/html5/thumbnails/9.jpg)
9
Measurement Solutions• Go from the general to the specific
– Now I want to start with a very general question. Please tell me some of the things that you learned from your participation in the XXX Fellowship that you consider at this point to have been most important to your work as a journalist. Of course, if you didn’t learn anything just tell me that.
– OK. Now I’m going to ask about some specifics. You may already have touched on some of these, but I want to make sure I cover everything. Did your participation in the XXX Fellowship help you develop your skills in findings sources for stories you have reported since you returned?
• Use what we can a “jab and probe” questioning technique.– Are there specific sources that you came into contact with
through the XX Fellowship that you still use today? • IF YES: Who are some of those?
![Page 10: Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d155503460f949e9aa1/html5/thumbnails/10.jpg)
10
Validation Measures• Interviews others who also can attest to
attitudes, behaviors and changes in both– Colleagues– Supervisors
• Examples– Since you completed the XXX Fellowship, have you
held any “brown bag” or other information discussions about your experiences in the program with other journalists?
– As far as you know, has (NAME OF FELLOW) made any efforts to share her/his experiences at the Fellowship with others at your organization since she/he returned?
![Page 11: Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d155503460f949e9aa1/html5/thumbnails/11.jpg)
11
Design Solutions
• Treat timing as a variable– We have compared short-term programs
conducted across time• Timing matters—effect several years• But topic matters too and can offset lag
• Develop a “control” group– True experimental design isn’t often possible– Imperfect “control” better than none
• Develop before and after design
![Page 12: Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d155503460f949e9aa1/html5/thumbnails/12.jpg)
12
Some Control Examples
• Control group made up of matched respondents– Successful applicants for subsequent ongoing
programs– Those who work in similar settings
• Can match based on lists • Can get respondent to help create a match
• Control groups made up of matched organizations
• Best control is the group itself– Before and after has limitation of systemic change
![Page 13: Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d155503460f949e9aa1/html5/thumbnails/13.jpg)
13
Post-Employment Training Common Part of Media Assistance
• When evaluating journalism training component of media assistance, keep in mind– Measurement problems– Design problems
• Use tactics to overcome them
![Page 14: Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d155503460f949e9aa1/html5/thumbnails/14.jpg)
14
Central Concerns of Media Assistance
• Concern with Media Freedom or media independence
• Concern with media performance
• Concern with the connection between Media Freedom and media performance
• Concern with the relationship between media performance and democracy
![Page 15: Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d155503460f949e9aa1/html5/thumbnails/15.jpg)
15
Mapping Media Assistance
• Effort by Monroe Price and colleagues
• Our purpose was to expand that
• Shifted to narrower short-term goal
• Rough estimate: $1 billion year spent on media assistance each year
• Ultimately, valuable to understand media assistance this way
![Page 16: Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d155503460f949e9aa1/html5/thumbnails/16.jpg)
16
Two Actor Groups
• Media assistance community– Donors– Media assistance providers– Targets of media assistance– Media monitors
• Academic community– Political scientists– Communication scientists
![Page 17: Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d155503460f949e9aa1/html5/thumbnails/17.jpg)
17
Academic Community
• Political scientists– Extensive literature on democratization– Extensive debate on meaning of democracy
• Communication scientists– Extensive literature on media freedom– Extensive debate on meaning of media freedom
• Relatively little empirical testing of contribution of media freedom to democracy or democratization
![Page 18: Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d155503460f949e9aa1/html5/thumbnails/18.jpg)
18
Different Perspectives of Advocates of Media Assistance
and Political and Communication Scientists
• Former believe media assistance leads to democratization– Role of the research is to document this
• Scientists think all of these questions are open to examination
![Page 19: Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d155503460f949e9aa1/html5/thumbnails/19.jpg)
19
Media Assistance
Trained, Skilled, Motivated Journalists
Media Organizations that Facilitate and Distribute the Work of Journalists
Independent Media System
Information Needed for Functioning of Democracy, Open Economy
Institutions of Civil Society
Functioning Democracy, Economy
Civil Society Assistance
?
Model of Impact of Media Assistance
![Page 20: Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d155503460f949e9aa1/html5/thumbnails/20.jpg)
20
Media Assistance
Trained, Skilled, Motivated Journalists
Media Organizations that Facilitate and Distribute the Work of Journalists
Independent Media System
Information Needed for Functioning of Democracy, Open Economy
Institutions of Civil Society
Functioning Democracy, Economy
Civil Society Assistance
?
Model of Impact of Media Assistance
![Page 21: Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d155503460f949e9aa1/html5/thumbnails/21.jpg)
21
Does Media Assistance Lead to Trained, Skilled, Motivated
Journalists?
– Yes, probably• Evaluation is spotty
– Process evaluation rather than impact evaluation
• Often rely on self-reports of impact• Control groups are rarely used• Evaluations often not independent of funder,
media assistance provider
![Page 22: Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d155503460f949e9aa1/html5/thumbnails/22.jpg)
22
Media Assistance
Trained, Skilled, Motivated Journalists
Media Organizations that Facilitate and Distribute the Work of Journalists
Independent Media System
Information Needed for Functioning of Democracy, Open Economy
Institutions of Civil Society
Functioning Democracy, Economy
Civil Society Assistance
?
Model of Impact of Media Assistance
![Page 23: Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d155503460f949e9aa1/html5/thumbnails/23.jpg)
23
Does Media Assistance Lead to Facilitative Media Organizations?
• Maybe– Most evidence is indirect, based on
observations of participants– Designs are inadequate– Many examples of failed investments– Problems of sustainability paramount
![Page 24: Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d155503460f949e9aa1/html5/thumbnails/24.jpg)
24
Media Assistance
Trained, Skilled, Motivated Journalists
Media Organizations that Facilitate and Distribute the Work of Journalists
Independent Media System
Information Needed for Functioning of Democracy, Open Economy
Institutions of Civil Society
Functioning Democracy, Economy
Civil Society Assistance
?
Model of Impact of Media Assistance
![Page 25: Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d155503460f949e9aa1/html5/thumbnails/25.jpg)
25
Does Media Assistance Produce Independent Media Systems?
• Some limited concrete evidence it does
• Comparative study by Steven Finkel and colleagues for USAID– Found that USAID Media investments across
165 countries led to gains in media freedom– Used Freedom House indicators
• Considerable controversy over what is meant by media freedom
![Page 26: Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d155503460f949e9aa1/html5/thumbnails/26.jpg)
26
Media Assistance
Trained, Skilled, Motivated Journalists
Media Organizations that Facilitate and Distribute the Work of Journalists
Independent Media System
Information Needed for Functioning of Democracy, Open Economy
Institutions of Civil Society
Functioning Democracy, Economy
Civil Society Assistance
?
Model of Impact of Media Assistance
![Page 27: Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d155503460f949e9aa1/html5/thumbnails/27.jpg)
27
Do Independent Media Produce Information Needed for
Democracy?• Almost no evidence either way
– No real explication of what information is needed for democracy
![Page 28: Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d155503460f949e9aa1/html5/thumbnails/28.jpg)
28
Media Assistance
Trained, Skilled, Motivated Journalists
Media Organizations that Facilitate and Distribute the Work of Journalists
Independent Media System
Information Needed for Functioning of Democracy, Open Economy
Institutions of Civil Society
Functioning Democracy, Economy
Civil Society Assistance
?
Model of Impact of Media Assistance
![Page 29: Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d155503460f949e9aa1/html5/thumbnails/29.jpg)
29
Do Independent Media Produce Civil Society Institutions?
• No one knows
• Mixed evidence of the effectiveness of civil society assistance as well
![Page 30: Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d155503460f949e9aa1/html5/thumbnails/30.jpg)
30
Media Assistance
Trained, Skilled, Motivated Journalists
Media Organizations that Facilitate and Distribute the Work of Journalists
Independent Media System
Information Needed for Functioning of Democracy, Open Economy
Institutions of Civil Society
Functioning Democracy, Economy
Civil Society Assistance
?
Model of Impact of Media Assistance
![Page 31: Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d155503460f949e9aa1/html5/thumbnails/31.jpg)
31
Do Independent Media Foster Democratization?
• Surprisingly little empirical evidence has been gathered– No understanding of what conditions might be
placed on this relationship
![Page 32: Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d155503460f949e9aa1/html5/thumbnails/32.jpg)
32
Six Hypotheses1. The media-supremacist position, which holds that media
freedom and independence produce democracy.2. The democracy-supremacist position, which holds that
democratic reform determines and produces media freedom and independence.
3. The media-freedoms-are-an-element-of-democracy position, which argues that media freedoms are a part of democracy and, as such, have no causal force leading to democracy.
4. The null-effect position, which holds that there is no relationship between media freedom and democracy.
5. A media-freedom-hinders-democracy argument.6. A democratization-hinders-media-freedoms stance.
![Page 33: Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d155503460f949e9aa1/html5/thumbnails/33.jpg)
33
Media Reform
Pretransition PrimaryTransition
SecondaryStage
Mature Stage
Dem
ocra
tizat
ion Pretransition X
Transition X
Consolidation X
MatureDemocracy X
Media Reform
Pretransition PrimaryTransition
SecondaryStage
Mature Stage
Dem
ocra
tizat
ion Pretransition X
Transition X
Consolidation X
MatureDemocracy X
Rozumilowicz Stages of Media Reform Linked to Stage Theory of Democracy
![Page 34: Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d155503460f949e9aa1/html5/thumbnails/34.jpg)
34
Year of Press Freedom
Political Rights +1
Political Liberty +1
Political Rights -1
Political Liberty -1
1981 0.79 0.82 0.79 0.81
1982 0.80 0.84 0.82 0.85
1983 0.79 0.82 0.81 0.85
1984 0.80 0.84 0.82 0.87
1985 0.82 0.83 0.82 0.87
1986 0.79 0.83 0.82 0.87
1987 0.77 0.81 0.81 0.86
1988 0.84 0.80 0.83 0.86
1989 0.82 0.77 0.93 0.89
1990 0.83 0.77 0.94 0.88
1991 0.81 0.77 0.93 0.91
1992 0.82 0.84 0.90 0.88
1993 0.81 0.81 0.89 0.85
1994 0.86 0.88 0.88 0.90
1995 0.88 0.91 0.88 0.91
1996 0.88 0.90 0.90 0.91
1997 0.88 0.90 0.90 0.92
1998 0.89 0.91 0.91 0.92
1999 0.89 0.91 0.90 0.93
2000 0.89 0.90 0.91 0.93
2001 0.91 0.92 0.90 0.93
2002 0.94 0.94 0.93 0.94
2003 0.93 0.94 0.94 0.95
2004 0.93 0.94
Med
ia D
emoc
racy
Li
nk
![Page 35: Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d155503460f949e9aa1/html5/thumbnails/35.jpg)
35
What We Know About Media Freedom Measures?
• Strong evidence of reliability across time• Strong evidence of reliability across measures• Some evidence of validity
– Criterion• FH measures reflected changes in Warsaw Pact countries
– Construct• IREX MSI related in predicted way in analysis of impact of
hypercompetition on press performance
![Page 36: Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d155503460f949e9aa1/html5/thumbnails/36.jpg)
36
![Page 37: Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d155503460f949e9aa1/html5/thumbnails/37.jpg)
37
Professional journalism and GDP per media outlets
0
0,5
1
1,5
2
2,5
3
020406080100120140160180
GDP in USD millions per media outlet
Pro
fess
iona
l jou
rnal
ism
Uzbekistan
Kosovo
Kazakhstan
Georgia
Croatia
BulgariaBosnia &
Herzegovina
Belarus
Azerbaijan
ArmeniaAlbania
Russia
Romania
Moldova
Macedonia
Kyrgyzstan
UkraineTajikistan
Serbia
Relationship between Market Competition and Press Performance
![Page 38: Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d155503460f949e9aa1/html5/thumbnails/38.jpg)
38
Limitations of Media Freedom Measures
• Focus on structure of media system
• Limited evidence of performance
• Limited notion of media independence– Focus on independence from government– Do not recognize pressures of commercial
dependence
• Largely ignore audience
• Largely ignore “needs” of democracy
![Page 39: Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d155503460f949e9aa1/html5/thumbnails/39.jpg)
39
Additional Monitoring
• Develop checklist of needs of democracy– A democratic state is a state in which all citizens
have access to information about how the state operates.
– A democratic state is one in which citizens have the ability to communicate to each other.
– A democratic state is one in which conflicts are managed without resort to violence.
– A democratic state is one in which representations of members of the state are presented in a way that foster appreciation and understanding.
![Page 40: Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d155503460f949e9aa1/html5/thumbnails/40.jpg)
40
Relationship between Press Freedom and Confidence in Media
![Page 41: Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d155503460f949e9aa1/html5/thumbnails/41.jpg)
41
Media Assistance Trained, Skilled,
Motivated Journalists
Media Organizations that Facilitate and Distribute the Work of Journalists
Independent Media System
Information Needed for Functioning of Democracy
Institutions of
Civil Society
Functioning Democracy
Civil Society Assistance
?
Model of Impact of Media Assistance
![Page 42: Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad](https://reader030.vdocuments.us/reader030/viewer/2022032723/56649d155503460f949e9aa1/html5/thumbnails/42.jpg)
42