by Mary Joyce with John Kennedy for the Open Society Foundation | December 2015 (v 1.1) | Part 1

Image: Flickr/Very Quiet

VIDEO IMPACT ASSESSMENT

Closing the Impact Gap

Public Version

CONTENTS

Executive Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . slide 3

Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .slide 4

Recommendations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .slide 7

Methods. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .slide 9

Grantee Report Cards. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . slide 14

Appendices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .slide 54

This public version is anonymized to remove information identifying the nonprofit organizations being assessed and the foundation for which the assessment was prepared. The purpose of this anonymization is to protect the privacy of the entities involved and instead put the focus on the method of analysis and its presentation. Redacted content is covered with gray boxes.

About the Public Version

* Assessment Part 1 includes a 51% non-random sample of 37 video-producing grantees. Image: Flickr/Very Quiet

EXECUTIVE SUMMARY

•This assessment of 2009-2015 grantees* revealed a gap between video content and video impact.•Despite strengths in storytelling, production, and distribution, grantees often fail to benefit from their videos. •Strategy is often rudimentary. While the vast majority of grantees understand the need for a goal and audience, most videos lack theories of change, precise audience targeting, and systematic measurement of impact.•To meet these challenges, the Media and Narratives Division should adopt a population benefit strategy that would focus funding priorities and methods of proposal evaluation and grantee support on videos that aim to achieve concrete benefits for marginalized populations.

Image: Flickr/Very Quiet

FINDINGS

FINDINGSImpact is Little Understood & Poorly Measured

Lack of Consensus on Impact•Impact claims were diverse, indicating little consensus on what constitutes video impact.

•27% of grantees claimed that the video benefited the population on whose behalf their organization advocates, such as improved health services for Roma or kinder community treatment of HIV+ women.•14% claimed the video benefited their organization, such as helping them raise money or get invitations to speak at conferences.•32% of grantees claimed an effect on their audience, such as raising awareness or increasing understanding, though not a direct benefit to themselves or their population.•The remaining 27% made no claim of video effect or benefit, instead describing the scope of distribution or the quality of the video’s content.

Population BenefitOrganizational BenefitAudience Effect w/o BenefitNo Effect or Benefit Claimn=37

32%14%

Impact Claims

Shortage of Evidence•Many impact claims were unsupported by observable evidence, and none were measured systematically.

•Of the 27 grantees who claimed an effect or benefit from their video, only 63% supported that claim with any kind of observable evidence, such as audience self-reports of opinion change or an audience member taking a desired action.•The other 37% of that group did not support their impact claims with any evidence or confused evidence of distribution, such as online view counts, with evidence of impact (effects and benefits).•0% of grantees engaged in systematic measurement of impact. Evidence was ad hoc and anecdotal.

Note: See slide 55 for definitions of terms.

FINDINGSStrong Production/Distribution Hindered by Weak Strategy

Great Video Content and Production •Grantees are telling compelling stories and producing high quality videos.

•The overall quality of story and production is very high. •Selection of more compelling visuals and editing (to make them shorter and more conducive to online sharing) was a common suggestion for improvement.• Yet poor strategy often means that even excellent videos have little impact.

Distribution Shines Despite Lack of Focus•Grantees are creative and energetic in their distribution, yet lack focus on specific audiences.

•Grantees used a range of online and offline methods to distribute their videos. They used social media, webpages, and even temporary graffiti. They showed videos at workshops, conferences, trainings, public debates, and at meetings with decision-makers. •Their shortcoming was a lack of focus on audience. Of the 30 grantees with sufficient data to evaluate distribution, 70% did not connect their distribution methods to specific audiences they were targeting. •As a result, distribution was enthusiastic, but at times haphazard.

The Strategic Deficit•Behind the impact deficit lies a strategy deficit. Grantees have mastered only the basics.

•95% of grantees identified a goal and 86% identified one or more audiences, indicating widespread understanding of the most basic strategic principles of video advocacy.•Yet 32% are having problems identifying precise audiences, instead trying to reach the “general population” or “open public.” •Only 7 grantees (14%) were able to make an evidence-backed claim that their video created an audience effect that benefited either themselves or their population. •Outside of these 7, there was little evidence that grantees had in mind a video theory of change* that linked the audience effect of their video to benefits for the marginalized population.

* Definition on slide 55, model on slide 56.

Image: Flickr/Very Quiet

RECOMMENDATIONS

RECOMMENDATIONSAdopt a Population Benefit Strategy for Video

What is a Population Benefit Strategy for Video?•This new Media and Narratives Division strategy would inform funding priorities, proposal evaluation criteria, and grantee support.1.Funded videos would focus on achieving concrete improvements in the welfare of marginalized populations.2.Funded videos would have a credible theory of change that describes the role the video will play in achieving that benefit.3.Grantees would receive high- and low-touch support to gain the capacity to create video theories of change.

Prioritize Advocacy Over Promotion

•If population benefit is the goal, reconsider funding of promotional videos.

•The videos in this assessment included both advocacy videos, which make an argument for the improvement in the welfare of marginalized populations, and promotional videos, which tout past successes of an organization.•Promotional videos help grantees become more sustainable by acting as a fundraising tool, but their connection to population benefit is indirect. •Advocacy videos provide a more direct effect on marginalized populations because that is their explicit goal.

Focus on the Video’s Theory of Change

•Proposals should outline a series of causal steps that link video use to population benefit.

•Grantees may need help seeing the big picture, as video impact and theory of change are still unclear to many.

Provide a Suite of Support Services

•Different grantees will need different levels of support to get to video theory of change.

•For those with strong strategic thinking, more focused proposal feedback and visual aids (slides 12, 56) may be sufficient.•For those with weaker strategic thinking, technical assistance would be necessary. •Current strategic skills are summarized in the Grantee Report Cards, beginning on slide 14.

Consider a Broader Assessment

•This assessment is based on a 51% non-random sample of grantees who self-selected to report on their video impact.•Part 2 would analyze the impact of the other 49%, and give a more complete picture of grantee video impact.

IMPACTImage: Flickr/Tom Blackwell

FindingsImage: Flickr/Very Quiet

METHODS

Data & Model

DATA

THE DATA

Grantee Self-ReportsGrantees wrote the following 3 self-report statements about each video they submitted. All 3 statements were analyzed for this assessment.1) Description (Please provide a 50-100 word description of the video.)2) Use Statement (How did your organization use this video in your advocacy work?)3) Impact Statement (Did your video have a positive impact on your cause? If yes, what was that impact? If not, what prevented that impact?)

TextsVideos

Grantee-Created VideosA total of 72 videos were submitted to the Health Rights Online Film Festival and analyzed for this assessment.

2015 Health Rights Online Film FestivalAn online festival of grantee videos (filmfest.health-rights.org) provided the perfect opportunity for an assessment of video advocacy. Of the 73 grantees invited to submit a video, 37 (51%) responded and are included in this assessment. They submitted not only videos, but also self-reports on both how they used those videos in advocacy and the impact they realized from that use. Both the videos submitted and the self-reports describing those videos were analyzed for this assessment.

ANALYSTS

Videos analyzed by John Kennedy

Self-report texts analyzed by Mary Joyce

THE ANALYSIS

Mary Joyce used grounded content analysis to analyze all 72 sets of video statements submitted by grantees. This qualitative method involved reading and annotating the 217 texts to identify patterns (image above). John Kennedy viewed the 72 videos associated with these texts to make his assessment.

Goal Desired effect on the audience and how this e f fect wi l l lead to b e n e fi t s f o r t h e marginalized population.

Audience Individuals whose change of mind or action would have a benefic i a l e f f e c t on t he marginalized population.

StoryNarrative structures and characterization used to persuade the audience.

THE MODEL

Production Technical elements of the video, including sound, lighting, and editing.

Distribution Means of ensuring the audience sees the video.

ImpactExtent to which the desired audience effect and population b e n e fi t s o c c u r r e d , a s demonstrated by observable evidence.

For each grantee, analyses of both the video and texts are presented in an additive model of video advocacy impact. In this model, early choices on Goal identification and Audience specification lead to decisions about the Story a video will tell and the Production that will best express that story. The video is then Distributed according to the audience the grantee wishes to reach. Impact is evaluated by comparing the videos effect on the audience to the video’s initial goal.

Analysis: Mary Mary Mary MaryJohnJohn

cc

cc Mary Joyce, 2015

CAVEATSLimitations of the Assessment

• Selection Bias: The grantees self-selected to submit videos to the Health Rights Online Film Festival and to write the self-reports about those videos. As a result, the 37 grantees in this assessment represent a 51% non-random sample of the population of 2009-2015 video-producer grantees.

• Positive Skew: This self-selection likely resulted in a positive skew on the data. Those who participated are likely more actively involved in video advocacy than grantees who did not participate.

• Self-Reporting: The texts used in this assessment are written by the grantees themselves. One would expect a tendency to portray their own video work in a positive light.

• Repurposed Data: The texts were produced as part of an application for the film festival’s Impact Award. Grantees were asked to speak about their video’s impact in general terms. They were not asked to respond directly to the elements in the model presented on the previous slide.

IMPACTImage: Flickr/Tom Blackwell

FindingsImage: Flickr/Very Quiet 1

GRANTEE REPORT CARDS

1. African Palliative Care Association (APCA) 2. Association for Development and Social Inclusion

(ADIS)3. Association for Emancipation, Solidarity and Equality of

Women of Republic of Madeconia (ESE)4. Association for Supporting the Marginalized Workers

(Star-Star) 5. Center for Interethnic Dialogue and Tolerance Amalipe

"AMALIPE" 6. Centre For Advocacy And Research / Center for Health

and Social Justice (CHSJ) 7. Centro de Estudios para la Equidad y Gobernanza en

los Sistemas de Salud (CEGSS) 8. Ceva de Spus 9. Children of Georgia 10.Citizen Association SOLEM11.English Collective of Prostitutes 12.ENPUD, Alliance Ukraine 13.Eurasian Harm Reduction Network 14.Fundación Salud por Derecho 15.Georgian Young Lawyers Association (GYLA) 16.Gral Film 17.Health Education and Research Association (HERA) 18.Healthy Options Project Skopje (HOPS) 19.Hungarian Civil Liberties Union (HCLU)

20.International Committee on the Rights of Sex Workers in Europe (ICRSE)

21.Justice and Women Charitable Trust 22.Kenya Hospices and Palliative care Association

(KEHPCA) 23.Kenya Legal and Ethical Issues Network on HIV and

AIDS (KELIN) 24.Keystone Human Services Moldova 25.KHAM 26.New Zealand Prostitutes Collective (NZPC) 27.Pivot Legal Society 28.Rattapallax "DAVIDA" 29.Red Latinoamericana por el Acceso a Medicamentos 30.Roma SOS 31.Sex Worker Education and Advocacy Taskforce

(SWEAT) 32.Sexual and Health Rights of Marginalized Communities

Coalition 33.Sonke Gender Justice 34.Transgender Equality Network Ireland (TENI) 35.Transgender Europe 36.Uganda National Health Users'/Consumers'

Organization (UNHCO) 37.Ukrainian Community Advisory Board (UCAB) Patients

of Ukraine

THE GRANTEES

Report cards are arranged in this section in the above order.

Of the 37 grantees reviewed in the original version, a representative selection of ten are included here. They illustrate common challenges in making advocacy videos.

1. Filmmaking and Strategy are Difference Skills2. The Importance of a Clear Goal3. The Problem with “Raising Awareness”4. The Importance of Active Distribution5. The Problem with Targeting the “Public”6. Impact Without Evidence7. When Goal and Impact are Disconnected8. Media Coverage ≠ Impact9. Video Views ≠ Impact10. Simple Strategy Wins the Day

THE REPORT CARDS

The assessment of each grantee’s video advocacy is presented in a report card, with above elements.

ScoresThe grantee is scored on its mastery of each element of the model, as revealed through the videos and self-report texts included in the assessment.

QuotesGrantee quotes from the self-reports are used whenever possible to explain the score.

RecommendationProvides advice directly to the grantee, based on the model analysis.

VideosV i d e o s i n c l u d e d i n t h e assessment are listed, with links.

ImageA still from one of the videos included in this grantee’s assessment

Goal: I*“[P]eople with intellectual or psychosocial disabilities who a l s o h a v e p h y s i c a l disabilities... are confined... for two main reasons: there is... societal prejudice against them... and their physical e n v i r o n m e n t s d o n o t accommodate” them. This is great problem analysis. The next step is to figure out who has the power to address either reason and decide what action you want them to take. Then you’ll have your goal.

Audience: CT h e “ p u b l i c ” i s identified, but this is an audience too v a g u e t o b e effectively targeted.

Story: AFantastic. The portraits are excel lent. The i n t e r v i e w s w i t h spokespeople feel a little jarring - as they interrupt the flow of the por traits. Does Judy ever visit these institutions? I'm just looking for a visual way to link her to the very strong video.

Distribution: A- Very active and diverse distribution methods - “Blog on OSF website, O S M H I F a c e b o o k , Y o u T u b e , V i m e o , International Film Festivals' screenings and public d i s c u s s i o n s , s h a r i n g b e t we e n N G O s , T V broadcasting “ yet without an audience identified (or an action desired of that audience) it’s hard to know if these methods were well-selected.

Impact: C-The impact described - “Pub l i c awarenes s , public discussions...” - seems to indicate your confusion as to what the film was meant to ach ieve and which e f f e c t s w o u l d b e e v i d e n c e o f t h a t achievement. Also, these are methods of distribution, not impacts (effects) of distribution.

Production:ASuch great filming and visual choices.

RecommendationThis is a fantastic video. Yet there’s a disconnect between filmmaking and strategy. The filmmaker is clearly talented at production, storytelling, and distribution. But without clarity on goal or audience it’s very hard to make an impact... and this is what happened to this film. The next time you work with an NGO, make sure they tell you exactly who they are trying to influence and what successful influence would look like to them. They may not be clear on the answers, so you may need to press them. The result, however, will be a more effective advocacy video.

Videos in Assessment: New Day (2015): video.health-rights.org/index.php/list/product/view/1/55.html

* Incomplete data

GRAL FILMFilmmaking and Strategy are Different Skills1

This talented filmmaker was at a loss when it came to strategy.

CENTER FOR INTERETHNIC DIALOGUE AND TOLERANCE AMALIPE (AMALIPE)

Goal: A-“We use the movie to make RHSP popular and to advocatte [sic.] for its sus ta inab i l i t y” - The sustainability goal is clear, precise, and beneficial. Focus on that. Being “popular” is not precise and its benefit is also unclear.

Audience: B+This is a great example of how a clear goal leads to clear audiences, but unclear ones are a dead end. “ [ R ] e l e v a n t n a t i o n a l institutions” that could provide funding are a clear a u d i e n c e f o r t h e s u s t a i n a b i l i t y g o a l . Unsurprisingly, there is no goal for the vague goal of p o p u l a r i t y. A n d t h e “advocacy work at local level” - what is that about?

Story: A-It looks more like a mini documentary about your work than an advocacy video as a tool. Production: A

Really well filmed.

Distribution: I*There’s little information on how you showed the video to the funding institution. Maybe this was because you thought a f ace- to- face meeting was bor ing or unworthy of mention. But distribution strategy does not need to be high-tech. It just needs to put your video together with its intended audience. And distribution for popular ity? And the advocacy work? Need more info!

Impact: CImpact assessment should link back to the goal using evidence to make an argument, but that didn’t happen here. There’s no mention of whether the video helped you get funding. You say that the video helped you popularize your program, but provide no evidence. You mention again advocacy at the local level, but this is a means of achieving impact, not the impact itself.

Recommendation“Community Monitoring” is a very well filmed and edited video. However, it looks more like a mini documentary about your work than an advocacy tool. It’s hard to tell from the incomplete information, but it seems that you made this strategic analysis harder than it had to be. You needed funds. You created a video illustrating your past work to convince “relevant national institutions” to continue funding that work. That’s a completely acceptable video strategy! It just needs a means of measurement. Did you get funded? Did you receive more money than you had before you were able to use video as a promotional tool? Measurement of impact just means using observable evidence to evaluate the effect your video on your audience. If you keep your strategy simple it will be easier to both create video strategies and measure their impact.

Videos in Assessment: Videos in Assessment: The White Swallows III (2015) video.health-rights.org/index.php/list/product/view/1/284.html; Community Monitoring (2015) video.health-rights.org/index.php/list/product/view/1/285.html

* Incomplete data

2 The Importance of a Clear GoalA clear goal is the focus and foundation of video strategy.

Goal: B“To raise awareness toward disabi l i ty issues.” - While a common goal of advocacy v ideos, raising awareness is not enough. What do you want to happen after people are aware? Sketch the full theory of change to keep yourself focused.

Audience: I*“The v ideo has a posit ive impact on a u d i e n c e - t h e audience is convinced” - There is an awareness that the audience is an impor tant par t of video strategy, but there is no actual identification of who the audience is.

Story: AVery good storytelling. Continue to push your team - what are the best visuals and most impor tant messages.( “A l l Boys” i s my favorite video in the assessment - MJ)

Distribution: A-The video was actively distributed on TV, social media, and through public events. You were also attentive to evidence of audience exposure, such as comment s and l i ke s . However, without a clear idea of who should be seeing the film it’s hard to tell whether the right distribution methods were used.

Impact: B-I appreciate your honesty in saying you felt you couldn’t measure impact directly. This is because you did not identify a specific audience, so you didn’t know where to measure. If you select an audience, you will be able to measure impact because you will know where to seek evidence of effect. Remember that you cannot say a video had a “positive impact” if you have no evidence of audience effect.

Production: A-Nice simple filming and titles.

Grantee Recommendation Your film has a clear and excellent message and the filming is simple and well-done. Now it’s time to build on that by working on the other elements of your strategy so you can get a bigger impact next time. The next time you create a video, create your strategy first. Here are the basics of linking all the elements above into a clear strategy. The purpose of any advocacy video is to persuade an audience, either to take an action or to change their mind about something. The action/mental change they are persuaded to undertake is the goal. The argument the video makes to persuade them is the story. How you will make sure the audience sees the video is distribution. When you measure impact you are looking for evidence as to whether the audience took the action (easier to observe) or changed their minds (harder to observe). If you follow this strategic process you will get more value out of your future videos - and the ones you’ve already made. Good luck!

Videos in Assessment: Institution or community? (2015) video.health-rights.org/index.php/list/product/view/1/318.html; Returning Home (2015) video.health-rights.org/index.php/list/product/view/1/320.html; I wish all boys from residential institutions could live in a family (2013) video.health-rights.org/index.php/list/product/view/1/319.html

* Incomplete data

KEYSTONE HUMAN SERVICES3 The Problem with “Raising Awareness”“Raising awareness” is too vague a goal for precise and effective strategy.

Goal: B“[T]o raise awareness around sex workers human rights and health issues, as well as the need for change in law” - You identify the direct effects of the video (raising awareness) and the ultimate impact you are aiming for (change in the law), but you don’t link them. Will raising awareness lead directly to changing the law? If not, what are the intermediate steps and how will you move through them?

Audience: I*“Change in the law” i m p l i e s a g o v e r n m e n t a l audience, though none is specified. The individuals who would be targeted to sign the e-petition are also no specified.

Story: AVery clear, well done.

Impact: C-You write that you cannot measure impact yet, but since you do not even note the number of signatures on the e-petition, I suspect the number is lower than you would have wished, rather than being uncountable. You also “hope” the video “reach[es] people and bring[s] about change,” which is oddly passive given that it is the aim of your own campaign to achieve these ends. Don’t give up. You can make it happen!

RecommendationOnline advocacy is not a silver bullet. It just implies a new set of tools. Making an (admittedly high quality) digital video does not imply virality. Placing that video online does not guarantee engagement in an online tactic like e-petition signing. In fact, because there is so much content online vying for our attention, active promotion to clearly specified audiences is critically important. Although I sense that this effort might have been discouraging, you’ll get a better result if you engage in more intentional video strategy. Here are the basics: The purpose of any advocacy video is to persuade an audience to take action (or change their mind). The action they are persuaded to do is the goal. The argument the video makes to persuade them to take that action is the story. How you will make sure the audience sees the video is distribution. And when you measure impact you are looking for evidence as to whether the audience took the action (or changed their mind, though that is harder to measure). Good luck!

Videos in Assessment: Decriminalise Sex Work Now!(2015): video.health-rights.org/index.php/list/product/view/1/41.html

* Incomplete data

SEX WORKER EDUCATION AND ADVOCACY TASKFORCE (SWEAT)

Distribution: BThe video was posted on its own page on your website, but this really isn’t enough. There i s no m e n t i o n o f a c t i v e outreach (for example, via listservs and social media) to drive traffic to the page. You also do not specify the audiences you were trying to reach with each method.

Production: A G o o d s i m p l e animation.

4 The Importance of Active DistributionSimply posting a video online isn’t enough to engage an audience.

CEVA DE SPUS

Goal: A-“ [T ]o c h an ge t h e mentality of the public towards people with intellectual disabilities... t o p r o v e [ t h e i r ] abilities" - Ambitious, but I’d like to see the video linked to a more s p e c i fi c a n d measurable action from the audience. Perhaps t h e r e a c t i o n yo u w a n t e d f r o m t h e U n i v e r s i t y admin is tr ator s you mention later?

Audience: B“[T]he open public” - Trying to target the public is a common mistake. It’s too broad t o b e t a r g e t e d effectively. “[D]ecision makers... civil society o r g an i z a t i o n s a nd donors” - These are a lot clearer.

Story: A-Wonderful videos - so clear why we should all have equal rights. Think about more visuals. I k now t he s e we r e c rea ted a s s imp le portraits - but I also know that Zoli is very skil led at captur ing small moments which show the personality of a person.

Production: A Good filming, Zoli!

Distribution: A"[U]ploaded on social media and website... different events... in press releases.... projections at different events, round tables, conferences." D iver se and ac t i ve d i s t r i bu t ion . Wou ld benefit from a bit more l i n k a g e b e t w e e n distribution method and aud i ence . Wh i ch audiences were you hoping to reach with e a c h m e t h o d o f distribution?

Impact: A“The west University started an accessibility project, they make efforts to fulfill all of our demands... For sure this video had a small contribution to the declaration of Ministry of Labor regarding starting the deinstitutionalisation process” Though I wish there were some evidence behind the claim about the MoL, the p o l i c y c h a n g e a t t h e University is a great example about how focused pressure on specific institutions can be a starting place for broader social change.

RecommendationWe are both huge fans of this video series. You make it so obvious why no one should be institutionalized permanently. You’ve used video to realize concrete advances towards broad social change. Next time you make a video, be as specific as you can about who you are trying to persuade and what you want to persuade them to do or think. This will prevent you from trying to target overly broad audiences like “the open public.”

Videos in Assessment: There is life outside the walls (2013) video.health-rights.org/index.php/list/product/view/1/334.html; Andrei (2012) video.health-rights.org/index.php/list/product/view/1/335.html; Raluca (2011) video.health-rights.org/index.php/list/product/view/1/336.html

5 The Problem with Targeting the “Public”This grantee made the common mistake of selecting an audience too broad to target effectively.

Goal: A+A very clear goal: to help NGO’s in other countries learn from N Z P C ’ s b e s t practices as they m a k e s i m i l a r dec r im ina l i z a t ion effor ts. Clear ly connects the effect of the video on the audience (learning) to the desired impact ( i m p l e m e n t i n g dec r im ina l i z a t ion internationally).

Audience: A+“ [ S ] e x w o r k e r o r g a n i s a t i o n s internationally” - A str ategica l ly log ica l audience given the goal.

Story: B+The message is clear as a report about the collective, but less of an advocacy message.

Impact: B“We believe the video has had a positive impact as sex workers in other countries are using the information within it to build support for decriminalisation” - This could be true, but there’s no evidence provided to support your claim. (How do you know it is being used?) This is the only place where you video strategy falls short. The rest is excellent.

Production: B+Clear sound and visuals.

Recommendation

You are a great example of the strategic use of a promotional video. It is very rare for a video promoting an organization’s past work to have a clear and logical goal, audience, and distribution method, but you’ve done it. Congratulations! The place for you to focus now is on measuring your impact. You could easily get a sense of the impact of your video by following up with the people who ordered the DVD to see how they used it or by following up with anyone who contacted your through the video website.

Videos in Assessment: Decriminalisation of Sex Work in New Zealand (2015): video.health-rights.org/index.php/list/product/view/1/314.html

NEW ZEALAND PROSTITUTES COLLECTIVE (NZPC)

Distribution: A+Your distribution methods makes a lot of sense given your audience and goal. The video is available via DVD (perhaps for NGOs w i t h o u t i n t e r n e t connection) as well as on a purpose-built website. I also like that the video was actively distributed to other NGOs, rather than simply letting it sit on a website.

6 Impact Without EvidenceThis grantee’s strategy was really strong until the impact assessment stage, when they failed to measure the effect of their work.

ENGLISH COLLECTIVE OF PROSTITUTES

Goal: B“1. to publicize our legal victories... 2. To build the m o v e m e n t f o r decriminalisation.“ - The value of the first goal is unclear. What future benefit will result from publicizing these past victories? The second goal is a bit vague. What exactly does “build the movement” mean here? How wi l l the v ideo accomplish this?

Audience: C+”[M]any individuals and organisations” - Who exactly? Why is it important that they h e a r a b o u t y o u r victory? What role do t h e y p l a y i n t h e movement you are building?

Story: BIt's a very good video - but we need sex worker vo ices . Without their voices we can't be 100% sure this is what they are asking for.

Distribution: B “[S]howing in December 2014” - Who was in the a u d i e n c e t h a t yo u wanted to persuade? “sent the film to our network” - To what end? What actions were you hoping for as a result?

Impact: B-“We...have been invited t o s p e a k a t fi l m showings on a number of occasions.” - This benefic i a l ou t come seems more by chance than by design. There’s a disconnect between goal and impact.

Production: AGood filming - both interior and exterior (protest) shots are captured clearly.

RecommendationI know you are awesome activists and I love both the spirit and the method of your activism, so I am going to be a little tough on you (as in touch love) regarding your video strategy. I get the sense that there wasn’t really a clear vision of what benefit you were hoping to get out of this video. As a result, the audiences targeted, the distribution method used to reach those audiences, and the impacts recorded seem rather haphazard. Bottom line - advocacy video is about persuading and audience to take an action (often mediated by a change in state of mind) and impact is about measuring the extent to which that action was taken. If you are specific about the action you want from the beginning, when you set your goal, the rest of the strategic process will be a lot easier.

Videos in Assessment: Soho Trot (2015): video.health-rights.org/index.php/list/product/view/1/312.html

7 When Goal and Impact are Disconnected

A big goal (movement-building) leads to minimal benefits (speaking invites) without clear causal steps connecting the two.

EURASIAN NETWORK OF PEOPLE WHO USE DRUGS (ENPUD), ALLIANCE UKRAINE

Goal: B“ F o r d r a w i n g attent ion to the problem of clients of OST program in Donbass“ - Focus more on the action y o u w a n t t h e audience to take. Whose attention? To what end?

Audience: I*“ [ D ] r a w i n g a t t e n t i o n ,” b u t whose attention? In t h e a b s e n c e o f audience analysis, realizing impact is very difficult.

Story: AYou’ve done a really good job eliciting stories from the protagonists. Their d i s cus s ions o f the i r families and emotional lives humanize them. I would have liked to see s o m e o n e w h o i s a recovering addict give a tour of the city he knew as an addict.

Distribution: B+ “A lot of famous mass-media, such as BBC... key events of EEC region” - Though getting coverage from a media partner with a wide audience is impressive, this is only meaningful if the BBC’s audience is the audience you want to reach to achieve your goal. Also, remember that reaching an audience is evidence o f d i s t r i bu t i on , no t evidence of impact.

Impact: BIt’s possible that the media exposure resulted in actions helpful to the OST clients, but there’s n o e v i d e n c e . Remember that views are not impact, action is. Next time, think about who has the power to help and then target them specifically.

Production: A-The filming is of good quality, but the visual choice (the candle, for example) could relate to the subject matter more directly.

RecommendationYour passion about this topic really comes through and the suffering of the film participants is clear. For a crisis as challenging as this one, filmmakers need to connect to other partners who are capable of carrying out more direct (in-person) advocacy with audiences capable of providing practical assistance to those in need. If filmmakers are not currently connected to these actors, OSF could likely assist in making those connections. Massive humanitarian crises cannot be addresses by filmmakers working alone. They need help.

Videos in Assessment: Slow Death in Ukraine (2015) video.health-rights.org/index.php/list/product/view/1/248.html; Live or Die in Donbas? (2015) video.health-rights.org/index.php/list/product/view/1/249.html; The First Crimean Victims(2015) video.health-rights.org/index.php/list/product/view/1/268.html; The Right to be Dependent - Irina (2011) video.health-rights.org/index.php/list/product/view/1/67.html

* Incomplete data

8 Media Coverage ≠ ImpactMedia coverage is a mechanism of distribution, not a measure of impact.

OST = opioid substitutions therapy

Goal: A-“ [ D ] e m a n d s f o r decriminalisation” - This is a good campaign goal (or e ve n o r g a n i z a t i o n a l mission), but I’d like to see a goal specific to the video. What role should t h e v i d e o p l a y i n convincing politicians to make this change? Do t h e y n e e d s p e c i fi c information? A more general emotional appeal to empathy to lay the groundwork for later policy arguments?

Audience: B“[O]ur contacts... other networks and organisations” - I’d like to see more clarity about why e a c h a u d i e n c e needs to see the video.

Story: B+The fact that sex workers were willing to be filmed without anonymization is very rare. Yet the power of their self-advocacy is muted by the lack of a narrative arc.

Distribution: B+ Online methods seems to be the primary means of distribution, including an online training and social media. Again, there doesn’t seem to be much clarity on who needs to see the video, thus also a lack of clarity on how to reach that audience.

Impact: B-"I don't know. The video was seen 3,500 times (July 2015) so we assume it has allowed many peop l e t o make themselves familiar with our message." - I appreciate your h o n e s t y a b o u t y o u r uncer tainty in measur ing impact. I also like that you are look ing fo r obse r vab le evidence This is the right direction. In the future, note that views are a measure of distribution, not impact. What did you want people to do after they received your message?

Production: A-A nice use of natural light. Good looking interviews - but think about visuals that can also tell a story.

RecommendationYour video contains truly excellent self-advocacy from a range of sex workers across the European region. Next time you create a video, create your strategy first. The purpose of any advocacy video is to persuade, so a video strategy needs to identify who needs to be persuaded (audience), what they need to be persuaded to do (goal), the argument the video will make to persuade them (story), how you will make sure the audience sees the video (distribution), and how you will know whether the audience persuasion took place (impact measurement). In particular, try to narrow down your goal to a specific change in state of mind or action needed from a specific audience, which will further your larger goal of decriminalization. This focus will help you develop a distribution plan linked more directly to specific audiences and you will have a clearer idea of whose reaction to the film you need to measure. Remember that it’s easier to measure action than changes in state of mind, so see if you can link the change you want from your audience into an observable action.

Videos in Assessment: Sex workers are the solution not the problem - Sex work, HIV and human rights (2014): video.health-rights.org/index.php/list/product/view/1/303.html

INTERNATIONAL COMMITTEE ON THE RIGHTS OF SEX WORKERS IN EUROPE (ICRSE)9 Video Views ≠ Impact

Though they are easy to measure, video views are evidence of distribution, not of impact.

Goal: A+“[T]o raise surrppor t [sic.]” among “families [of] women living with Hiv/Aids” - A modest goal but one that is clearly tied to the video. “[T]o raise community awareness to counter stigma” - Clearly links the ef fect of the v ideo (raising awareness) to b e n e fi c i a l i m p a c t s (reducing stigma, raising suppor t ) . L i nk i ng awareness raising to a broader purpose is so rare - great job!

Audience: A+“ [ C ] o m m u n i t y members... families [of] women living with Hiv/Aids”” - The audiences are specific and are clear ly tied to the goal. Excellent!

Story: B+Keep trying to find peop l e who a re willing to have their f a c e s s h ow n o n camera. I know this is a sensitive subject, but there will be some champ ions who want to tell their story.

Distribution: A+ “The video was viewed by ove r 1000 commun i t y m e m b e r s i n t h e i r homesteads where trained c o m mu n i t y vo l u n t e e r s facilitated discussions on issues raised by women in the video.” - An excellent e x a m p l e o f l i n k i n g d i s t r ibut ion method to audience in order to address a specific goal. Also excellent use of quantitative data as evidence. Despite the fact that the video is digital, distribution is all offline. This is exactly what what needed to reach your rural audience.

Impact: A"The women involved in making the video and in facilitating the discussions reported qualitative changes with families showing increasing support to women living with Hiv/Aids.” - Using qualitative data (oral reports), this impact links back directly to the initial goal of the video. Again, it is rare to see this type of clear linkage. It would be nice to see some kind of impact assessment of the awareness-raising/stigma-reducing goal, and there i s some confus ion between distribution (views) and impact (action), but this is really good work.

Production: A-Good visuals. Think about shor tening the video - a shorter video wil l often be more powerful.

Recommendation

Congratulations! Your video has the strongest strategy of all the grantees in this assessment! Your goals, audience, distribution, and impact are all clearly linked. The use of evidence (views, oral reports) to support your impact claims is also excellent. You should be teaching other NGOs how to create video strategies!

Videos in Assessment: The Triple Burden (2015): video.health-rights.org/index.php/list/product/view/1/302.html

JUSTICE AND WOMEN CHARITABLE TRUST10 Simple Strategy Wins the DayThis grantee’s simple strategy with clear causal linkages led to a beneficial outcome clearly connected to their use of video.

IMPACTImage: Flickr/Tom Blackwell

FindingsImage: Flickr/Very Quiet

APPENDICES

• Advocacy Video: Recording of moving images designed to benefit a population or individual whose interests are marginalized or under-represented.

• Audience: Individuals whose change of mind or action would have a beneficial effect.• Benefit: Concrete improvement in quality of life.• Distribution: Means of ensuring the audience sees the video. • Goal: Desired outcome. Video goals should reference both the direct effect on the audience and how this

effect will lead to benefits for the grantee’s population of interest.• Impact: Extent to which the desired audience effect and population benefits occurred, as demonstrated by

observable evidence. • OST: Opioid substitution therapy, a form of harm reduction in which users of opiates, such as heroin, are given

substitution drugs, such as methadone, to manage their addictions.• Production: Technical elements of the video, including sound, lighting, and editing. • Promotional Video: Recording of moving images touting past successes, often for the purposes of gaining

funds for future work.• Story: Narrative structures and characterization used to persuade the audience to undertake an action or

change of mind.• Strategy: A plan for achieving a goal.• Theory of Change: A series of causal steps that explains why it is logical that the actions in the strategy will

achieve the goal. While the strategy describes the actions of the advocate, the theory of change takes into account the broader context in which the advocate is operating.

GLOSSARYVideo Advocacy Terminology

VIDEO THEORY OF CHANGEBasic Model

Desired

Realized

MarginalizedPopulation Audience

cc Mary Joyce, 2015

Goals & Planning

Impact and Evaluation

Campaign Goal

Desired benefit for marginalized population

1

Video Goal

Desired effect on audience

2

34

Video Impact

Realized effect on audience

Campaign Impact

Realized benefit for marginalized population

Start goal-setting here.

End evaluation here. Campaign Video

Compareto evaluate

Compareto evaluate

Audience action

ID audience that can

help population

cc

Desired

Realized

MarginalizedPopulation Audience

cc Mary Joyce, 2015

Campaign Goal: Desired benefit for marginalized population

1 2

34

“Stop Sending TB Patients to Prison” by KELIN

Video Goal: Desired effect on audience

Video Impact: Realized effect on audience

Campaign Impact: Realized benefit for marginalized population

VIDEO THEORY OF CHANGEExample: KELIN

cc

IMPACTImage: Flickr/Tom Blackwell

FindingsImage: Flickr/Very Quiet 1

THANK YOU!

Mary Joyce | contact@maryjoyce.comVideo Strategy Analysis, Findings and Recommendations, Information Visualization and Models

John Kennedy | Video Content and Production Analysis | 505050john@gmail.com