Evaluating the effectiveness of social media in promoting audience engagement
Jennifer BerktoldSenior Study Director
Westat
National Conference on Health Communication, Marketing, and Media, Atlanta, Georgia, August 11, 2009
The Boss of Me case study
“I don’t want no drama in my life,
even though we have a little bit, but
no more letting people control you.
That’s drama, because then you
become something that you’re not.”
-- Mary J. Blige
INTIMATE PARTNERVIOLENCE
Teens don’t achievetrue Intimacy
Partners are constantlybeing redefined making the word less relevant Reducing violence isour ultimate goal butstrategy should be prevention
EARLY RELATIONSHIP CONFLICTS
Formative years should be emphasized
Relationship is a more open term reflective of shifting expectations
Conflicts can be a frequent precursor to violence
Reframing the Issue
Five key strategies from research for reaching teens (ages 14 – 18, both genders, all races and ethnicities) on this issue:
2) Target the “drama” with humor and authentic teen language
3) Promote self-empowerment
4) Offer tools and resources to “dial down the drama”
5) Leverage teens’ organic media culture
1) Take an empathic and non-judgmental stance
The Strategic Approach
Being the boss of you means you’re in complete control of you at all times in your relationships. It means you refuse to submit to your own feelings of anger, fear, jealousy or insecurity. When you allow your emotions to negatively influence your actions, you lose control. And when you lose control, you’re not the “boss of” anything.
BOM (“Boss of Me”) is dedicated to helping young people stay in charge of their own relationship drama.
BOM provides tools to help teens maintain and manage their cool, instead of losing it, during a relationship conflict.
BOM or “Boss of Me”
Organizational Partners
Funder
Managing the Campaign
Brand Management
Evaluation
Evaluating BOM
Research goals
• Identify delivery mechanisms that engage young people with BOM
• Understand how young people engage with BOM– Gender differences
• Understand how the campaign impacts its target audiences
Intended uses of BOM evaluation results• Learning, best practices
• Communicate with the teen dating violence prevention community– Effective delivery mechanisms to reach teens– Engaging messages / content– Appeal to new audiences?
Basic model of an outcome evaluation
Products from activities
Web traffic
Number of radio shows developed
Number of ads
Number of events
Reach
Inputs OutcomesActivities Outputs
Resources
Money
Staff and volunteer time
Facilities
What is done
Website
Radio show
Ads / PSAs
Community events
Changes for participants
Increased knowledge
Attitudinal change
Behavior change
Outputs, Outtakes, Outcomes
Outtakes
• Awareness• Buy-in• Engagement• Participation
Outcomes
• Knowledge change• Attitudinal change• Behavior change
Outputs
• Products• Activities• Reach
New Logic Model, Simplified
Products from activities
Web traffic
Number of videos developed
Number of ads/ad buy
Number of events
Reach
Inputs OutcomesActivities Outputs
Resources
Money
Staff and volunteer time
Facilities
What is done
Tools development
Website development
Ads / PSAs
Community events
Contests
Changes for participants
Increased knowledge
Attitudinal change
Behavior change
Outtakes
Participation
Creative content
Conversation content
Tone of conversation
“Friending”
Texting opt-in
Approach
Heard of BOM but hasn’t engaged Radio ads / PSAs Outreach at school
Our challenge…
Engaged with BOM BOM411 site visits Social media friends Text messaging Community events User-generated content Hasn’t heard
of BOM
5-Point Evaluation Plan
• Monitoring• Content analysis• Web survey of teens• Pop-up survey at BOM411• Visits to community events
Web Monitoring
BOM Sites• BOM411• Facebook• MySpace
(2 sites)• Twitter• YouTube
Outside BOM• RSS (e.g., Google
blogs, Google news)
Data• Google analytics• Comment text• Comment analysis
• Articles/posts with BOM mentions
17
Content analysis
Subjects• Posted content on
BOM411• Posted content on
social networking sites
• User-generated content– Skateboard
designs– Rap lyrics
• Outside buzz
Measures (some examples)
• Type of conversation
• Affect • Attributes• Type of information
shared
Other Monitoring
Subjects
Radio
Cell phones
Voicemail line
Measures (some examples)
GRPs
Number of subscribers
Number of texts
Number of phone messages
Web survey of teens
Sample• Teens 14-18 in SF
Bay Area and Los Angeles
• Recruited from panel that agreed to take surveys online
• Parental consent obtained
• Knowledge Networks
Measures• Self-reported
exposure to campaign
• Awareness of relationship drama among peers
• Attitudes toward / relationship with BOM
• Self-reported attitude/behavior change
Pop-up survey at BOM411
Sample• Random selection
of visitors to BOM411
Measures• How found out
about website• Engagement with
website content (e.g., rating different parts of website, would you recommend to others)
Visits to community events
Sample• Random intercept
interviews
• Interviewer observation
Measures• How found out
about event• Why attending the
event• Activities performed
at the event• Description of event• Estimated
attendance
Timeline
Task Date
Site Launch August 24
Monitoring Launch thru February 2010
Content Analysis Launch thru February 2010
Other monitoring Launch thru February 2010
Pop-up survey September 2009, January 2010
Community events October 2009
Web survey of teens October /November 2009
Questions