measuring impact-methods and metrics

Measuring Impact: Methods and Metrics for Face-to-Face Classes and

Social Media Outreach

Barbara O’Neill, Ph.D., CFP®, CRPC®

Distinguished Professor, Rutgers University

oneill@aesop.rutgers.edu

Are You Evaluating Your Educational Programs?

If so, how?

The Million Dollar “So What?” Question…

At the end of the day…did your educational program make a difference?

How do you know?

• What gets measured gets funded

• With evaluation results, you can…

– assess impact of programs on learners

– see if you accomplished what you planned

– know if a program was “worth it”

– celebrate success and learn from failure

– make informed decisions to improve, hold, or fold programs

– promote your program and win public support

We are in an “Accountability Era”

Consider a Third Party Program Evaluator

• Some funders require this (e.g., FINRA Investor Education Foundation)

• No potential for conflict of interest by one entity performing both the education and evaluation role

• Takes evaluation “monkey” off busy educators’ back

• Build into grant funding requests

• See http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html

• Begins with the end in mind

• Explains what a program is and what it will accomplish

• Shows relationships between inputs, outputs, and outcomes

Introducing the Logic Model

Building a Strong Financial Education Program

INPUTS OUTPUTS OUTCOMES

Program

investments

Activities Participation Short Medium

What

We

Invest

What

We Do

Who We

ReachWhat Results

SO WHAT??

What is the VALUE?

Long-

term

Staff

Money

Time

Volunteers

Partners

Equipment/Technology

Policies

Research

INPUTS

OUTPUTSWhat We Do Who We Reach

ACTIVITIES

Assess needs and assets

Design curriculum

Educate students

Conduct workshops

Facilitate learning groups

Sponsor conferences

Work with the media

Partner – collaborate

PARTICIPATION

Participants

Clients

Customers

Users

Groups

Reactions - Satisfaction

OUTCOMESWhat Results for Individuals, Organizations, Communities..…

SHORT

Learning

Awareness

Knowledge

Attitudes

Skills

Opinion

Aspirations

Motivation

MEDIUM

Action

Behavior

Practice

Decisions

Policies

Social action

LONG-TERM

Conditions

Human

Economic

Civic

Environment

OUTCOMES

Commonly Measured Items That Are Not Outcomes

• Participant satisfaction

• Number of people taught

• Units of education completed

• Number of events held

• Time and money spent

• Level of effort

Question

What are some outcomes for your educational programs?

Impact Evaluation Data Collection Methods

• Surveys (Paper or Online)

– Post-evaluation only (short programs)

– Pre- and post-evaluation

– Follow-up (e.g., 3 months later)

• Focus groups

• Interviews

• Observations

• Tests of knowledge/ability

• RARE: Control groups and longitudinal studies

Typical Survey Questions

• General reactions to the program

• Changes in knowledge

• Changes in motivation, confidence, and abilities

• Intended changes in behavior

• Actual changes in behavior

• Future programming needs and preferences

• Demographics of participants

• Qualitative/open-ended responses

Post-Then-Pre (Retrospective) Evaluation

Method • Compares knowledge and attitudes before and

after a financial education intervention

• Administered once at the end of the intervention

• Helps avoid biases due to people thinking they know more than they know

Case Example: Humpty Dumpty

“Humpty Dumpty sat on a wall,

Humpty Dumpty had a great fall,

All the king’s horses and all the king’s men

Couldn’t put Humpty together again”

So Who/What Was Humpty Dumpty?

Probably

NOT what

you think!

Post-Then-Pre (Retrospective) Evaluation Method

• Why? Because people don’t know what they don’t know!

• Who knew what Humpty Dumpty really was?

• Five point rating scale

– 1 = Strongly Disagree

– 5 = Strongly Agree

• Post: After listening to Barb, I know the history of the Humpty Dumpty nursery rhyme

• Pre: Before listening to Barb, I knew the history of the Humpty Dumpty nursery rhyme

Post-Then-Pre Evaluation• Also known as a “Retrospective” evaluation

• Helps identify changes in knowledge, attitudes, and behavior

Another Post-Then-Pre Evaluation Example

Big Advantage: Avoids the problem of learners under-estimating what they don’t know before a program

Critical Incident Technique

• Qualitative evaluation research method where program participants tell personal stories

• Often used with “Train the Trainer” programs for professionals (e.g., teachers, librarians)

• Incidents are categorized and deemed successful (positive results) or unsuccessful (negative results)

http://www.joe.org/joe/2013june/tt2.php (Journal of Extension article)

More About the Critical Incident Technique

http://www.adb.org/documents/information/knowledge-solutions/the-critical-incident-technique.pdf

•Ask subjects to describe- through interviews-incidents that they handled well or poorly (need not be spectacular events)

– Example: How did library staff handle patrons’ personal finance questions?

•Provides rich personal perspectives

•Pre-intervention and Post-intervention comparison of number and type of incidents

Critical Incident Technique Example: NYPL Grant

Program Title: Money Matters

CIT Questions in NYPL Financial Education

Evaluation• Think about your experiences helping patrons with personal finance

questions. Remember a time when you had a successful experience helping someone with these types of questions. Please write down what happened.

• What made this a successful, positive experience?

• Think about your experiences helping patrons with personal finance questions. Remember a time when you had an unsuccessful experience helping someone with these types of questions. Please write down what happened.

• What made this an unsuccessful or challenging experience?

CIT Summary in Final NYPL Project Evaluation Report

“The change in the percentages of the types of incidents is indicative of change in attitudes and abilities. Also, a new category appeared in the successful incidents, Increased Knowledge/Confidence/Satisfaction, which was not present in the Pre-training Survey. Furthermore, for the unsuccessful critical incidents, the category Lack of Training/ Knowledge which was the most frequently seen theme for the Pre-training Survey was not found at all in the Post-training Survey.

These findings resonate with quantitative findings and provide confirming evidence that the Money Matters training has been successful in improving participants’ knowledge of personal finance and their ability to be successful in handling patron inquiries. Additionally, staff members provided eloquent testament in their qualitative responses that their skills have been enhanced and their attitudes and behaviors became more positive for ably handling personal finance queries” (Radford, 2013).

Social Media Evaluation Methods

Triangulation (Multiple Methods) Evaluation Approach

• Unique Twitter hashtag: #eXasw

• Follow-up follower/friend survey

• Follow-up project participant survey

• bit.ly analytics to determine number of clicks on unique embedded links

• Pre- and post-ASW Twitter influence metrics (Klout score)

Klout Score Progression• Klout is a measure of a person’s “influence” on twitter

• Based on an algorithm with factors such as number of followers and number of retweets

• Go to www.klout.com and log in with Twitter or Facebook

– 2011: 11.22 to 19.68

– 2012: 20.3 to 29.3

– 2013: 32.6 to 39.6

– 2014: 29.76 to 38.84

– 2015: 23.8 to 30.7

Twitter Chat Evaluation Methods

• Online survey (e.g., Qualtrics) link embedded into final tweets (with prizes as an incentive to complete)

Dr. Barbara O'Neill @moneytalk1Apr 29

Please let us know if U found #SSHWchat helpful & take this brief survey: https://rutgers.qualtrics.com/SE/?SID=SV_9X0V77kUKhOqlWB … Will pull winners at 1:30 pm #sshwchat

• TweetReach: http://tweetreach.com/ or Hashtracking: https://www.hashtracking.com/ (type in hashtag to pull a report)

• Follow-up contact from participants (Twitter direct messages, e-mail, etc.)

• Traffic to Web site during and after a chat

• Other?

Sample TweetReach Report

Archive a Twitter Chat With Storify and Track Use

https://storify.com/ Storify is an organized collection of tweets

with links, photos, etc. to “tell a story” from Twitter contentVideo: https://www.youtube.com/watch?v=l9iHniFjiVc

Storify Samples:

https://storify.com/RutgersNJAES/small-steps-to-health-and-wealth

https://storify.com/RutgersSEBS/cook-douglass-community-day-2014-at-rutgers

https://storify.com/wisebread/how-are-you-saving-for-retirement

https://storify.com/JerryBuchko/mcpd2014-twitter-chat-archive-1

One Last Step: Share Your Program Evaluation Results

WHAT?

SO

WHAT?

NOW

WHAT?

Who Needs to Know About the Results of

Your Educational Programs? Why?

Impact Statements: Intentions

As a result of participating in this financial education program, X% of participants reported that they…

• plan to do/use/adopt…

• are more knowledgeable about…

• are more confident in their ability to…

• are more likely than before to do/use/adopt…

• will do/use/adopt…

…a particular attitude, piece of information, or behavior.

Social Media Impact Reports

• Number of participants

• TweetReach report outreach numbers

• Participant survey data

• Data from other feedback methods

• Sample Evaluation Report: http://www.slideshare.net/BarbaraONeill/sshw-twitter-chat-impact-statement-0414

Public Value Approach

Private benefits accrue to program participants

Public benefits are benefits to those who did not

participate in a program

© 2007 Regents of the University of Minnesota

Public Value Statements

• Focus on an outcome that matters to stakeholder(s)

• Use stakeholder’s language

• Avoid jargon and empty words

• Should be short and believable

Participants in a nutrition program…

have been shown to increase their

consumption of fruits and

vegetables…

which has been shown to reduce the

incidence of diabetes…

which directly improves

quality of life for

participants.

which reduces public

health costs for all

citizens.

© 2007 Regents of the University of Minnesota

Participants

Changes

Outcome

Private benefits Public value

© 2007 Regents of the University of Minnesota

Public Value Statement Template

When you support _______________________program,

participants will ______________________________,

(changes)

which leads to _______________________________,

(outcomes)

which will benefit other community members by

_____________________________________________.

(public value)

© 2007 Regents of the University of Minnesota

Question

What are some public values for your educational programs?

• Launched in 2012

• Active online teaching community comprised of – Cooperative Extension faculty – Military family service professionals from the DoD, branch services– Non-governmental organizations

• Federal government grant-funded program is a collaborative effort of– eXtension– Cooperative Extension– DoD– NIFA, USDA

Military Families Learning Network

Personal Finance (MFLNPF)

•Personal Finance Concentration Area

– Builds the financial management capability and educational background of financial counselors and educators

Military Families Learning Network

Personal Finance

• Monthly web conferences offer credentialed participants the opportunity to earn continuing education unit (CEU) credit at no cost.

• Since 2012, the MFLNPF Concentration Area has provided certificates for almost 8,000 CEUs to personal finance web conference participants through the delivery of 61 online web conferences on a wide variety of personal finance topics.

• Link for webinars presented by all MFLN concentration areas: http://www.extension.org/pages/62581/military-families-learning-network-webinars

–Seven topic areas including Personal Finance

• Webinars are free and require no advance registration

• All webinars are marketed and archived on an eXtension Learn page

–Example: https://learn.extension.org/events/2149

MFLN Webinars

2012-2015 MFLNPF Webinars

• All MFLNPF webinars are archived at https://learn.extension.org/events/tag/mfln%20personal%20finance

• First webinar: 2/2/12- Housing and Foreclosure for Military Families

• Last 2015 webinar: 12/1/15- 2015 Personal Finance Year in Review

• Total of 56 webinars in four years (14 webinars per year)

More About MFLNPF Webinars

• Received 2014 AFCPE Outstanding Educational

Program Award

• Often include two presenters on a topic

–One for a general overview and one

for military-specific applications

• Available online 24/7/365

• CEU credit available upon 80% score on a 10-

question quiz

Image derived from photo by Dr. Irene Leech, Virginia Tech

Webinar Evaluation

48

Webinar Evaluation

49

Webinar Evaluation

50

Measuring Virtual Engagement

PF SMS iconsPF SMS icons

51

Facebook Insights

Sprout Social

Google Analytics

PF SMS iconsPF SMS icons

52

Facebook Insights

PF SMS iconsPF SMS icons

53

Sprout Social

PF SMS iconsPF SMS icons

54

Google Analytics

PF SMS iconsPF SMS icons

55

Social Media measurement tools can help

online communities of practice measure their

impact within their communities.

This data is valuable to stakeholders who wish

to see proof of interaction and engagement

among online community members.

Many social media measurement tools are free

and easy to use.

Questions and Comments?

Barbara O'Neill, Ph.D., CFP®, CRPC

Extension Specialist in Financial Resource Management and Distinguished Professor, Rutgers University

Phone: 848-932-9126

E-mail: oneill@aesop.rutgers.edu

Internet: http://njaes.rutgers.edu/money/

Twitter: http://twitter.com/moneytalk1