evaluating health communication strategies: part...

Evaluating Health Communication Strategies: Part 2

of 27

Evaluating Health Communication

Strategies: Part 2

Ms. McDivitt: What I have done sometimes

is, with the awareness and knowledge, I say they

will know this, this, and this, and they will

feel they are capable of doing this, this, and

this. That kind of says what do they need to

know, which you might want to make more obvious,

and we are going to have messages on that as well,

because, otherwise, you are sort of making this

assumption. Logic models are great for

bringing assumptions out.

Mr. Petty: Just a quick comment on your

situation because that is something that we are

facing with the National Diabetes Prevention

Program, trying to coordinate messages between

CDC, the Y, and UHG. Part of that depends on

the MOUs or the different kinds of arrangements

that you have with your partners.

To some extent, you cannot control

everything your partner does, obviously. But


of 27

that is something that you definitely want to

monitor and try to keep as consistent as

possible for a branding perspective, I mean for

a lot of different implications that relate to

communication.

But one thing that we are doing after a

year, we are setting up a meeting next month with

all of the communications folks from the three

organizations to really sit down and share each

other’s marketing plans and try to make things

a lot more consistent. So that we are all at

least putting out the same messages and kind of

having similar strategies, and are a little more

aware of what each other is doing.

But, I mean, with regular communication,

and you try to involve the partners as best you

can, but it is certainly a challenge, and you

are not alone with facing that. So, I just

wanted to make that comment.

Ms. McDivitt: You will often have a lot

of different things that you could look at. So,

you need to figure out, what am I going to look


of 27

at, because we never have enough money to look

at everything we want to. The kinds of things

that you think about selecting, again, these big

research questions, what you are going to look

at are the stakeholders. Who are the major

stakeholders? What do they want to know? How

important is it to answer their questions?

So, really trying to look at, if your

program is in danger of having its budget cut

if you don’t show impact, you probably would

like to measure that because that is a

stakeholder that is important, but, also,

answering their question is important. And

actually, to you, answering that question is

important.

Another is, how are you going to use the

answer if you get it? What are you going to do

if you get the answer you want or the answer you

don’t want? Is there something that you can do

in response? Is it going to help you make

decisions?

It is very tempting, particularly if there


of 27

is a survey going out, to add a whole pile of

other questions because they are interesting,

but they actually have nothing to do with trying

to really measure the different pieces of your

program. Surveys cost, and the more time you

have, the more someone is going to say, “You

know, I am tired of doing this,” and they quit.

So, really thinking about, if I find out

my program didn’t work after two years of doing

it, what am I going to do? Are you going to use

it? What can you do?

You might want to think about, you know,

I think maybe we had better measure that a little

bit earlier because, if at the end of the program

I find this out and I haven’t had any inkling

in those two years, that is going to be a problem

because you can’t use the answer. I mean, the

funder can use the answer. They can say, “Well,

we are not going to fund you again.”

So, really trying to think about, what are

you going to do if you get the answer to that

question? How are you going to use it? Are you


of 27

going to use it to make a decision? Or do you

just want to know this because you think it would

be fun to know it?

The thing is evaluation is not a research

study. I mean, you are trying to have

real-world questions. It is not just because

it would be nice to know.

How important is it to answer it? Here’s

where I make another plug for it is really

important to measure reach and exposure. It may

be that this is something that has to happen

before something else can happen.

The example of the partners, it may be very

important to answer that question because, if

they don’t do their part or if they do their part

in a way that doesn’t match the rest of it, you

are going to have a less-successful program.

The same thing with, you know, oftentimes, it

is important to measure was there an impact

because your funders want to know.

Where are there uncertainties?

Where are there potential problems? Are you


of 27

concerned that maybe the materials are not

really being developed properly and

disseminated properly? Are you concerned

about whether people are actually going to be

reached? Are you concerned about whether they

are going to reach the right people? It is kind

of, where are you uncertain?

Because, again, you don’t want to measure

something where you already know the answer. Or

you don’t want to spend a lot of time measuring

something where you already know the answer.

So, I think oftentimes, because it is easier,

we spend time measuring the inputs and the

outputs. Oftentimes, that is a

relatively-easy thing to do.

And so, you know that if you develop

materials, that they are going to be of high

quality. You know how many you sent out. But

what you don’t know oftentimes is the stuff that

happens afterwards. Again, here’s this whole

situation of uncertainty about what the

partners are going to do. You start kind of


of 27

looking at this and you say, okay, who wants to

know that? Can I use the answer? How important

is to know? Am I uncertain? And do I have the

resources to answer this question? That is, of

course, the million-dollar question.

You may say, “Yes, yes, yes” to all, which

would suggest, aha, I think maybe I should

focus, do something on this one. So, I

definitely want to measure whether the partners

are doing what they should do.

But the resources, you really have to kind

of look at: do you have the money? Do you have

the time? Do you have the staff and the skills

or can you get the skills to answer the kinds

of questions you have?

If you have big questions about causal

direction and whether any impact can be

attributed to your program only, you may have

to bring in some expertise that you don’t have.

It is going to cost you more. So, again, this

is where none of us have enough resources to do

this kind of thing the way we really want.


of 27

Identifying what you can do within the resources

you have got.

I think evaluators can be very creative.

There are lots of different ways to try to get

at stuff. There are kind of gold standards.

But if you don’t have any gold, then reaching

the gold standard is not really going to be

possible.

I think we are all in a situation right

now where we are trying to think about how can

we get the best answer we can within the

resources that we have got. And my big thing

is anything is better than nothing, and try to

do it the best you can within what you have got.

Once you know the big questions that you

have, the kinds of things that you are going to

want to know, and let’s say it is you want to

know whether you had any impact and kind of how

that happened, and then, you also have this big

question about whether the partners did their

part. We will just sort of pick those two. You

need to think about, okay, how am I actually


of 27

going to design the evaluation? How am I going

to collect the data?

As I said before, oftentimes, you are

going to need help from someone who has research

skills. You have a bunch of choices. But,

again, the kinds of things that you are going

to have to consider are: what is the purpose

of the evaluation? Am I doing this evaluation

mainly for me to understand the process? Or am

I doing this evaluation because the CDC has told

me I have to? What questions are you trying to

answer? Who are the stakeholders?

And here is where things come in of,

depending on some of that, how precise and how

good does your evaluation have to be? Do you

really need to have a whiz-bang, complicated

design? Or can you have something that will be

less costly, less complicated, and that will be

enough?

Once you kind of get a sense of am I going

to do a survey, am I going to do observations,

how am I going to organize this, you need to think


of 27

about what you are going to measure, specific

indicators of the concepts that you have got in

your logic model. If you say, “I have to reach

people. People need to be exposed,” what the

heck does that mean? This is kind of getting

to the question of how do you measure exposure

or knowledge or awareness or behavior change.

I mean, what does that really mean? What

specifically can you ask people or can you

observe to measure that?

With whom are you going to measure it, the

sources? Who needs to be in your sample? Who

are the people that you are addressing? What

are the kinds of people who need to be in your

evaluation?

The quality that is required, and I think

oftentimes, if you are doing some work mainly

to sort of do a gut check for yourself, then the

quality doesn’t need to be as great as for

something that has got bigger implications.

How much do you need to collect? How big

does your sample need to be? How much data do


of 27

you have to collect?

And then, there is sort of all the

logistical stuff that I think a lot of you are

probably aware of, but how you actually go and

collect the data, the specifics. Again, you

have to have a plan for how you are going to do

that.

So, in thinking about sort of the methods

you are going to use or the design you are going

to use, one of the things that we really would

like to be able to do, I think, is to say that

the impact that you got is due to your program.

So, essentially, looking at causal models.

Unfortunately, you need more complex, more

rigorous, and more expensive designs in order

to do that, particularly if you have

communication activities that are part of other

activities that are going on.

If you want to say, “Well, it was my

communication activity, but there was a policy

piece and the partners were providing more

services,” so really trying to identify was that


of 27

you. And so, this is not the answer you wanted

me to give you, but for communication programs

looking at was it my program or was it something

else can be pretty complicated.

And so, there are all these various

designs of like quasi-experimental designs and

before/after and control groups, and so forth.

Again, this is why oftentimes you are going to

want to bring a researcher into this.

I think one complication, I mean, this is

sort of a problem with evaluation, period, of

any activity that you are doing, is how can I

attribute it to my program. For communication

there is one complication, particularly for

mass media. That is most of the time mass media

anybody can get it. So, you can’t have a control

group. There isn’t anybody who didn’t get it.

One of the examples that is used is it would

be great if you had a whole sort of homogenous

group of people and half of them heard the

message and the other half didn’t because there

is nothing different, except there was a big


of 27

mountain in the way and you just couldn’t get

the radio signal. And so, that would be perfect

because, then, you would have people who are

pretty much the same. One got it; the other

didn’t get it.

But that usually doesn’t happen. The

people who don’t get reached are often

different. They often have fewer resources,

less access. They are crabby. Look at

differences in people using the internet, using

the web. People who don’t use the web tend to

be older, tend to have different socioeconomic

status. You just see all these differences.

You don’t have comparable groups. And so, you

can end up in trouble there.

This is one of the things. If you have a

communication activity where you can really

focus down to a narrow group of people, then you

have a better chance of looking at this

question. But, oftentimes, particularly for

mass media, it goes out to everybody.

So, okay, what are some other options,


of 27

then, in trying to think about some of the

programs that we do? Well, one is you can do

the program everywhere, but focus your

evaluation only in a few places. And you can

think about, okay, do I want places where

everybody is sort of the same or do I want a

variety? So that you can say, well, I am going

to do a community that is mostly low-income

African-American, and I am going to do a

community that is older adults, just so that you

can say, okay, did it work in different people?

This is not the gold standard. This is not

perfect. But really trying to think about what

is it you are trying to accomplish. Because if

you don’t expect a lot of change, you have to

have a really huge sample to be able to see

anything at all.

I think another thing, too, is to really

try – as I said before, we measure inputs. We

measure activities. Often, we don’t move

anywhere else down the line. So, if there is

any way at all of trying to measure exposure,


of 27

whether you actually reach the people you wanted

to reach, and this is a little bit what we are

doing in terms of some of the activities that

we have.

We do have a way of measuring, if we do

some promotional activities, did people

actually respond in any way whatsoever? So, you

are kind of getting to: did they ask for

materials? Did they call us? Did they call the

warehouse? Did they call NDEP at NIH? Did they

download anything? There is the capacity to do

this. So, that is kind of one thing: did they

do anything at all?

Now the problem here, though, is they may

have asked for it, and then they got it and they

put it on the shelf or they threw it in the trash.

Again, is there some way that you can, then, move

a little farther down and say, okay, these

people ordered this; these people requested

this. Then what? Can you call some of them up?

Again, you might want to call 100. If you can’t

afford 100, then you might want to call 25.


of 27

Again, I think sort of my constant refrain

is anything is better than nothing. Do the best

you can. Do the best evaluation work you can

within the resources you have got. It is

frustrating, but I think we have been finding

some creative ways to do things within the

limitations we have that are providing valuable

information. You do have to think, though, is

it valuable information or are you just

collecting data that is not useful?

Now there was a question yesterday and

also today about measuring exposure. There are

a whole bunch of ways you can do this. One is

just sort of basic exposure where you can count

people who attended a course, people who came

to a webinar, people who came to an event. That

doesn’t give you huge amounts of information.

It just tells you sort of numbers of people.

Sometimes you can know at least who they are;

sometimes you can’t.

And often, for exposure, you are going to

have to ask people questions. Asking people


of 27

questions about whether they have seen or heard

something, it is self-report. So, that always

ends up being a little problematic.

These are measures that the VERB Campaign

used. They started with an unprompted message.

Now I have tweaked it a little bit. “Have you

seen a booklet about steps for controlling

diabetes?” Yes/no?

Now what happens with a yes/no question?

Participant: You usually ask them and,

then, they will say yes.

Ms. McDivitt: Yes, people will often say

yes because they want to be nice to you. So,

then, you want to validate this a little bit.

You could say, “What’s it called?” Or you could

say, “What color is it?”

This can get a little ridiculous. I was

working on a project in Liberia where one of the

messages had a jingle. We would say, “Have you

heard messages on the radio about family

planning?” If they said yes, we said, “Can you

sing the jingle for us?” So, we are making


of 27

people sing. You know, who was the main

character? A couple of things that will help

you identify was it actually this or was it

something else. Again, there is tons of stuff

out there. So, some other kind of validation

measure.

VERB essentially said, “Have you heard any

messages or seen anything on TV about kids being

more active,” or something? And then, they

said, “Well, what was it called?” because the

brand was pretty in your face.

So, prompted, then, people who say no or

people who when you say, “What’s it called?”

they say, “the ladder to diabetes success.”

Well, that is not the name of it. It is called

Four Steps.

Then, you actually ask them, “Have you

seen a booklet called Four Steps?” Again, you

are going to end up, probably it is going to be

skewed, but you are going to have a couple of

measures now because you got the people who did

it unprompted. At some level, you can say they


of 27

are more aware.

And then, they also asked self-reported

frequency of exposure. Among those who were

aware from these two measures, “How often did

you see or hear these?” They gave them “every

day” – so they gave them categories of answers.

Again, this is not perfect. I mean, if I

asked you, “On average, how many hours of

television do you watch a week,” could you

answer that? Okay, so she could, but I would

have to go, okay, well, I don’t know. Did I

watch TV this week? I don’t know. What did I

watch?

You could say, “Okay, well, did you watch

it yesterday?” “Yes, but yesterday I was in the

middle of this course and there wasn’t anything

good, so I didn’t watch anything. But the day

before, I watched five hours.” So, these

exposure measures are pretty complicated.

But if you are asking it in a more general

way, oftentimes people can’t really remember.

If you say, “Have you ever heard about a pamphlet


of 27

called Four steps,” and someone says yes, but

that has been out for a long time. So, if I

asked, “Where did you hear about it,” how are

they going to answer? Are they going to

remember, I don’t know, the last thing? Are

they going to remember their favorite channel?

Are they going to say, “I saw it on TV,” but it

was not on TV. That kind of question is really

hard to answer.

I mean, “Have you heard any messages about

healthy diet lately?” I never know whether to

trust the answer to that question because it is

kind of is someone capable of answering it. Is

it true?

Then, there is another one, which is

essentially, “Did you do anything in response

to this material?” which a lot of people use.

It is also not one of my favorite ways of doing

this, although sometimes you have to. People

are going to say yes, because you asked me and

I want to give you the answer that you want.

There is this whole way of getting people


of 27

to answer sensitive questions. There is this

wonderful book on asking questions that has a

whole series of questions on how you ask the

question of whether somebody killed their wife

or not, because you have to give them the option

of saying no.

So, you can say, “How strongly do you agree

with the following: `I killed my wife.’?”

Strongly agree. Or “Some people kill their

wives; other people don’t. Do you happen to

have killed yours?”

Trying to find a way to give someone

permission to say they didn’t do it, but there

tends to be a strong bias to agreeing. So, even

on strongly agree or disagree – because people

know what the right answer is, which actually

is why it is very important to pretest any of

your instruments. Because, oh, I have had

experiences where you ask a question and it is

like, wow, that question stank.

You collect the data. Now you have to

analyze it. You have to make conclusions, and


of 27

you have to justify those conclusions. You have

got a bunch of stakeholders, and some of them

may not like the answers that you have come up

with.

So, there is a lot of thinking about how

do you pass the laugh test, so that the

stakeholders are going to believe what you found

and use it, especially if they don’t like it,

especially if you didn’t get what they wanted.

Here is where – and I think someone

mentioned this already – what are the standards

for the stakeholders? What do they think

success would be? Do they think that success

is 10 percent and you are thrilled because you

got 2 percent, but they are saying, “Failure.”?

Because, again, evaluation is a judgment. So,

that part of it is somewhat subjective. You

need to sort of come to some agreements on what

is going to be considered success.

The point that you brought up here of, to

you, success is if we got more partners; we did

this, we did this, we did this, but we didn’t


of 27

get down to being able to show someone actually

changed their behavior.

What is success and who is defining it?

Really thinking about doing the best analysis

that you can as objectively as you can. The same

thing with interpretation. With

interpretation, often it is helpful if you say,

particularly if it is successful, you say, “We

changed behavior, but here are some things we

didn’t change.”

I have been in situations where there are

people where everything is a success, and they

never talk about any of the problems. They

never talk about any of the difficulties. I

just keep on thinking I am not sure I believe

them.

So, really trying to make sure that you

have looked at things in a whole bunch of

different ways. Again, there is a lot of

judgment involved. Really thinking about one

person’s success can be another person’s

failure.


of 27

It is really important to come up with

recommendations. You really have to try to find

a way to make all this seem fair and unbiased,

that you don’t have sort of an axe to grind. Or

have other people come in and kind of help you

out with that.

Very important step, make sure that it is

going to be used and that lessons learned are

going to get out there. Again, you don’t do this

at the end where you say, “Okay, now I am ready

to pass it out to people.” If you are working

with your stakeholders throughout, if you are

thinking about use and how you are going to use

it, how other people may use it, then,

hopefully, by the time you get here, you are not

even really going to think about that.

You really need to design the evaluation

so it is going to answer the questions that

stakeholders have to make decisions. You need

to prepare the stakeholders for the results,

particularly if they are not what they wanted,

and understand sort of what their concerns are.


of 27

Get feedback. Involve them. Follow up with

people.

And really get them out to people in the

format they need. Programs I have been involved

with have been guilty of this. You produce this

huge report, and that is not really what people

need. They need something that is much shorter.

They need something that is in a completely

different format that is much more accessible.

Then, from a program planning point of

view, you want to be able to use the results

yourself. So, you may be thinking about, you

know, my goals and objectives, I don’t think

they were really realistic based on my

evaluation. Maybe I will change that. Or I

think maybe I need to do more. Or I need to do

something different. Or I need to really kind

of look at was it worth the cost, considering

what I got back from it. Oftentimes,

accountability to the funder is something that

you really need to think about. Again, really

using the knowledge that you learned.


of 27

These are some pitfalls in program

evaluation. Again, I alluded to this.

Collecting data that you are not going to use,

just because you can.

Questions on surveys and focus groups and

talking to people, those are very valuable

opportunities. And so, you really want to think

carefully before you sort of gum them up with

extra stuff.

Again – and it is kind of related – looking

at outcomes that are really beyond what you

could have been expected to do. So, doing

something that is looking at something that is

too big. Or looking at too many people. I think

you can really think about – and this is, again,

related to focus – if the program is focusing

on American Indians and Alaska Natives, just

because you can doesn’t mean you want to survey

everyone in three states where there are high

populations. Throughout this, I think we have

been talking about kind of focusing down and

thinking in a systematic way.


of 27

This is in your handouts, an example of

different kinds of questions that could be asked

in our small state, other evaluation. When I

was thinking about this last night, I am not sure

that we have gotten far enough on our case study

to really think about evaluation. We would be

having to make it all up. But I think you can

start to think about in your own programs how

you might want to design the evaluation.

evaluating health communication strategies: part...

Documents