Recruiting Sites and Participants for Random Assignment Studies

IES/NCER Summer InstituteJune 25, 2007Fred Doolittle

Overview of the Session Topics What is the purpose of site and

participant selection? The art of recruiting sites for random

assignment studies Topics to address Building the foundation for longer term

success of the study Addressing the tough issues in advance Assuring cooperation in study

implementation

The Goal of a “Fair Test” of the Intervention/Program Some elements of the “fair test”

Sites that serve the intended target group Sites that can operate the intervention reasonably

well Sites that can provide the intended service contrast A research design that can produce credible

findings on impacts and can be implemented A data collection plan that preserves the good

impact design and documents differences in services and outcomes at the various stages of the theory of action

An analysis strategy and report process that produces useful findings

In various ways, all of these involve the “sites”

Topics for a conversation The process of recruiting sites Embedding a random assignment design and

data collection into normal program operations Asking the tough questions

Real understanding of the design Political issues Ethical concerns

Working to achieve long-term cooperation in study implementation

Lack of crossovers and withdrawals Data collection

Reading the Site Recruitment Context

Factors to Consider How many sites do you need? What restrictions are there on your choices?

Geographic- need to be close to you, need to be a specific places because of funder, need to test in specific kinds of places, need to be representative

Programmatic- do sites need to be doing something on which you build not doing something so get service contrast

Size – need to be big enough to provide the desired sample of “clusters”, teachers and/or kids

Factors (cont.) Are you offering something new or

wanting to study existing programs? Does what you offer fit into their existing plans

programmatically and operationally so easy sell?

After-school example Reading Professional Development example

Is the existing program long established and popular?

Is it oversubscribed so there is “scarcity?” How have selection decisions been made? Does it have outcome-based performance standards?

Others????

Recruiting Framework What is the benefit-cost balance in

participating from the perspective of sites? Possible Benefits- the potential benefits for kids

because of new services, staff professional development, the visibility of being in a study, site-level findings, etc

Possible Costs – the need to devote management time to the intervention and study, a divergence from local priorities, hassle of integrating the intervention into local operations, controversy of random assignment, operational hassles of random assignment, the hassle of data collection, etc.

Responding to the Site Recruitment Context The context affects the sequence of topics

The more it looks like a tough sell, the more you lead with benefits of participating

The more it looks like an easy sell, the more you move quickly through the benefits and into the roles and responsibilities

In either case: The goal is to get folks interested in participating so

they can help you solve the inevitable problems, find ways to lessen the costs, and identify locally-relevant benefits

Always include the benefits in any material because it may be passed around to new folks who need to understand why to bother with this.

Examples of Strategies Building site “recruitment” into special program funding decisions

Still need engagement process Selecting “representative” sites

When might this be appropriate? What is the pitch?

Using data bases to identify prospects that fit the profile and then making “cold call” contact

Data bases may not include key criteria With whom to make the contact? What is the pitch?

Working through service provider or funding networks to gain access

Pros and cons What is the pitch?

Substantial outreach to build demand and stimulate “applications” What is the balance between benefits and obligations in initial

outreach? How orchestrate the application and selection process?

What to Expect When Make Contact

In districts/schools/programs: Everyone is overburdened and stressed Many things are changing simultaneously A new intervention is of greater interest than the

research Getting chosen for a study and/or getting something

free might be regarded as significant--but not for long Evaluation results –either overall or site-specific -

often do not matter much because way off in the future

Recruiting Process What is the point of entry?

The pros and cons of starting “high” in an organization Recovering from an initial rejection

First dates (i.e., meetings) are pivotal Make a “good impression” by bringing the right info

and people Get the right people from the site in the room

Build relationships from the outset Be the buyer and the seller—explain the

benefits but ask the tough questions relatively early

Try to understand their perspective Be willing to dig in to details

“Embedding” Random Assignment into Normal Operations

Why Does This Matter? Your goal is to mesh the research

procedures with their operations If you do this well, it significantly

reduces the cost to a site of participating

If you get them interested enough in the possible benefits, they will start helping you figure this out

Understanding Normal Program Operations How does the “sample” usually get to the

place (program, school, classroom, etc) where you want to introduce the intervention and do random assignment?

Ask managers and line staff Focus on understanding the usual way and any

exceptions, alternative routes in, etc. Understand the timing of the steps and the

information that is available at each step Understand the frequency of later “corrections” and

how done Can be because of mobility of students, late hires,

turnover Cannot “fix” this after random assignment

Understanding Normal Program Operations (cont.) For the kids:

Do kids apply? If so how and when? If not, how are appropriate kids identified?

Who is involved? Counselors, other teachers, referral agencies? Do they expect all they refer to get in?

Who decides which kids are “accepted”? When are these decisions made? What data is normally available or collected

on the kids at various stages? What percent of those selected actually

participate?

Understanding Normal Program Operations (cont.) For the staff:

How and when are staff assignments made? What is the usual skill set of staff? What other responsibilities do these staff have that

might affect your ability to train on intervention? Are there other constraints, for example union rules?

Matching kids with staff and services How and when is this normally done? Is there a tradition of special cases – parents, staff

requesting changes? Is there a tradition of deciding some kids “just have

to have” this staff person, program services, etc. and these are handled outside the usual process?

Normal Program Operations (cont.) Ask about information and time

available at key stages What info is available on kids at different

stages – to see if eligible/appropriate for intervention or to serve as baseline data?

Is there time to introduce informed consent and baseline data collection prior to random assignment?

When will any research delays be a problem – because scheduling complicated, because teachers/kids/parents need to know what is up, etc. ?

Examples Individual random assignment

After-school services Upward Bound

School level random assignment Elementary school teacher professional

development in reading through summer institutes and coaching

Middle school teacher professional development in math through summer institutes and coaching

Focusing on Service Receipt Two parts to this are key because the

service contrast is what drives impacts Program group: Want strong

implementation of the intervention to intended kids, with intended participation Worry about mobility of kids and turnover of

staff Control group: Want clear service

contrast for the control group Not necessarily no service But clearly different services

Picking the Point of Random Assignment How close to the start of services do you put

the lottery? Choice affects:

the question you address, the sample, the baseline data you will have, and service participation for program and control group

The tradeoff: “Late” means greater participation rate for the

program group but also more hassle for staff, harder to plan, more disappointment if not selected, and -usually - more motivated control group so more services for controls

“Early” means the opposite

Examples

Individual level random assignment KIPP middle school evaluation Mentoring programs

Cluster random assignment The previous professional

development examples

Setting Random Assignment Ratios This can be part of the site recruitment

discussion A balanced design is best for power, but not necessary The power drop-off is not major until move past 2:1 An unbalanced design can make a big difference to

sites and help address ethical concerns A very unbalanced design with many sites is an option

if data collection costs are manageable Avoiding empty program slots is important

“Overbook” somewhat the program group Use a non-research waiting list to fill empty slots In extreme, can set program group size to fill available

program slots and the control group is the rest

Handling the Sensitive Cases Much better if do this prior to random assignment

Tell programs if absolutely necessary they have a few “chips” they can use prior to the lottery for cases where they could not live with control group designation

They apply to use a chip for a person or cluster before random assignment

Affects generalizability but not internal validity In cluster random assignment, can include some “extra”

non-study staff from program clusters in program services Sometimes have to accept a “crossover” to save a

study Small numbers in a large sample do not matter a lot

Don’t let post-lottery procedures undo the “randomness”

Post-random assignment allocation of kids to clusters Compare school-level random

assignment to teacher-level random assignment

In the latter, important to get commitment of kids to clusters or a clear process to do it before conduct random assignment

Avoid Undoing Randomness Data collection problems

Low rates for sample Differential rates across research groups

Build tracking into design Try to mesh data collection with normal participation in

services Create incentives

Cover the costs at a minimum Provide positive incentives- there are OMB limits Special incentives for control sample?

Clusters Individuals

Pay for a local study liaison to coordinate data collection

Asking the Tough Questions Early

Tough Questions to Ask Do you really want to do this or is

someone forcing you? Do you really understand random

assignment? Are you committed to using a lottery to

decide who gets the intervention and who does not?

Do you really understand your roles and responsibilities related to: Research procedures Data collection Implementation of the intervention

More Tough Things to Ask Who could object to the study

procedures and are you prepared to confront these objections and stay the course? unions, parents, governing board,

participant referral sources Can you devote the staff time

needed to manage the study and the intervention?

Even More Tough Questions to Ask Are you about to (likely to) do

something else that will affect our ability to pull off the study? Change local priorities so not

interested? Introduce some other similar

intervention so not a service contrast? Lay off staff so cannot staff it? Study champion is about to retire or

take a new job?

Still More Tough Questions Are there lurking ethical concerns?

Uncertainty about real scarcity? Need to recruit more to have a control

group? Local certainty the intervention really works

so should not deny access? Local certainty can identify those most in

need who will benefit? Sense of coercion despite informed consent? ????

Negotiating and Closing the Deal

Negotiating a Formal Agreement

A formal, detailed agreement is important

It should specify respective roles and responsibilities about both program and research activities, timelines, costs

The agreement must be cleared directly with all relevant decision makers

The head person must read and sign This takes time but worth the effort in

the long run

Negotiating and setting the stage (cont.)

Once signed an agreement is gradually forgotten by many

Therefore, one must have at least one properly placed internal champion to push forward the study

The champion must be tended and kept engaged via ongoing communication

Other relationships also matter; make the rounds periodically