Evaluating the Efficacy of the Research Initiative for Scientific Enhancement (RISE) by using Propensity Scores to

Identify a Matched Comparison Group

Mica Estrada-Hollenbeck1

Anna Woodcock2

David Morella3

Wesley Schultz1

California State University, San Marcos2Purdue University

3Kent State University

Presented at the November 14, 2009 AEA Conference, Orlando, Florida.

Research Questions

1. Does participating in the RISE program increase the likelihood that a minority student will pursue a career in the biomedical sciences?

2. Are there some types of students who benefit more from the RISE program than others?

3. Are there elements of the RISE program that are linked with the success of the students?

4. Are the underlying assumptions regarding the efficacy of the elements of the RISE program valid?

The Challenge

POPULATION

TREATMENT

PP

PP

P

P

PPP

P

PP

P P

P

P

TT T

T PPP

Potential Statistical Solutions

When there is no randomized control group

• Blocked design

• Analysis of covariance

• Propensity scores

Corrects for selection bias when randomization is not possible

Propensity Scores Purpose

Population

Treatment

Propensity Score Definition

• Probability that a subject would receive treatment given a set of observed variables.

• People with similar propensity scores are similarly likely to receive treatment and can be compared to estimate the effect of treatment.

• Rosenbaum and Rubin (1983:420) suggest: If treatment assignment is strongly ignorable given

[covariates used to estimate propensity scores], then the difference between treatment and control means at each value of a [propensity] score is an

unbiased estimate of the treatment effect…”

> Propensity score matching assumes that all variables related to both outcomes and treatment assignment are included in the vector of observed variables (Rosenbaum & Rubin, 1983)

> The size of the population/sample from which the propensity scores are derived is important

- Minimally, there need to be people in the population who are similar to the treatment group.

Propensity Score Caveats

Previous Uses

Used primarily with studies retrospectively

• Education (Lee and Staff 2007; Wu et al. 2007)

• Economics (Dehejia and Wahba 2002, Benjamin 2003, Michalopoulos et al. 2004)

• Medical literature (Rubin 1999, Weitzen et al. 2004, Erosheva et al. 2007)

Overview: The Science Study

Longitudinal study of minority science students

From 45 campuses nationwide, 25 of these have RISE programs

1,380 participants Data collected twice yearly from students Propensity score matched control design Completing fourth year (8 waves of data)

Longitudinal Panel (at recruitment)

Longitudinal Panel

72% Female

Ethnicity/Race: 49% African American 39% Hispanic/Latino(a) 1% Native American

Major: 63% Biological Sciences 21% Natural Sciences 12% Behavioral & Social Sciences 4% Mathematics & Engineering

Survey Data Collection

Data collected through secure web interfacewww.TheScienceStudy.com

Propensity Score Generation

1. Identify what variables are predictive of treatment group -- Must be measurable items from both population and

treatment sample-- In a prospective longitudinal study, we use the first data

collected to identify the groups.

2. Using variables as predictors, a propensity score is generated for each person in the sample (logistic regression, SPSS).

3. Recruited match pool using faculty referrals:- RISE funded: N=750 possible, recruited 402

(added new students in W2 and W4)- Match pool: N=2166, wanted 402 (plus overmatch)

Propensity Score Generation

Additionally, age squared and interactions of Gender and GPA, Gender and Transfer Student Status, and Gender by educational status were added.

1. GPA 6. Parent Education2. Educational Status (LD, UD, Grad)

7. Academic major

3. Intention to pursue a scientific career

8. First Generation College Student

4. Ethnicity 9. Gender5. Transfers student status

10. Age

Everyone received a score

Propensity Scores

Population

Treatment

=.75

=.75

=.75

=.75

=.75=.85

=.85

=.85

=.85

=.85

=.93

=.93

=.93

=.93

=.93 =.97

=.93

=.93

=.93

=.93

=.94

=.67

=.67

=.67=.67

=.87

Propensity Scores Matching Process

RISE MATCHSubject 1=.86 Subject 10=.66Subject 2=.77 Subject 11=.72Subject 3=.81 Subject 12=.80Subject 4=.75 Subject 13=.43Subject 5=.68 Subject 14=.79Subject 6=.45 Subject 15=.44Subject 7=.57 Subject 16=.73Subject 8=.71 Subject 17=.59Subject 9=.82 Subject 18=.63

Subject 19=.51Subject 20=.85

Note: Change over time analyses conducted as a hierarchical linear model, with both linear and quadratic terms. Analyses are based on students who were undergraduates (jr. or sr.) at W0. Propensity score (W0) used as time invariant covariate. RISE = students continuously funded (N=101), and MATCH = students never funded (N=200) by any program and enrolled on a RISE campus. Dropped = students who were at one time enrolled in RISE but did not complete it. Intention to pursue career as biomedical scientist.

0 1 2 3 4 555.5

66.5

77.5

88.5

99.510

Growth Model: Student Intentions to Pursue a Scientific Career

RISE Match Dropped

Wave

Inte

ntio

n to

Pur

sue

a Ca

reer

in t

he B

iom

edic

al S

cien

ces

(Mod

eled

)

Baccalaureate Graduation (Fall, 08)

Group Graduated Not

MTP Funded 86% 14%

Dropped 74% 26%

Match 73% 27%

• No difference between Dropped & Match, χ2(1)=0.58, p=.81• Significant difference between RISE and Combined Dropped/Match, χ2(1)=10.37, p=.001

Graduate School:Applications & Offers

Group Completed Application No / Not Reported

MTP Funded (n=357) 43% 57%

Dropped (n=246) 33% 67%

Match (n=594) 26% 74%

Applications: Science-related Programs

Offers: Science-related Programs

Group Successful Offer No / Not Reported

MTP Funded (n=357) 38% 62%

Dropped (n=246) 22% 78%

Match (n=594) 19% 81%

Post Baccalaureate Outcomes (Fall, 08)

21 panel members with a Ph.D. (W6, 12/08) 1 panel member MD

2010/2011 first large wave of Ph.D. eligible panel members

Group MA/MS Ph.D. M.D.“Other”

Graduate Program

MTP Funded (n=125) 32% 38% 6% 19%

Dropped(n=104) 19% 11% 8% 22%

Match(n=212) 34% 8% 8% 16%

Enrollment

Attainment

Thank You

Bibliography

1. Benjamin, Daniel. 2003. “Does 401(k) Eligibility Increase Savings? Evidence from Propensity Score Classification.” Journal of Public Economics 87: 1259-1290.

2. Dehejia, Rajeev and Sadek Wahba. 2002. “Propensity Score Matching Methods for Non-Experimental Causal Studies.” Review of Economics and Statistics 84 (1): 151-161.

3. Lee, Jennifer C. and Jeremy Staff. 2007. “When Work Matters: The Varying Impact of Work Intensity on High School Dropout.” Sociology of Education 80 (2): 158-178.

4. Michalopoulos, Charles, Howard S. Bloom Carolyn J. Hill. 2004. “Can Propensity-Score Methods Match the Findings from a Random Assignment Evaluation of Mandatory Welfare-to-Work Programs?” Review of Economics and Statistics 86 (1): 156-179.

5. Rosenbaum, Paul and Donald Rubin. 1983. “The Central Role of the Propensity Score in Observation Studies for Causal Effects.” Biometrika 70 (1): 41-56.

6. Rubin, Donald. 2001. “Using Propensity Scores to Help Design Observational Studies: Application to the Tobacco Litigation.” Health Services and Outcomes Research Methodology 2: 169-188.

7. Weitzen, Sherry, Kate Lapane, Alicia Toledano, Anne Hume and Vincent Mor. 2004. “Principles for Modeling Propensity Scores in Medical Research: A Systemic Literature Review.” Pharmacoepidemiolgy and Drug Safety 13: 841-853.

8. Wu, Wei, Stephen West, and Jan Hughes. 2007. “Short-Term Effects of Grade Retention of the Growth Rate of Woodcock Johnson III Broad Math and Reading Scores.” Journal of School Psychology.